Journal article 645 views 140 downloads
Deep Collaborative Learning for Randomly Wired Neural Networks
Electronics, Volume: 10, Issue: 14, Start page: 1669
Swansea University Author:
Xianghua Xie
-
PDF | Version of Record
Copyright: © 2021 by the authors. This is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license
Download (790.29KB)
DOI (Published version): 10.3390/electronics10141669
Abstract
A deep collaborative learning approach is introduced in which a chain of randomly wired neural networks is trained simultaneously to improve the overall generalization and form a strong ensemble model. The proposed method takes advantage of functional-preserving transfer learning and knowledge disti...
| Published in: | Electronics |
|---|---|
| ISSN: | 2079-9292 |
| Published: |
MDPI AG
2021
|
| Online Access: |
Check full text
|
| URI: | https://cronfa.swan.ac.uk/Record/cronfa57529 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract: |
A deep collaborative learning approach is introduced in which a chain of randomly wired neural networks is trained simultaneously to improve the overall generalization and form a strong ensemble model. The proposed method takes advantage of functional-preserving transfer learning and knowledge distillation to produce an ensemble model. Knowledge distillation is an effective learning scheme for improving the performance of small neural networks by using the knowledge learned by teacher networks. Most of the previous methods learn from one or more teachers but not in a collaborative way. In this paper, we created a chain of randomly wired neural networks based on a random graph algorithm and collaboratively trained the models using functional-preserving transfer learning, so that the small network in the chain could learn from the largest one simultaneously. The training method applies knowledge distillation between randomly wired models, where each model is considered as a teacher to the next model in the chain. The decision of multiple chains of models can be combined to produce a robust ensemble model. The proposed method is evaluated on CIFAR-10, CIFAR-100, and TinyImageNet. The experimental results show that the collaborative training significantly improved the generalization of each model, which allowed for obtaining a small model that can mimic the performance of a large model and produce a more robust ensemble approach. |
|---|---|
| Keywords: |
randomly wired neural networks; model distillation; ensemble model; deep learning |
| College: |
Faculty of Science and Engineering |
| Funders: |
Serˆ Cymru COFUND Fellowship |
| Issue: |
14 |
| Start Page: |
1669 |