Journal article 587 views 52 downloads
Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
,
Benjamin Mora ,
Xianghua Xie
IEEE Transactions on Neural Networks and Learning Systems, Pages: 1 - 14
Swansea University Authors:
Hassan Eshkiki , Benjamin Mora , Xianghua Xie
-
PDF | Accepted Manuscript
Distributed under the terms of a Creative Commons Attribution 4.0 License (CC BY 4.0).
Download (3.89MB)
DOI (Published version): 10.1109/tnnls.2022.3176197
Abstract
This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the pr...
| Published in: | IEEE Transactions on Neural Networks and Learning Systems |
|---|---|
| ISSN: | 2162-237X 2162-2388 |
| Published: |
Institute of Electrical and Electronics Engineers (IEEE)
2022
|
| Online Access: |
Check full text
|
| URI: | https://cronfa.swan.ac.uk/Record/cronfa60038 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract: |
This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the precision desired. The number of instructions carried out is mainly bounded by bitwise operators, amenable to a simplified processing architecture and compressed matrix weights. Results show that the method is superior in size and number of operations to the standard approximation with signed matrices. Equally important, this paper demonstrates a first application to machine learning inference by showing that weights of fully-connected layers can be compressed between 30× and 100× with little to no loss in inference accuracy. The requirements for pure floating-point operations are also down as our algorithm relies mainly on simpler bitwise operators. |
|---|---|
| Item Description: |
In Press. |
| Keywords: |
Complexity theory, Approximation algorithms, Neural networks, Monte Carlo methods, Heart, Transforms, Inference algorithms |
| College: |
Faculty of Science and Engineering |
| Funders: |
This work was supported in part by the UK Government through the Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/R51312X/1 and in part by the Natural Environment Research Council (NERC) under Grant NE/W502911/1. |
| Start Page: |
1 |
| End Page: |
14 |