Journal article 614 views 55 downloads
Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication
IEEE Transactions on Neural Networks and Learning Systems, Pages: 1 - 14
Swansea University Authors: Hassan Eshkiki , Benjamin Mora , Xianghua Xie
-
PDF | Accepted Manuscript
Distributed under the terms of a Creative Commons Attribution 4.0 License (CC BY 4.0).
Download (3.89MB)
DOI (Published version): 10.1109/tnnls.2022.3176197
Abstract
This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the pr...
Published in: | IEEE Transactions on Neural Networks and Learning Systems |
---|---|
ISSN: | 2162-237X 2162-2388 |
Published: |
Institute of Electrical and Electronics Engineers (IEEE)
2022
|
Online Access: |
Check full text
|
URI: | https://cronfa.swan.ac.uk/Record/cronfa60038 |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
first_indexed |
2022-05-16T16:49:00Z |
---|---|
last_indexed |
2023-04-19T03:18:47Z |
id |
cronfa60038 |
recordtype |
SURis |
fullrecord |
<?xml version="1.0" encoding="utf-8"?><rfc1807 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><bib-version>v2</bib-version><id>60038</id><entry>2022-05-16</entry><title>Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication</title><swanseaauthors><author><sid>c9972b26a83de11ffe211070f26fe16b</sid><ORCID>0000-0001-7795-453X</ORCID><firstname>Hassan</firstname><surname>Eshkiki</surname><name>Hassan Eshkiki</name><active>true</active><ethesisStudent>false</ethesisStudent></author><author><sid>557f93dfae240600e5bd4398bf203821</sid><ORCID>0000-0002-2945-3519</ORCID><firstname>Benjamin</firstname><surname>Mora</surname><name>Benjamin Mora</name><active>true</active><ethesisStudent>false</ethesisStudent></author><author><sid>b334d40963c7a2f435f06d2c26c74e11</sid><ORCID>0000-0002-2701-8660</ORCID><firstname>Xianghua</firstname><surname>Xie</surname><name>Xianghua Xie</name><active>true</active><ethesisStudent>false</ethesisStudent></author></swanseaauthors><date>2022-05-16</date><deptcode>SCS</deptcode><abstract>This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the precision desired. The number of instructions carried out is mainly bounded by bitwise operators, amenable to a simplified processing architecture and compressed matrix weights. Results show that the method is superior in size and number of operations to the standard approximation with signed matrices. Equally important, this paper demonstrates a first application to machine learning inference by showing that weights of fully-connected layers can be compressed between 30× and 100× with little to no loss in inference accuracy. The requirements for pure floating-point operations are also down as our algorithm relies mainly on simpler bitwise operators.</abstract><type>Journal Article</type><journal>IEEE Transactions on Neural Networks and Learning Systems</journal><volume>0</volume><journalNumber/><paginationStart>1</paginationStart><paginationEnd>14</paginationEnd><publisher>Institute of Electrical and Electronics Engineers (IEEE)</publisher><placeOfPublication/><isbnPrint/><isbnElectronic/><issnPrint>2162-237X</issnPrint><issnElectronic>2162-2388</issnElectronic><keywords>Complexity theory, Approximation algorithms, Neural networks, Monte Carlo methods, Heart, Transforms, Inference algorithms</keywords><publishedDay>1</publishedDay><publishedMonth>1</publishedMonth><publishedYear>2022</publishedYear><publishedDate>2022-01-01</publishedDate><doi>10.1109/tnnls.2022.3176197</doi><url>http://dx.doi.org/10.1109/tnnls.2022.3176197</url><notes>In Press.</notes><college>COLLEGE NANME</college><department>Computer Science</department><CollegeCode>COLLEGE CODE</CollegeCode><DepartmentCode>SCS</DepartmentCode><institution>Swansea University</institution><apcterm>SU College/Department paid the OA fee</apcterm><funders>This work was supported in part by the UK Government through the Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/R51312X/1 and in part by the Natural Environment Research Council (NERC) under Grant NE/W502911/1.</funders><projectreference/><lastEdited>2023-11-08T15:53:00.3476012</lastEdited><Created>2022-05-16T17:37:16.9524707</Created><path><level id="1">Faculty of Science and Engineering</level><level id="2">School of Mathematics and Computer Science - Computer Science</level></path><authors><author><firstname>Hassan</firstname><surname>Eshkiki</surname><orcid>0000-0001-7795-453X</orcid><order>1</order></author><author><firstname>Benjamin</firstname><surname>Mora</surname><orcid>0000-0002-2945-3519</orcid><order>2</order></author><author><firstname>Xianghua</firstname><surname>Xie</surname><orcid>0000-0002-2701-8660</orcid><order>3</order></author></authors><documents><document><filename>60038__24511__86ec71af591f437aa3f0505e3a023b37.pdf</filename><originalFilename>60038.pdf</originalFilename><uploaded>2022-07-08T14:34:15.9357207</uploaded><type>Output</type><contentLength>4081808</contentLength><contentType>application/pdf</contentType><version>Accepted Manuscript</version><cronfaStatus>true</cronfaStatus><documentNotes>Distributed under the terms of a Creative Commons Attribution 4.0 License (CC BY 4.0).</documentNotes><copyrightCorrect>true</copyrightCorrect><language>eng</language><licence>https://creativecommons.org/licenses/by/4.0/</licence></document></documents><OutputDurs/></rfc1807> |
spelling |
v2 60038 2022-05-16 Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication c9972b26a83de11ffe211070f26fe16b 0000-0001-7795-453X Hassan Eshkiki Hassan Eshkiki true false 557f93dfae240600e5bd4398bf203821 0000-0002-2945-3519 Benjamin Mora Benjamin Mora true false b334d40963c7a2f435f06d2c26c74e11 0000-0002-2701-8660 Xianghua Xie Xianghua Xie true false 2022-05-16 SCS This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the precision desired. The number of instructions carried out is mainly bounded by bitwise operators, amenable to a simplified processing architecture and compressed matrix weights. Results show that the method is superior in size and number of operations to the standard approximation with signed matrices. Equally important, this paper demonstrates a first application to machine learning inference by showing that weights of fully-connected layers can be compressed between 30× and 100× with little to no loss in inference accuracy. The requirements for pure floating-point operations are also down as our algorithm relies mainly on simpler bitwise operators. Journal Article IEEE Transactions on Neural Networks and Learning Systems 0 1 14 Institute of Electrical and Electronics Engineers (IEEE) 2162-237X 2162-2388 Complexity theory, Approximation algorithms, Neural networks, Monte Carlo methods, Heart, Transforms, Inference algorithms 1 1 2022 2022-01-01 10.1109/tnnls.2022.3176197 http://dx.doi.org/10.1109/tnnls.2022.3176197 In Press. COLLEGE NANME Computer Science COLLEGE CODE SCS Swansea University SU College/Department paid the OA fee This work was supported in part by the UK Government through the Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/R51312X/1 and in part by the Natural Environment Research Council (NERC) under Grant NE/W502911/1. 2023-11-08T15:53:00.3476012 2022-05-16T17:37:16.9524707 Faculty of Science and Engineering School of Mathematics and Computer Science - Computer Science Hassan Eshkiki 0000-0001-7795-453X 1 Benjamin Mora 0000-0002-2945-3519 2 Xianghua Xie 0000-0002-2701-8660 3 60038__24511__86ec71af591f437aa3f0505e3a023b37.pdf 60038.pdf 2022-07-08T14:34:15.9357207 Output 4081808 application/pdf Accepted Manuscript true Distributed under the terms of a Creative Commons Attribution 4.0 License (CC BY 4.0). true eng https://creativecommons.org/licenses/by/4.0/ |
title |
Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication |
spellingShingle |
Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication Hassan Eshkiki Benjamin Mora Xianghua Xie |
title_short |
Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication |
title_full |
Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication |
title_fullStr |
Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication |
title_full_unstemmed |
Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication |
title_sort |
Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication |
author_id_str_mv |
c9972b26a83de11ffe211070f26fe16b 557f93dfae240600e5bd4398bf203821 b334d40963c7a2f435f06d2c26c74e11 |
author_id_fullname_str_mv |
c9972b26a83de11ffe211070f26fe16b_***_Hassan Eshkiki 557f93dfae240600e5bd4398bf203821_***_Benjamin Mora b334d40963c7a2f435f06d2c26c74e11_***_Xianghua Xie |
author |
Hassan Eshkiki Benjamin Mora Xianghua Xie |
author2 |
Hassan Eshkiki Benjamin Mora Xianghua Xie |
format |
Journal article |
container_title |
IEEE Transactions on Neural Networks and Learning Systems |
container_volume |
0 |
container_start_page |
1 |
publishDate |
2022 |
institution |
Swansea University |
issn |
2162-237X 2162-2388 |
doi_str_mv |
10.1109/tnnls.2022.3176197 |
publisher |
Institute of Electrical and Electronics Engineers (IEEE) |
college_str |
Faculty of Science and Engineering |
hierarchytype |
|
hierarchy_top_id |
facultyofscienceandengineering |
hierarchy_top_title |
Faculty of Science and Engineering |
hierarchy_parent_id |
facultyofscienceandengineering |
hierarchy_parent_title |
Faculty of Science and Engineering |
department_str |
School of Mathematics and Computer Science - Computer Science{{{_:::_}}}Faculty of Science and Engineering{{{_:::_}}}School of Mathematics and Computer Science - Computer Science |
url |
http://dx.doi.org/10.1109/tnnls.2022.3176197 |
document_store_str |
1 |
active_str |
0 |
description |
This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the precision desired. The number of instructions carried out is mainly bounded by bitwise operators, amenable to a simplified processing architecture and compressed matrix weights. Results show that the method is superior in size and number of operations to the standard approximation with signed matrices. Equally important, this paper demonstrates a first application to machine learning inference by showing that weights of fully-connected layers can be compressed between 30× and 100× with little to no loss in inference accuracy. The requirements for pure floating-point operations are also down as our algorithm relies mainly on simpler bitwise operators. |
published_date |
2022-01-01T15:53:03Z |
_version_ |
1782011693848592384 |
score |
11.016235 |