No Cover Image

Journal article 35 views

Fully Connected Networks on a Diet With the Mediterranean Matrix Multiplication

Hassan Eshkiki, Benjamin Mora Orcid Logo, Xianghua Xie Orcid Logo

IEEE Transactions on Neural Networks and Learning Systems, Pages: 1 - 14

Swansea University Authors: Hassan Eshkiki, Benjamin Mora Orcid Logo, Xianghua Xie Orcid Logo

  • 60038.pdf

    PDF | Accepted Manuscript

    This work is licensed under a Creative Commons Attribution 4.0 License

    Download (3.89MB)

Abstract

This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the pr...

Full description

Published in: IEEE Transactions on Neural Networks and Learning Systems
ISSN: 2162-237X 2162-2388
Published: Institute of Electrical and Electronics Engineers (IEEE)
Online Access: Check full text

URI: https://cronfa.swan.ac.uk/Record/cronfa60038
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract: This paper proposes the Mediterranean Matrix Multiplication, a new, simple and practical randomized algorithm that samples angles between the rows and columns of two matrices with sizes m, n, p to approximate matrix multiplication in O(k(mn+np+mp)) steps, where k is a constant only related to the precision desired. The number of instructions carried out is mainly bounded by bitwise operators, amenable to a simplified processing architecture and compressed matrix weights. Results show that the method is superior in size and number of operations to the standard approximation with signed matrices. Equally important, this paper demonstrates a first application to machine learning inference by showing that weights of fully-connected layers can be compressed between 30× and 100× with little to no loss in inference accuracy. The requirements for pure floating-point operations are also down as our algorithm relies mainly on simpler bitwise operators.
College: College of Science
Funders: This work was supported in part by the UK Government through the Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/R51312X/1 and in part by the Natural Environment Research Council (NERC) under Grant NE/W502911/1.
Start Page: 1
End Page: 14