Journal article 126 views 32 downloads
Stochastic weight matrix dynamics during learning and Dyson Brownian motion
,
Biagio Lucini ,
Chanju Park
Physical Review E, Volume: 111, Issue: 1
Swansea University Authors:
Gert Aarts , Biagio Lucini
-
PDF | Version of Record
Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license.
Download (915.79KB)
DOI (Published version): 10.1103/physreve.111.015303
Abstract
We demonstrate that the update of weight matrices in learning algorithms can be described in the framework of Dyson Brownian motion, thereby inheriting many features of random matrix theory. We relate the level of stochasticity to the ratio of the learning rate and the minibatch size, providing more...
| Published in: | Physical Review E |
|---|---|
| ISSN: | 2470-0045 2470-0053 |
| Published: |
American Physical Society (APS)
2025
|
| Online Access: |
Check full text
|
| URI: | https://cronfa.swan.ac.uk/Record/cronfa68607 |
| Abstract: |
We demonstrate that the update of weight matrices in learning algorithms can be described in the framework of Dyson Brownian motion, thereby inheriting many features of random matrix theory. We relate the level of stochasticity to the ratio of the learning rate and the minibatch size, providing more robust evidence to a previously conjectured scaling relationship. We discuss universal and nonuniversal features in the resulting Coulomb gas distribution and identify the Wigner surmise and Wigner semicircle explicitly in a teacher-student model and in the (near-)solvable case of the Gaussian restricted Boltzmann machine. |
|---|---|
| College: |
Faculty of Science and Engineering |
| Funders: |
G.A. andB.L. are supported by STFC Consolidated Grant No. ST/X000648/1 .B. L. is further supported by the UKRI EPSRC ExCALIBURExaTEPP Project No. EP/X017168/1. C. P. is supported by the UKRI AIMLAC CDT EP/S023992/1. |
| Issue: |
1 |