Skip to Main content Skip to Navigation
Journal articles

Thermodynamics of Restricted Boltzmann Machines and Related Learning Dynamics

Aurélien Decelle 1 Giancarlo Fissore 1 Cyril Furtlehner 2
2 TAU - TAckling the Underspecified
Inria Saclay - Ile de France, LRI - Laboratoire de Recherche en Informatique
Abstract : We analyze the learning process of the restricted Boltzmann machine (RBM), a certain type of generative models used in the context of unsupervized learning. In a first step, we investigate its thermodynamics properties by considering a realistic statistical ensemble of RBM. We adopt the viewpoint that the information content of the RBM is mainly reflected by the spectral properties of its weight matrix $W$, i.e. the couplings matrix. Schematically the bottom of the spectrum is occupied by a Marchenko-Pastur (MP) distribution of singular values representing noise, while actual information is contained in modes outside this bulk. A phase diagram is obtained which seems at first sight similar to the one of the Sherrington-Kirkpatrick (SK) with ferromagnetic couplings. The main difference resides in the structure of the ferromagnetic phase, which depending on the distribution of the singular vectors components, may or may not be of compositional type, i.e. combining or not dominant modes of $W$ for expressing magnetization. In a second step the learning dynamics of an RBM given arbitrary data is studied in the thermodynamic limit. A ``typical'' learning trajectory is shown to solve an effective equation, which is obtained by making use of the aforementioned ensemble average and where the ferromagnetic order parameters enter explicitly. This accounts in particular for the dominant singular values evolution and how this is driven by the input data: in the linear regime at the beginning of the learning, they correspond to unstable deformation modes of $W$ reflecting dominant covariance modes of the data. In the non-linear regime is unveiled in some way how the selected modes interact in later stages of the learning procedure. Experiments on both artificial and real data illustrate these considerations, showing in particular how the RBM operates in the ferromagnetic compositional phase.
Complete list of metadata
Contributor : Cyril Furtlehner Connect in order to contact the contributor
Submitted on : Friday, March 9, 2018 - 9:26:52 AM
Last modification on : Thursday, July 8, 2021 - 3:50:35 AM
Long-term archiving on: : Sunday, June 10, 2018 - 1:06:41 PM


Files produced by the author(s)



Aurélien Decelle, Giancarlo Fissore, Cyril Furtlehner. Thermodynamics of Restricted Boltzmann Machines and Related Learning Dynamics. Journal of Statistical Physics, Springer Verlag, 2018, 172 (6), pp.1576-1608. ⟨10.1007/s10955-018-2105-y⟩. ⟨hal-01675310v2⟩



Record views


Files downloads