Thermodynamics of Restricted Boltzmann Machines and Related Learning Dynamics

Aurélien Decelle 1 Giancarlo Fissore 1 Cyril Furtlehner 2
2 TAU - TAckling the Underspecified
LRI - Laboratoire de Recherche en Informatique, UP11 - Université Paris-Sud - Paris 11, Inria Saclay - Ile de France, CNRS - Centre National de la Recherche Scientifique : UMR8623
Abstract : We analyze the learning process of the restricted Boltzmann machine (RBM), a certain type of generative models used in the context of unsupervized learning. In a first step, we investigate its thermodynamics properties by considering a realistic statistical ensemble of RBM. We adopt the viewpoint that the information content of the RBM is mainly reflected by the spectral properties of its weight matrix $W$, i.e. the couplings matrix. Schematically the bottom of the spectrum is occupied by a Marchenko-Pastur (MP) distribution of singular values representing noise, while actual information is contained in modes outside this bulk. A phase diagram is obtained which seems at first sight similar to the one of the Sherrington-Kirkpatrick (SK) with ferromagnetic couplings. The main difference resides in the structure of the ferromagnetic phase, which depending on the distribution of the singular vectors components, may or may not be of compositional type, i.e. combining or not dominant modes of $W$ for expressing magnetization. In a second step the learning dynamics of an RBM given arbitrary data is studied in the thermodynamic limit. A ``typical'' learning trajectory is shown to solve an effective equation, which is obtained by making use of the aforementioned ensemble average and where the ferromagnetic order parameters enter explicitly. This accounts in particular for the dominant singular values evolution and how this is driven by the input data: in the linear regime at the beginning of the learning, they correspond to unstable deformation modes of $W$ reflecting dominant covariance modes of the data. In the non-linear regime is unveiled in some way how the selected modes interact in later stages of the learning procedure. Experiments on both artificial and real data illustrate these considerations, showing in particular how the RBM operates in the ferromagnetic compositional phase.
Complete list of metadatas

Cited literature [35 references]  Display  Hide  Download
Contributor : Cyril Furtlehner <>
Submitted on : Thursday, January 4, 2018 - 3:32:10 PM
Last modification on : Tuesday, January 8, 2019 - 8:36:01 AM
Long-term archiving on : Wednesday, May 2, 2018 - 8:29:07 PM


Files produced by the author(s)


  • HAL Id : hal-01675310, version 1



Aurélien Decelle, Giancarlo Fissore, Cyril Furtlehner. Thermodynamics of Restricted Boltzmann Machines and Related Learning Dynamics. [Research Report] RR-9139, Inria Saclay Ile de France; LRI, Université Paris-Sud. 2018, pp.1-36. ⟨hal-01675310v1⟩



Record views


Files downloads