# Thermodynamics of Restricted Boltzmann Machines and Related Learning Dynamics

2 TAU - TAckling the Underspecified
LRI - Laboratoire de Recherche en Informatique, Inria Saclay - Ile de France
Abstract : We analyze the learning process of the restricted Boltzmann machine (RBM), a certain type of generative models used in the context of unsupervized learning. In a first step, we investigate its thermodynamics properties by considering a realistic statistical ensemble of RBM. We adopt the viewpoint that the information content of the RBM is mainly reflected by the spectral properties of its weight matrix $W$, i.e. the couplings matrix. Schematically the bottom of the spectrum is occupied by a Marchenko-Pastur (MP) distribution of singular values representing noise, while actual information is contained in modes outside this bulk. A phase diagram is obtained which seems at first sight similar to the one of the Sherrington-Kirkpatrick (SK) with ferromagnetic couplings. The main difference resides in the structure of the ferromagnetic phase, which depending on the distribution of the singular vectors components, may or may not be of compositional type, i.e. combining or not dominant modes of $W$ for expressing magnetization. In a second step the learning dynamics of an RBM given arbitrary data is studied in the thermodynamic limit. A typical'' learning trajectory is shown to solve an effective equation, which is obtained by making use of the aforementioned ensemble average and where the ferromagnetic order parameters enter explicitly. This accounts in particular for the dominant singular values evolution and how this is driven by the input data: in the linear regime at the beginning of the learning, they correspond to unstable deformation modes of $W$ reflecting dominant covariance modes of the data. In the non-linear regime is unveiled in some way how the selected modes interact in later stages of the learning procedure. Experiments on both artificial and real data illustrate these considerations, showing in particular how the RBM operates in the ferromagnetic compositional phase.
Keywords :
Document type :
Journal articles
Domain :

https://hal.inria.fr/hal-01675310
Contributor : Cyril Furtlehner <>
Submitted on : Friday, March 9, 2018 - 9:26:52 AM
Last modification on : Wednesday, October 14, 2020 - 4:13:28 AM
Long-term archiving on: : Sunday, June 10, 2018 - 1:06:41 PM

### File

RR-9139.pdf
Files produced by the author(s)

### Citation

Aurélien Decelle, Giancarlo Fissore, Cyril Furtlehner. Thermodynamics of Restricted Boltzmann Machines and Related Learning Dynamics. Journal of Statistical Physics, Springer Verlag, 2018, 172 (6), pp.1576-1608. ⟨10.1007/s10955-018-2105-y⟩. ⟨hal-01675310v2⟩

Record views