Spectral Dynamics of Learning Restricted Boltzmann Machines

A Decelle 1 G Fissore 1 Cyril Furtlehner 2
2 TAU - TAckling the Underspeficied
LRI - Laboratoire de Recherche en Informatique, UP11 - Université Paris-Sud - Paris 11, Inria Saclay - Ile de France, CNRS - Centre National de la Recherche Scientifique : UMR8623
Abstract : The Restricted Boltzmann Machine (RBM), an important tool used in machine learning in particular for unsupervized learning tasks, is investigated from the perspective of its spectral properties. Starting from empirical observations, we propose a generic statistical ensemble for the weight matrix of the RBM and characterize its mean evolution. This let us show how in the linear regime, in which the RBM is found to operate at the beginning of the training, the statistical properties of the data drive the selection of the unstable modes of the weight matrix. A set of equations characterizing the non-linear regime is then derived, unveiling in some way how the selected modes interact in later stages of the learning procedure and defining a deterministic learning curve for the RBM.
Type de document :
Article dans une revue
EPL - Europhysics Letters, European Physical Society/EDP Sciences/Società Italiana di Fisica/IOP Publishing, 2017
Liste complète des métadonnées

Littérature citée [14 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01652314
Contributeur : Cyril Furtlehner <>
Soumis le : jeudi 30 novembre 2017 - 10:58:50
Dernière modification le : jeudi 5 avril 2018 - 12:30:26

Fichier

main.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01652314, version 1

Citation

A Decelle, G Fissore, Cyril Furtlehner. Spectral Dynamics of Learning Restricted Boltzmann Machines. EPL - Europhysics Letters, European Physical Society/EDP Sciences/Società Italiana di Fisica/IOP Publishing, 2017. 〈hal-01652314〉

Partager

Métriques

Consultations de la notice

304

Téléchargements de fichiers

101