Information Geometry of Gaussian Distributions in View of Stochastic Optimization

Luigi Malagò 1, 2 Giovanni Pistone 3
2 TAO - Machine Learning and Optimisation
LRI - Laboratoire de Recherche en Informatique, UP11 - Université Paris-Sud - Paris 11, Inria Saclay - Ile de France, CNRS - Centre National de la Recherche Scientifique : UMR8623
Abstract : We study the optimization of a continuous function by its stochastic relaxation, i.e., the optimization of the expected value of the function itself with respect to a density in a statistical model. We focus on gradient descent techniques applied to models from the exponential family and in particular on the multivariate Gaussian distribution. From the theory of the exponential family, we reparametrize the Gaussian distribution using natural and expectation parameters, and we derive formulas for natural gradients in both parameterizations. We discuss some advantages of the natural parameterization for the identification of sub-models in the Gaussian distribution based on conditional independence assumptions among variables. Gaussian distributions are widely used in stochastic optimization and in particular in model-based Evolutionary Computation, as in Estimation of Distribution Algorithms and Evolutionary Strategies. By studying natural gradient flows over Gaussian distributions our analysis and results directly apply to the study of CMA-ES and NES algorithms
Type de document :
Communication dans un congrès
Foundations of Genetic Algorithms XIII, Jan 2015, Aberystwyth, United Kingdom. ACM, pp.150-162, 2015, 〈http://foga2015.dcs.aber.ac.uk/schedule.html〉
Liste complète des métadonnées

https://hal.inria.fr/hal-01108986
Contributeur : Luigi Malagò <>
Soumis le : vendredi 23 janvier 2015 - 18:13:15
Dernière modification le : jeudi 5 avril 2018 - 12:30:12

Identifiants

  • HAL Id : hal-01108986, version 1

Citation

Luigi Malagò, Giovanni Pistone. Information Geometry of Gaussian Distributions in View of Stochastic Optimization. Foundations of Genetic Algorithms XIII, Jan 2015, Aberystwyth, United Kingdom. ACM, pp.150-162, 2015, 〈http://foga2015.dcs.aber.ac.uk/schedule.html〉. 〈hal-01108986〉

Partager

Métriques

Consultations de la notice

346