Algorithms for Hyper-Parameter Optimization

James Bergstra 1 R. Bardenet 2, 3 Yoshua Bengio 4 Balázs Kégl 2, 3, 5
3 TAO - Machine Learning and Optimisation
LRI - Laboratoire de Recherche en Informatique, UP11 - Université Paris-Sud - Paris 11, Inria Saclay - Ile de France, CNRS - Centre National de la Recherche Scientifique : UMR8623
Abstract : Several recent advances to the state of the art in image classification benchmarks have come from better configurations of existing techniques rather than novel ap- proaches to feature learning. Traditionally, hyper-parameter optimization has been the job of humans because they can be very efficient in regimes where only a few trials are possible. Presently, computer clusters and GPU processors make it pos- sible to run more trials and we show that algorithmic approaches can find better results. We present hyper-parameter optimization results on tasks of training neu- ral networks and deep belief networks (DBNs). We optimize hyper-parameters using random search and two new greedy sequential methods based on the ex- pected improvement criterion. Random search has been shown to be sufficiently efficient for learning neural networks for several datasets, but we show it is unreli- able for training DBNs. The sequential algorithms are applied to the most difficult DBN learning problems from [1] and find significantly better results than the best previously reported. This work contributes novel techniques for making response surface models P(y|x) in which many elements of hyper-parameter assignment (x) are known to be irrelevant given particular values of other elements.
Type de document :
Communication dans un congrès
J. Shawe-Taylor, R.S. Zemel, P. Bartlett, F. Pereira, K.Q. Weinberger. 25th Annual Conference on Neural Information Processing Systems (NIPS 2011), Dec 2011, Granada, Spain. Neural Information Processing Systems Foundation, 24, 2011, Advances in Neural Information Processing Systems
Liste complète des métadonnées


https://hal.inria.fr/hal-00642998
Contributeur : Balázs Kégl <>
Soumis le : dimanche 20 novembre 2011 - 22:39:52
Dernière modification le : jeudi 9 février 2017 - 15:58:17
Document(s) archivé(s) le : vendredi 16 novembre 2012 - 11:31:30

Fichier

draft1.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-00642998, version 1

Collections

Citation

James Bergstra, R. Bardenet, Yoshua Bengio, Balázs Kégl. Algorithms for Hyper-Parameter Optimization. J. Shawe-Taylor, R.S. Zemel, P. Bartlett, F. Pereira, K.Q. Weinberger. 25th Annual Conference on Neural Information Processing Systems (NIPS 2011), Dec 2011, Granada, Spain. Neural Information Processing Systems Foundation, 24, 2011, Advances in Neural Information Processing Systems. <hal-00642998>

Partager

Métriques

Consultations de
la notice

991

Téléchargements du document

727