Algorithms for Hyper-Parameter Optimization

James Bergstra 1 R. Bardenet 2, 3 Yoshua Bengio 4 Balázs Kégl 2, 3, 5
3 TAO - Machine Learning and Optimisation
Inria Saclay - Ile de France, UP11 - Université Paris-Sud - Paris 11, CNRS - Centre National de la Recherche Scientifique : UMR8623, LRI - Laboratoire de Recherche en Informatique
Abstract : Several recent advances to the state of the art in image classification benchmarks have come from better configurations of existing techniques rather than novel ap- proaches to feature learning. Traditionally, hyper-parameter optimization has been the job of humans because they can be very efficient in regimes where only a few trials are possible. Presently, computer clusters and GPU processors make it pos- sible to run more trials and we show that algorithmic approaches can find better results. We present hyper-parameter optimization results on tasks of training neu- ral networks and deep belief networks (DBNs). We optimize hyper-parameters using random search and two new greedy sequential methods based on the ex- pected improvement criterion. Random search has been shown to be sufficiently efficient for learning neural networks for several datasets, but we show it is unreli- able for training DBNs. The sequential algorithms are applied to the most difficult DBN learning problems from [1] and find significantly better results than the best previously reported. This work contributes novel techniques for making response surface models P(y|x) in which many elements of hyper-parameter assignment (x) are known to be irrelevant given particular values of other elements.
Type de document :
Communication dans un congrès
J. Shawe-Taylor, R.S. Zemel, P. Bartlett, F. Pereira, K.Q. Weinberger. 25th Annual Conference on Neural Information Processing Systems (NIPS 2011), Dec 2011, Granada, Spain. Neural Information Processing Systems Foundation, 24, 2011, Advances in Neural Information Processing Systems


https://hal.inria.fr/hal-00642998
Contributeur : Balázs Kégl <>
Soumis le : dimanche 20 novembre 2011 - 22:39:52
Dernière modification le : mercredi 20 janvier 2016 - 01:07:20
Document(s) archivé(s) le : vendredi 16 novembre 2012 - 11:31:30

Fichier

draft1.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-00642998, version 1

Collections

Citation

James Bergstra, R. Bardenet, Yoshua Bengio, Balázs Kégl. Algorithms for Hyper-Parameter Optimization. J. Shawe-Taylor, R.S. Zemel, P. Bartlett, F. Pereira, K.Q. Weinberger. 25th Annual Conference on Neural Information Processing Systems (NIPS 2011), Dec 2011, Granada, Spain. Neural Information Processing Systems Foundation, 24, 2011, Advances in Neural Information Processing Systems. <hal-00642998>

Exporter

Partager

Métriques

Consultations de
la notice

715

Téléchargements du document

548