Hyperparameter optimization with approximate gradient

Abstract : Most models in machine learning contain at least one hyperparameter to control for model complexity. Choosing an appropriate set of hyperparameters is both crucial in terms of model accuracy and computationally challenging. In this work we propose an algorithm for the optimization of continuous hyperparameters using inexact gradient information. An advantage of this method is that hyperparameters can be updated before model parameters have fully converged. We also give sufficient conditions for the global convergence of this method, based on regularity conditions of the involved functions and summability of errors. Finally, we validate the empirical performance of this method on the estimation of regularization constants of L2-regularized logistic regression and kernel Ridge regression. Empirical benchmarks indicate that our approach is highly competitive with respect to state of the art methods.
Type de document :
Communication dans un congrès
Proceedings of the 33rd International Conference on Machine Learning, Jun 2016, New York, United States. 2016, 〈http://icml.cc〉
Liste complète des métadonnées

https://hal.inria.fr/hal-01386410
Contributeur : Fabian Pedregosa <>
Soumis le : lundi 24 octobre 2016 - 10:22:33
Dernière modification le : jeudi 11 janvier 2018 - 06:12:20

Lien texte intégral

Identifiants

  • HAL Id : hal-01386410, version 1
  • ARXIV : 1602.02355

Collections

Citation

Fabian Pedregosa. Hyperparameter optimization with approximate gradient. Proceedings of the 33rd International Conference on Machine Learning, Jun 2016, New York, United States. 2016, 〈http://icml.cc〉. 〈hal-01386410〉

Partager

Métriques

Consultations de la notice

73