Data-driven calibration of penalties for least-squares regression

Sylvain Arlot 1, 2, * Pascal Massart 1, 2
* Auteur correspondant
2 SELECT - Model selection in statistical learning
Inria Saclay - Ile de France, LMO - Laboratoire de Mathématiques d'Orsay, CNRS - Centre National de la Recherche Scientifique : UMR
Abstract : Penalization procedures often suffer from their dependence on multiplying factors, whose optimal values are either unknown or hard to estimate from the data. In this paper, we propose a completely data-driven calibration method for this parameter in the least-squares regression framework, without assuming a particular shape for the penalty. Our algorithm relies on the concept of minimal penalty, which has been introduced in a recent paper by Birg\'{e} and Massart (2007) in the context of penalized least squares for Gaussian homoscedastic regression. Interestingly, the minimal penalty can be evaluated from the data themselves, which leads to a data-driven estimation of an optimal penalty that one can use in practice. Unfortunately their approach heavily relies on the homoscedastic Gaussian nature of the stochastic framework that they consider. Our purpose in this paper is twofold: stating a more general heuristics to design a data-driven penalty (the slope heuristics) and proving that it works for penalized least squares random design regression, even when the data is heteroscedastic. For some technical reasons which are explained in the paper, we could prove some precise mathematical results only for histogram bin-width selection. Even though we could not work at the level of generality that we were expecting, this is at least a first step towards further results. Our mathematical results hold in some specific framework, but the approach and the method that we use are indeed general.
Type de document :
[Research Report] RR-6556, INRIA. 2008, pp.40
Liste complète des métadonnées

Littérature citée [38 références]  Voir  Masquer  Télécharger
Contributeur : Rapport de Recherche Inria <>
Soumis le : lundi 16 juin 2008 - 12:22:49
Dernière modification le : jeudi 11 janvier 2018 - 06:22:14
Document(s) archivé(s) le : vendredi 25 novembre 2016 - 22:32:02


Fichiers produits par l'(les) auteur(s)


  • HAL Id : inria-00287631, version 2



Sylvain Arlot, Pascal Massart. Data-driven calibration of penalties for least-squares regression. [Research Report] RR-6556, INRIA. 2008, pp.40. 〈inria-00287631v2〉



Consultations de la notice


Téléchargements de fichiers