SGD Algorithms based on Incomplete U-statistics: Large-Scale Minimization of Empirical Risk

Guillaume Papa 1 Stéphan Clémençon 1 Aurélien Bellet 2
2 MAGNET - Machine Learning in Information Networks
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189
Abstract : In many learning problems, ranging from clustering to ranking through metric learning, empirical estimates of the risk functional consist of an average over tuples (e.g., pairs or triplets) of observations, rather than over individual observations. In this paper, we focus on how to best implement a stochastic approximation approach to solve such risk minimization problems. We argue that in the large-scale setting, gradient estimates should be obtained by sampling tuples of data points with replacement (incomplete U-statistics) instead of sampling data points without replacement (complete U-statistics based on subsamples). We develop a theoretical framework accounting for the substantial impact of this strategy on the generalization ability of the prediction model returned by the Stochastic Gradient Descent (SGD) algorithm. It reveals that the method we promote achieves a much better trade-off between statistical accuracy and computational cost. Beyond the rate bound analysis, experiments on AUC maximization and metric learning provide strong empirical evidence of the superiority of the proposed approach.
Type de document :
Communication dans un congrès
Annual Conference on Neural Information Processing Systems (NIPS), Dec 2015, Montréal, Canada. 〈https://nips.cc/Conferences/2015〉
Liste complète des métadonnées

Littérature citée [26 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01214667
Contributeur : Aurélien Bellet <>
Soumis le : jeudi 5 novembre 2015 - 13:38:31
Dernière modification le : jeudi 11 janvier 2018 - 06:27:32
Document(s) archivé(s) le : vendredi 28 avril 2017 - 05:02:13

Fichiers

nips15_sgd.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01214667, version 1

Citation

Guillaume Papa, Stéphan Clémençon, Aurélien Bellet. SGD Algorithms based on Incomplete U-statistics: Large-Scale Minimization of Empirical Risk. Annual Conference on Neural Information Processing Systems (NIPS), Dec 2015, Montréal, Canada. 〈https://nips.cc/Conferences/2015〉. 〈hal-01214667〉

Partager

Métriques

Consultations de la notice

334

Téléchargements de fichiers

176