Minimizing Finite Sums with the Stochastic Average Gradient

Mark Schmidt 1, 2 Nicolas Le Roux 2, 1 Francis Bach 2, 1
1 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique de l'École normale supérieure, ENS Paris - École normale supérieure - Paris, CNRS - Centre National de la Recherche Scientifique, Inria de Paris
Abstract : We propose the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG method achieves a faster convergence rate than black-box SG methods. The convergence rate is improved from O(1/k^{1/2}) to O(1/k) in general, and when the sum is strongly-convex the convergence rate is improved from the sub-linear O(1/k) to a linear convergence rate of the form O(p^k) for p < 1. Further, in many cases the convergence rate of the new method is also faster than black-box deterministic gradient methods, in terms of the number of gradient evaluations. Numerical experiments indicate that the new algorithm often dramatically outperforms existing SG and deterministic gradient methods, and that the performance may be further improved through the use of non-uniform sampling strategies.
Type de document :
Pré-publication, Document de travail
Revision from January 2015 submission. Major changes: updated literature follow and discussion of.. 2016
Liste complète des métadonnées

https://hal.inria.fr/hal-00860051
Contributeur : Mark Schmidt <>
Soumis le : mardi 10 mai 2016 - 22:28:36
Dernière modification le : jeudi 11 janvier 2018 - 06:28:04

Fichiers

sagMP.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-00860051, version 2
  • ARXIV : 1309.2388

Collections

Citation

Mark Schmidt, Nicolas Le Roux, Francis Bach. Minimizing Finite Sums with the Stochastic Average Gradient. Revision from January 2015 submission. Major changes: updated literature follow and discussion of.. 2016. 〈hal-00860051v2〉

Partager

Métriques

Consultations de la notice

431

Téléchargements de fichiers

2634