Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite-Sum Structure

Abstract : Stochastic optimization algorithms with variance reduction have proven successful for minimizing large finite sums of functions. Unfortunately, these techniques are unable to deal with stochastic perturbations of input data, induced for example by data augmentation. In such cases, the objective is no longer a finite sum, and the main candidate for optimization is the stochastic gradient descent method (SGD). In this paper, we introduce a variance reduction approach for these settings when the objective is composite and strongly convex. The convergence rate outperforms SGD with a typically much smaller constant factor, which depends on the variance of gradient estimates only due to perturbations on a single example.
Liste complète des métadonnées


https://hal.inria.fr/hal-01375816
Contributeur : Alberto Bietti <>
Soumis le : jeudi 1 juin 2017 - 11:57:19
Dernière modification le : jeudi 15 juin 2017 - 09:09:01

Fichiers

stoch-miso.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01375816, version 5
  • ARXIV : 1610.00970

Collections

Citation

Alberto Bietti, Julien Mairal. Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite-Sum Structure. 2017. <hal-01375816v5>

Partager

Métriques

Consultations de
la notice

83

Téléchargements du document

22