Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite-Sum Structure - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2017

Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite-Sum Structure

Résumé

Stochastic optimization algorithms with variance reduction have proven successful for minimizing large finite sums of functions. Unfortunately, these techniques are unable to deal with stochastic perturbations of input data, induced for example by data augmentation. In such cases, the objective is no longer a finite sum, and the main candidate for optimization is the stochastic gradient descent method (SGD). In this paper, we introduce a variance reduction approach for these settings when the objective is composite and strongly convex. The convergence rate outperforms SGD with a typically much smaller constant factor, which depends on the variance of gradient estimates only due to perturbations on a single example.
Fichier principal
Vignette du fichier
stoch-miso-nips.pdf (580.18 Ko) Télécharger le fichier

Dates et versions

hal-01375816 , version 1 (03-10-2016)
hal-01375816 , version 2 (10-01-2017)
hal-01375816 , version 3 (23-01-2017)
hal-01375816 , version 4 (27-02-2017)
hal-01375816 , version 5 (01-06-2017)
hal-01375816 , version 6 (15-11-2017)

Identifiants

Citer

Alberto Bietti, Julien Mairal. Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite-Sum Structure. NIPS 2017 - Advances in Neural Information Processing Systems, Dec 2017, Long Beach, CA, United States. pp.1622-1632. ⟨hal-01375816v6⟩
1180 Consultations
753 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More