Skip to Main content Skip to Navigation
Conference papers

On the almost sure convergence of stochastic gradient descent in non-convex problems

Abstract : This paper analyzes the trajectories of stochastic gradient descent (SGD) to help understand the algorithm's convergence properties in non-convex problems. We first show that the sequence of iterates generated by SGD remains bounded and converges with probability 1 under a very broad range of step-size schedules. Subsequently, going beyond existing positive probability guarantees, we show that SGD avoids strict saddle points/manifolds with probability 1 for the entire spectrum of step-size policies considered. Finally, we prove that the algorithm's rate of convergence to Hurwicz minimizers is O(1/n p) if the method is employed with a Θ(1/n p) step-size. This provides an important guideline for tuning the algorithm's step-size as it suggests that a cool-down phase with a vanishing step-size could lead to faster convergence; we demonstrate this heuristic using ResNet architectures on CIFAR.
Document type :
Conference papers
Complete list of metadatas

https://hal.inria.fr/hal-03043771
Contributor : Panayotis Mertikopoulos <>
Submitted on : Monday, December 7, 2020 - 2:19:44 PM
Last modification on : Wednesday, December 16, 2020 - 4:08:42 AM

File

NonConvexSGD-NIPS.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03043771, version 1

Citation

Panayotis Mertikopoulos, Nadav Hallak, Ali Kavis, Volkan Cevher. On the almost sure convergence of stochastic gradient descent in non-convex problems. NeurIPS '20: The 34th International Conference on Neural Information Processing Systems, 2020, Vancouver, Canada. ⟨hal-03043771⟩

Share

Metrics

Record views

14

Files downloads

46