On the Inductive Bias of Neural Tangent Kernels

Alberto Bietti 1 Julien Mairal 1
1 Thoth - Apprentissage de modèles à partir de données massives
LJK - Laboratoire Jean Kuntzmann, Inria Grenoble - Rhône-Alpes
Abstract : State-of-the-art neural networks are heavily over-parameterized, making the optimization algorithm a crucial ingredient for learning predictive models with good generalization properties. A recent line of work has shown that in a certain over-parameterized regime, the learning dynamics of gradient descent are governed by a certain kernel obtained at initialization, called the neural tangent kernel. We study the inductive bias of learning in such a regime by analyzing this kernel and the corresponding function space (RKHS). In particular, we study smoothness, approximation, and stability properties of functions with finite norm, including stability to image deformations in the case of convolutional networks.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas

Cited literature [20 references]  Display  Hide  Download

https://hal.inria.fr/hal-02144221
Contributor : Alberto Bietti <>
Submitted on : Wednesday, May 29, 2019 - 10:50:49 PM
Last modification on : Thursday, June 6, 2019 - 12:32:27 AM

File

main.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02144221, version 1
  • ARXIV : 1905.12173

Collections

Citation

Alberto Bietti, Julien Mairal. On the Inductive Bias of Neural Tangent Kernels. 2019. ⟨hal-02144221⟩

Share

Metrics

Record views

41

Files downloads

293