Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

On Regularization and Robustness of Deep Neural Networks

Alberto Bietti 1 Grégoire Mialon 1 Julien Mairal 1
1 Thoth [2016-2019] - Apprentissage de modèles à partir de données massives [2016-2019]
Inria Grenoble - Rhône-Alpes, LJK [2016-2019] - Laboratoire Jean Kuntzmann [2016-2019]
Abstract : In this work, we study the connection between regularization and robustness of deep neural networks by viewing them as elements of a reproducing kernel Hilbert space (RKHS) of functions and by regularizing them using the RKHS norm. Even though this norm cannot be computed, we consider various approximations based on upper and lower bounds. These approximations lead to new strategies for regularization, but also to existing ones such as spectral norm penalties or constraints, gradient penalties, or adversarial training. Besides, the kernel framework allows us to obtain margin-based bounds on adversarial generalization. We show that our new algorithms lead to empirical benefits for learning on small datasets and learning adversarially robust models. We also discuss implications of our regularization framework for learning implicit generative models.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas

https://hal.inria.fr/hal-01884632
Contributor : Alberto Bietti <>
Submitted on : Friday, November 30, 2018 - 5:28:01 PM
Last modification on : Friday, July 17, 2020 - 11:40:04 AM
Document(s) archivé(s) le : Friday, March 1, 2019 - 3:50:56 PM

File

main.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01884632, version 2
  • ARXIV : 1810.00363

Citation

Alberto Bietti, Grégoire Mialon, Julien Mairal. On Regularization and Robustness of Deep Neural Networks. 2018. ⟨hal-01884632v2⟩

Share

Metrics

Record views

132

Files downloads

559