On Regularization and Robustness of Deep Neural Networks

Alberto Bietti 1 Grégoire Mialon 1 Julien Mairal 1
1 Thoth - Apprentissage de modèles à partir de données massives
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann
Abstract : Despite their success, deep neural networks suffer from several drawbacks: they lack robustness to small changes of input data known as "adversarial examples" and training them with small amounts of annotated data is challenging. In this work, we study the connection between regularization and robustness by viewing neural networks as elements of a reproducing kernel Hilbert space (RKHS) of functions and by regularizing them using the RKHS norm. Even though this norm cannot be computed, we consider various approximations based on upper and lower bounds. These approximations lead to new strategies for regularization, but also to existing ones such as spectral norm penalties or constraints, gradient penalties, or adversarial training. Besides, the kernel framework allows us to obtain margin-based bounds on adversarial generalization. We study the obtained algorithms for learning on small datasets, learning adversarially robust models, and discuss implications for learning implicit generative models.
Type de document :
Pré-publication, Document de travail
Liste complète des métadonnées

Contributeur : Alberto Bietti <>
Soumis le : lundi 1 octobre 2018 - 11:29:21
Dernière modification le : vendredi 5 octobre 2018 - 17:25:57


Fichiers produits par l'(les) auteur(s)


  • HAL Id : hal-01884632, version 1
  • ARXIV : 1810.00363



Alberto Bietti, Grégoire Mialon, Julien Mairal. On Regularization and Robustness of Deep Neural Networks. 2018. 〈hal-01884632〉



Consultations de la notice


Téléchargements de fichiers