Deep invariant networks with differentiable augmentation layers
Résumé
Designing learning systems which are invariant to certain data transformations is critical in machine learning. Practitioners can typically enforce a desired invariance on the trained model through the choice of a network architecture, e.g. using convolutions for translations, or using data augmentation. Yet, enforcing true invariance in the network can be difficult, and data invariances are not always known a piori. State-of-the-art methods for learning data augmentation policies require held-out data and are based on bilevel optimization problems, which are complex to solve and often computationally demanding. In this work we investigate new ways of learning invariances only from the training data. Using learnable augmentation layers built directly in the network, we demonstrate that our method is very versatile. It can incorporate any type of differentiable augmentation and be applied to a broad class of learning problems beyond computer vision. We provide empirical evidence showing that our approach is easier and faster to train than modern automatic data augmentation techniques based on bilevel optimization, while achieving comparable results. Experiments show that while the invariances transferred to a model through automatic data augmentation are limited by the model expressivity, the invariance yielded by our approach is insensitive to it by design.
Fichier principal
main.pdf (1.39 Mo)
Télécharger le fichier
ablation-weights-mags-heatmap.pdf (24.11 Ko)
Télécharger le fichier
architecture.pdf (78.12 Ko)
Télécharger le fichier
augnet-augerino-and-ablation.pdf (123.2 Ko)
Télécharger le fichier
benchmark_mass_with_adda.pdf (16.6 Ko)
Télécharger le fichier
cifar10-neurips-submission.pdf (26.31 Ko)
Télécharger le fichier
exp-illustration.pdf (39.35 Ko)
Télécharger le fichier
four-aug-layers.pdf (35.57 Ko)
Télécharger le fichier
learned_shift.pdf (10.33 Ko)
Télécharger le fichier
reg_illustration_color.pdf (8.16 Ko)
Télécharger le fichier
sin_capacity_study.pdf (13.98 Ko)
Télécharger le fichier
sin_dataset_illustration.pdf (37.32 Ko)
Télécharger le fichier
sin_learned_shift.pdf (11.89 Ko)
Télécharger le fichier
sin_params_heatmap.pdf (20.59 Ko)
Télécharger le fichier
sin_training_plot.pdf (14.21 Ko)
Télécharger le fichier
two-aug-layers.pdf (25.49 Ko)
Télécharger le fichier
weight_mags_seed1.pdf (20.58 Ko)
Télécharger le fichier
weight_mags_seed29.pdf (18.21 Ko)
Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Origine : Fichiers produits par l'(les) auteur(s)