The Spectral Bias of Polynomial Neural Networks - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

The Spectral Bias of Polynomial Neural Networks

Résumé

Polynomial neural networks (PNNs) have been recently shown to be particularly effective at image generation and face recognition, where high-frequency information is critical. Previous studies have revealed that neural networks demonstrate a spectral bias towards low-frequency functions, which yields faster learning of low-frequency components during training. Inspired by such studies, we conduct a spectral analysis of the Neural Tangent Kernel (NTK) of PNNs. We find that the Π-Net family, i.e., a recently proposed parametrization of PNNs, speeds up the learning of the higher frequencies. We verify the theoretical bias through extensive experiments. We expect our analysis to provide novel insights into designing architectures and learning frameworks by incorporating multiplicative interactions via polynomials.
Fichier principal
Vignette du fichier
3346_the_spectral_bias_of_polynomia.pdf (2.58 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03878338 , version 1 (29-11-2022)

Identifiants

  • HAL Id : hal-03878338 , version 1

Citer

Moulik Choraria, Leello Dadi, Grigorios G. Chrysos, Julien Mairal, Volkan Cevher. The Spectral Bias of Polynomial Neural Networks. ICLR 2022 - International Conference on Learning Representations, Apr 2022, Virtual, France. pp.1-30. ⟨hal-03878338⟩
59 Consultations
141 Téléchargements

Partager

Gmail Facebook X LinkedIn More