Skip to Main content Skip to Navigation
Journal articles

Approximation spaces of deep neural networks

Rémi Gribonval 1, 2 Gitta Kutyniok 3 Morten Nielsen 4 Felix Voigtlaender 5
1 PANAMA - Parcimonie et Nouveaux Algorithmes pour le Signal et la Modélisation Audio
Inria Rennes – Bretagne Atlantique , IRISA-D5 - SIGNAUX ET IMAGES NUMÉRIQUES, ROBOTIQUE
2 DANTE - Dynamic Networks : Temporal and Structural Capture Approach
Inria Grenoble - Rhône-Alpes, LIP - Laboratoire de l'Informatique du Parallélisme, IXXI - Institut Rhône-Alpin des systèmes complexes
Abstract : We study the expressivity of deep neural networks. Measuring a network's complexity by its number of connections or by its number of neurons, we consider the class of functions for which the error of best approximation with networks of a given complexity decays at a certain rate when increasing the complexity budget. Using results from classical approximation theory, we show that this class can be endowed with a (quasi)-norm that makes it a linear function space, called approximation space. We establish that allowing the networks to have certain types of "skip connections" does not change the resulting approximation spaces. We also discuss the role of the network's nonlinearity (also known as activation function) on the resulting spaces, as well as the role of depth. For the popular ReLU nonlinearity and its powers, we relate the newly constructed spaces to classical Besov spaces. The established embeddings highlight that some functions of very low Besov smoothness can nevertheless be well approximated by neural networks, if these networks are sufficiently deep.
Complete list of metadata

https://hal.inria.fr/hal-02117139
Contributor : Rémi Gribonval <>
Submitted on : Friday, July 10, 2020 - 11:58:18 AM
Last modification on : Friday, June 25, 2021 - 3:40:06 PM

Files

main.pdf
Files produced by the author(s)

Identifiers

Citation

Rémi Gribonval, Gitta Kutyniok, Morten Nielsen, Felix Voigtlaender. Approximation spaces of deep neural networks. Constructive Approximation, Springer Verlag, 2021, special issue on "Deep Networks in Approximation Theory", ⟨10.1007/s00365-021-09543-4⟩. ⟨hal-02117139v3⟩

Share

Metrics

Record views

394

Files downloads

301