Learning Sparse Features with an Auto-Associator

Sébastien Rebecchi 1 Hélène Paugam-Moisy 2 Michèle Sebag 3, 1
1 TAO - Machine Learning and Optimisation
CNRS - Centre National de la Recherche Scientifique : UMR8623, Inria Saclay - Ile de France, UP11 - Université Paris-Sud - Paris 11, LRI - Laboratoire de Recherche en Informatique
2 DM2L - Data Mining and Machine Learning
LIRIS - Laboratoire d'InfoRmatique en Image et Systèmes d'information
Abstract : A major issue in statistical machine learning is the design of a representa-tion, or feature space, facilitating the resolution of the learning task at hand. Sparse representations in particular facilitate discriminant learning: On the one hand, they are robust to noise. On the other hand, they disentangle the factors of variation mixed up in dense representations, favoring the separa-bility and interpretation of data. This chapter focuses on auto-associators (AAs), i.e. multi-layer neural networks trained to encode/decode the data and thus de facto defining a feature space. AAs, first investigated in the 80s, were recently reconsidered as building blocks for deep neural networks. This chapter surveys related work about building sparse representations, and presents a new non-linear explicit sparse representation method referred to as Sparse Auto-Associator (SAA), integrating a sparsity objective within the standard auto-associator learning criterion. The comparative empirical val-idation of SAAs on state-of-art handwritten digit recognition benchmarks shows that SAAs outperform standard auto-associators in terms of classifi-cation performance and yield similar results as denoising auto-associators. Furthermore, SAAs enable to control the representation size to some extent, through a conservative pruning of the feature space.
Document type :
Book sections
Complete list of metadatas

Cited literature [32 references]  Display  Hide  Download

https://hal.inria.fr/hal-01109773
Contributor : Marc Schoenauer <>
Submitted on : Monday, January 26, 2015 - 11:52:00 PM
Last modification on : Wednesday, November 20, 2019 - 2:57:53 AM
Long-term archiving on: Saturday, September 12, 2015 - 6:35:09 AM

File

chapterSAA-Paugam.pdf
Files produced by the author(s)

Identifiers

Citation

Sébastien Rebecchi, Hélène Paugam-Moisy, Michèle Sebag. Learning Sparse Features with an Auto-Associator. Kowaliw, Taras and Bredeche, Nicolas and Doursat, René. Growing Adaptive Machines, 557, Springer Verlag, pp.139 - 158, 2014, Studies in Computational Intelligence, ⟨10.1007/978-3-642-55337-0_4⟩. ⟨hal-01109773⟩

Share

Metrics

Record views

550

Files downloads

468