Skip to Main content Skip to Navigation
New interface
Conference papers

Multimodal space representation driven by self-evaluation of predictability

Abstract : PROPRE is a generic and modular neural learning paradigm that autonomously extracts meaningful concepts of multimodal data flows driven by predictability across modalities in an unsupervised, incremental and online way. For that purpose, PROPRE consists of the combination of projection and prediction. Firstly, each data flow is topologically projected with a self-organizing map, largely inspired from the Kohonen model. Secondly, each projection is predicted by each other map activities, by mean of linear regressions. The main originality of PROPRE is the use of a simple and generic predictability measure that compares predicted and real activities for each modal stream. This measure drives the corresponding projection learning to favor the mapping of predictable stimuli across modalities at the system level (i.e. that their predictability measure overcomes some threshold). This predictability measure acts as a self-evaluation module that tends to bias the representations extracted by the system so that to improve their correlations across modalities. We already showed that this modulation mechanism is able to bootstrap representation extraction from previously learned representations with artificial multimodal data related to basic robotic behaviors and improves performance of the system for classification of visual data within a supervised learning context. In this article, we improve the self-evaluation module of PROPRE, by introducing a sliding threshold, and apply it to the unsupervised classification of gestures caught from two time- of-flight (ToF) cameras. In this context, we illustrate that the modulation mechanism is still useful although less efficient than purely supervised learning.
Complete list of metadata

Cited literature [28 references]  Display  Hide  Download
Contributor : Mathieu Lefort Connect in order to contact the contributor
Submitted on : Friday, September 12, 2014 - 11:45:42 AM
Last modification on : Wednesday, May 11, 2022 - 3:20:02 PM
Long-term archiving on: : Saturday, December 13, 2014 - 10:10:44 AM


Files produced by the author(s)


  • HAL Id : hal-01061668, version 1



Mathieu Lefort, Thomas Kopinski, Alexander Gepperth. Multimodal space representation driven by self-evaluation of predictability. Joint IEEE International Conference Developmental Learning and Epigenetic Robotics (ICDL-EPIROB), Oct 2014, Gênes, Italy. ⟨hal-01061668⟩



Record views


Files downloads