Multimodal space representation driven by self-evaluation of predictability

Mathieu Lefort 1, 2 Thomas Kopinski 3 Alexander Gepperth 1, 2
1 Flowers - Flowing Epigenetic Robots and Systems
Inria Bordeaux - Sud-Ouest, U2IS - Unité d'Informatique et d'Ingénierie des Systèmes
Abstract : PROPRE is a generic and modular neural learning paradigm that autonomously extracts meaningful concepts of multimodal data flows driven by predictability across modalities in an unsupervised, incremental and online way. For that purpose, PROPRE consists of the combination of projection and prediction. Firstly, each data flow is topologically projected with a self-organizing map, largely inspired from the Kohonen model. Secondly, each projection is predicted by each other map activities, by mean of linear regressions. The main originality of PROPRE is the use of a simple and generic predictability measure that compares predicted and real activities for each modal stream. This measure drives the corresponding projection learning to favor the mapping of predictable stimuli across modalities at the system level (i.e. that their predictability measure overcomes some threshold). This predictability measure acts as a self-evaluation module that tends to bias the representations extracted by the system so that to improve their correlations across modalities. We already showed that this modulation mechanism is able to bootstrap representation extraction from previously learned representations with artificial multimodal data related to basic robotic behaviors and improves performance of the system for classification of visual data within a supervised learning context. In this article, we improve the self-evaluation module of PROPRE, by introducing a sliding threshold, and apply it to the unsupervised classification of gestures caught from two time- of-flight (ToF) cameras. In this context, we illustrate that the modulation mechanism is still useful although less efficient than purely supervised learning.
Type de document :
Communication dans un congrès
Joint IEEE International Conference Developmental Learning and Epigenetic Robotics (ICDL-EPIROB), Oct 2014, Gênes, Italy. 2014
Liste complète des métadonnées

Littérature citée [28 références]  Voir  Masquer  Télécharger
Contributeur : Mathieu Lefort <>
Soumis le : vendredi 12 septembre 2014 - 11:45:42
Dernière modification le : jeudi 12 avril 2018 - 18:27:32
Document(s) archivé(s) le : samedi 13 décembre 2014 - 10:10:44


Fichiers produits par l'(les) auteur(s)


  • HAL Id : hal-01061668, version 1



Mathieu Lefort, Thomas Kopinski, Alexander Gepperth. Multimodal space representation driven by self-evaluation of predictability. Joint IEEE International Conference Developmental Learning and Epigenetic Robotics (ICDL-EPIROB), Oct 2014, Gênes, Italy. 2014. 〈hal-01061668〉



Consultations de la notice


Téléchargements de fichiers