Learning Multi-Modal Dictionaries

Gianluca Monaci 1 Philippe Jost 1 Pierre Vandergheynst 1 Boris Mailhé 2 Sylvain Lesage 2 Rémi Gribonval 2
2 METISS - Speech and sound data modeling and processing
IRISA - Institut de Recherche en Informatique et Systèmes Aléatoires, Inria Rennes – Bretagne Atlantique
Abstract : Real-world phenomena involve complex interactions between multiple signal modalities. As a consequence, humans are used to integrate at each instant perceptions from all their senses in order to enrich their understanding of the surrounding world. This paradigm can be also extremely useful in many signal processing and computer vision problems involving mutually related signals. The simultaneous processing of multimodal data can, in fact, reveal information that is otherwise hidden when considering the signals independently. However, in natural multimodal signals, the statistical dependencies between modalities are in general not obvious. Learning fundamental multimodal patterns could offer deep insight into the structure of such signals. In this paper, we present a novel model of multimodal signals based on their sparse decomposition over a dictionary of multimodal structures. An algorithm for iteratively learning multimodal generating functions that can be shifted at all positions in the signal is proposed, as well. The learning is defined in such a way that it can be accomplished by iteratively solving a generalized eigenvector problem, which makes the algorithm fast, flexible, and free of user-defined parameters. The proposed algorithm is applied to audiovisual sequences and it is able to discover underlying structures in the data. The detection of such audio-video patterns in audiovisual clips allows to effectively localize the sound source on the video in presence of substantial acoustic and visual distractors, outperforming state-of-the-art audiovisual localization algorithms.
Type de document :
Article dans une revue
IEEE Transactions on Image Processing, Institute of Electrical and Electronics Engineers, 2007, 16 (9), pp.2272-2283. 〈10.1109/TIP.2007.901813〉
Liste complète des métadonnées

Littérature citée [35 références]  Voir  Masquer  Télécharger

Contributeur : Rémi Gribonval <>
Soumis le : lundi 7 février 2011 - 16:35:27
Dernière modification le : mercredi 11 avril 2018 - 01:57:34
Document(s) archivé(s) le : dimanche 8 mai 2011 - 02:32:50


Fichiers produits par l'(les) auteur(s)



Gianluca Monaci, Philippe Jost, Pierre Vandergheynst, Boris Mailhé, Sylvain Lesage, et al.. Learning Multi-Modal Dictionaries. IEEE Transactions on Image Processing, Institute of Electrical and Electronics Engineers, 2007, 16 (9), pp.2272-2283. 〈10.1109/TIP.2007.901813〉. 〈inria-00544772〉



Consultations de la notice


Téléchargements de fichiers