Blind Audiovisual Source Separation Based on Sparse Redundant Representations

Anna Llagostera Casanovas 1 Monaci Gianluca 1, 2 Pierre Vandergheynst 1 Rémi Gribonval 3
3 METISS - Speech and sound data modeling and processing
IRISA - Institut de Recherche en Informatique et Systèmes Aléatoires, Inria Rennes – Bretagne Atlantique
Abstract : In this paper, we propose a novel method which is able to detect and separate audiovisual sources present in a scene. Our method exploits the correlation between the video signal captured with a camera and a synchronously recorded one-microphone audio track. In a first stage, audio and video modalities are decomposed into relevant basic structures using redundant representations. Next, synchrony between relevant events in audio and video modalities is quantified. Based on this co-occurrence measure, audiovisual sources are counted and located in the image using a robust clustering algorithm that groups video structures exhibiting strong correlations with the audio. Next periods where each source is active alone are determined and used to build spectral Gaussian mixture models (GMMs) characterizing the sources acoustic behavior. Finally, these models are used to separate the audio signal in periods during which several sources are mixed. The proposed approach has been extensively tested on synthetic and natural sequences composed of speakers and music instruments. Results show that the proposed method is able to successfully detect, localize, separate, and reconstruct present audiovisual sources.
Complete list of metadatas

Cited literature [29 references]  Display  Hide  Download
Contributor : Rémi Gribonval <>
Submitted on : Thursday, January 27, 2011 - 9:58:42 PM
Last modification on : Thursday, February 7, 2019 - 5:54:59 PM
Long-term archiving on : Thursday, April 28, 2011 - 2:27:58 AM


Files produced by the author(s)



Anna Llagostera Casanovas, Monaci Gianluca, Pierre Vandergheynst, Rémi Gribonval. Blind Audiovisual Source Separation Based on Sparse Redundant Representations. IEEE Transactions on Multimedia, Institute of Electrical and Electronics Engineers, 2010, 12 (5), pp.358 -- 371. ⟨10.1109/TMM.2010.2050650⟩. ⟨inria-00541412⟩



Record views


Files downloads