A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking

Vedran Vukotić 1 Christian Raymond 1 Guillaume Gravier 1
1 LinkMedia - Creating and exploiting explicit links between multimedia fragments
Inria Rennes – Bretagne Atlantique , IRISA_D6 - MEDIA ET INTERACTIONS
Abstract : With the recent resurgence of neural networks and the proliferation of massive amounts of unlabeled data, unsupervised learning algorithms became very popular for organizing and retrieving large video collections in a task defined as video hyperlinking. Information stored as videos typically contain two modalities, namely an audio and a visual one, that are used conjointly in multimodal systems by undergoing fusion. Multimodal autoencoders have been long used for performing multimodal fusion. In this work, we start by evaluating different initial, single-modal representations for automatic speech transcripts and for video keyframes. We progress to evaluating different autoencoding methods of performing multimodal fusion in an offline setup. The best performing setup is then evaluated in a live setup at TRECVID's 2016 video hyperlinking task. As in offline evaluations, we show that focusing on crossmodal translations as a way of performing multimodal fusion yields improved multimodal representations and that our simple system, trained in an unsupervised manner, with no external information information, defines the new state of the art in a live video hyperlinking setup. We conclude by performing an analysis on data gathered after the live evaluations at TRECVID 2016 and express our thoughts on the overall performance of our proposed system.
Type de document :
Article dans une revue
IEEE MultiMedia, Institute of Electrical and Electronics Engineers, In press, 25 (2), pp.11-23. 〈10.1109/MMUL.2018.023121161〉
Liste complète des métadonnées

https://hal.inria.fr/hal-01848539
Contributeur : Guillaume Gravier <>
Soumis le : lundi 20 août 2018 - 17:09:50
Dernière modification le : jeudi 15 novembre 2018 - 11:59:01
Document(s) archivé(s) le : mercredi 21 novembre 2018 - 14:04:58

Fichier

Vukotic - A Crossmodal Approac...
Fichiers produits par l'(les) auteur(s)

Identifiants

Citation

Vedran Vukotić, Christian Raymond, Guillaume Gravier. A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking. IEEE MultiMedia, Institute of Electrical and Electronics Engineers, In press, 25 (2), pp.11-23. 〈10.1109/MMUL.2018.023121161〉. 〈hal-01848539v2〉

Partager

Métriques

Consultations de la notice

333

Téléchargements de fichiers

90