Skip to Main content Skip to Navigation
Journal articles

A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking

Vedran Vukotić 1 Christian Raymond 1 Guillaume Gravier 1
1 LinkMedia - Creating and exploiting explicit links between multimedia fragments
Inria Rennes – Bretagne Atlantique , IRISA-D6 - MEDIA ET INTERACTIONS
Abstract : With the recent resurgence of neural networks and the proliferation of massive amounts of unlabeled data, unsupervised learning algorithms became very popular for organizing and retrieving large video collections in a task defined as video hyperlinking. Information stored as videos typically contain two modalities, namely an audio and a visual one, that are used conjointly in multimodal systems by undergoing fusion. Multimodal autoencoders have been long used for performing multimodal fusion. In this work, we start by evaluating different initial, single-modal representations for automatic speech transcripts and for video keyframes. We progress to evaluating different autoencoding methods of performing multimodal fusion in an offline setup. The best performing setup is then evaluated in a live setup at TRECVID's 2016 video hyperlinking task. As in offline evaluations, we show that focusing on crossmodal translations as a way of performing multimodal fusion yields improved multimodal representations and that our simple system, trained in an unsupervised manner, with no external information information, defines the new state of the art in a live video hyperlinking setup. We conclude by performing an analysis on data gathered after the live evaluations at TRECVID 2016 and express our thoughts on the overall performance of our proposed system.
Complete list of metadata

Cited literature [20 references]  Display  Hide  Download
Contributor : Guillaume Gravier Connect in order to contact the contributor
Submitted on : Monday, August 20, 2018 - 5:09:50 PM
Last modification on : Wednesday, November 3, 2021 - 8:08:01 AM
Long-term archiving on: : Wednesday, November 21, 2018 - 2:04:58 PM


Vukotic - A Crossmodal Approac...
Files produced by the author(s)



Vedran Vukotić, Christian Raymond, Guillaume Gravier. A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking. IEEE MultiMedia, Institute of Electrical and Electronics Engineers, 2018, 25 (2), pp.11-23. ⟨10.1109/MMUL.2018.023121161⟩. ⟨hal-01848539v2⟩



Les métriques sont temporairement indisponibles