A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking - Archive ouverte HAL Access content directly
Journal Articles IEEE MultiMedia Year : 2018

A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking

(1) , (1) , (1)
1

Abstract

With the recent resurgence of neural networks and the proliferation of massive amounts of unlabeled data, unsupervised learning algorithms became very popular for organizing and retrieving large video collections in a task defined as video hyperlinking. Information stored as videos typically contain two modalities, namely an audio and a visual one, that are used conjointly in multimodal systems by undergoing fusion. Multimodal autoencoders have been long used for performing multimodal fusion. In this work, we start by evaluating different initial, single-modal representations for automatic speech transcripts and for video keyframes. We progress to evaluating different autoencoding methods of performing multimodal fusion in an offline setup. The best performing setup is then evaluated in a live setup at TRECVID's 2016 video hyperlinking task. As in offline evaluations, we show that focusing on crossmodal translations as a way of performing multimodal fusion yields improved multimodal representations and that our simple system, trained in an unsupervised manner, with no external information information, defines the new state of the art in a live video hyperlinking setup. We conclude by performing an analysis on data gathered after the live evaluations at TRECVID 2016 and express our thoughts on the overall performance of our proposed system.
Fichier principal
Vignette du fichier
Vukotic - A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking - last revision.pdf (4.19 Mo) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-01848539 , version 1 (24-07-2018)
hal-01848539 , version 2 (20-08-2018)

Identifiers

Cite

Vedran Vukotić, Christian Raymond, Guillaume Gravier. A Crossmodal Approach to Multimodal Fusion in Video Hyperlinking. IEEE MultiMedia, 2018, 25 (2), pp.11-23. ⟨10.1109/MMUL.2018.023121161⟩. ⟨hal-01848539v2⟩
388 View
414 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More