Modulating early visual processing by language

Abstract : It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic inputs are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the entire visual processing by a linguistic input. Specifically, we introduce Conditional Batch Normalization (CBN) as an efficient mechanism to modulate convolutional feature maps by a linguistic embedding. We apply CBN to a pre-trained Residual Network (ResNet), leading to the MODulatEd ResNet (MODERN) architecture, and show that this significantly improves strong baselines on two visual question answering tasks. Our ablation study confirms that modulating from the early stages of the visual processing is beneficial.
Type de document :
Communication dans un congrès
NIPS 2017 - Conference on Neural Information Processing Systems, Dec 2017, Long Beach, United States. pp.1-14
Liste complète des métadonnées

Littérature citée [28 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01648683
Contributeur : Florian Strub <>
Soumis le : lundi 27 novembre 2017 - 03:06:06
Dernière modification le : mardi 3 juillet 2018 - 11:35:59

Fichier

modulating-early-visual.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01648683, version 1
  • ARXIV : 1707.00683

Citation

Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, et al.. Modulating early visual processing by language. NIPS 2017 - Conference on Neural Information Processing Systems, Dec 2017, Long Beach, United States. pp.1-14. 〈hal-01648683〉

Partager

Métriques

Consultations de la notice

150

Téléchargements de fichiers

42