Skip to Main content Skip to Navigation
Conference papers

A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images

Melissa Ailem 1, 2 Bowen Zhang 1 Aurélien Bellet 2 Pascal Denis 2 Fei Sha 1
2 MAGNET - Machine Learning in Information Networks
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189
Abstract : Several recent studies have shown the benefits of combining language and perception to infer word embeddings. These multimodal approaches either simply combine pre-trained textual and visual representations (e.g. features extracted from convolutional neural networks), or use the latter to bias the learning of textual word embeddings. In this work, we propose a novel probabilistic model to formalize how linguistic and perceptual inputs can work in concert to explain the observed word-context pairs in a text corpus. Our approach learns textual and visual representations jointly: latent visual factors couple together a skip-gram model for co-occurrence in linguistic data and a generative latent variable model for visual data. Extensive experimental studies validate the proposed model. Concretely, on the tasks of assessing pairwise word similarity and image/caption retrieval, our approach attains equally competitive or stronger results when compared to other state-of-the-art multimodal models.
Complete list of metadata

Cited literature [39 references]  Display  Hide  Download
Contributor : Aurélien Bellet Connect in order to contact the contributor
Submitted on : Wednesday, November 14, 2018 - 7:08:22 PM
Last modification on : Friday, January 21, 2022 - 3:10:30 AM
Long-term archiving on: : Friday, February 15, 2019 - 4:40:00 PM


Files produced by the author(s)


  • HAL Id : hal-01922985, version 1


Melissa Ailem, Bowen Zhang, Aurélien Bellet, Pascal Denis, Fei Sha. A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images. Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), 2018, Brussels, Belgium. ⟨hal-01922985⟩



Les métriques sont temporairement indisponibles