A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images

Melissa Ailem 1, 2 Bowen Zhang 1 Aurélien Bellet 2 Pascal Denis 2 Fei Sha 1
2 MAGNET - Machine Learning in Information Networks
CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189, Inria Lille - Nord Europe
Abstract : Several recent studies have shown the benefits of combining language and perception to infer word embeddings. These multimodal approaches either simply combine pre-trained textual and visual representations (e.g. features extracted from convolutional neural networks), or use the latter to bias the learning of textual word embeddings. In this work, we propose a novel probabilistic model to formalize how linguistic and perceptual inputs can work in concert to explain the observed word-context pairs in a text corpus. Our approach learns textual and visual representations jointly: latent visual factors couple together a skip-gram model for co-occurrence in linguistic data and a generative latent variable model for visual data. Extensive experimental studies validate the proposed model. Concretely, on the tasks of assessing pairwise word similarity and image/caption retrieval, our approach attains equally competitive or stronger results when compared to other state-of-the-art multimodal models.
Complete list of metadatas

Cited literature [39 references]  Display  Hide  Download

https://hal.inria.fr/hal-01922985
Contributor : Aurélien Bellet <>
Submitted on : Wednesday, November 14, 2018 - 7:08:22 PM
Last modification on : Tuesday, September 10, 2019 - 11:32:02 AM
Long-term archiving on : Friday, February 15, 2019 - 4:40:00 PM

File

emnlp18.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01922985, version 1

Citation

Melissa Ailem, Bowen Zhang, Aurélien Bellet, Pascal Denis, Fei Sha. A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images. Conference on Empirical Methods in Natural Language Processing (EMNLP 2018), 2018, Brussels, Belgium. ⟨hal-01922985⟩

Share

Metrics

Record views

111

Files downloads

101