Learning Interactions and Relationships between Movie Characters - Archive ouverte HAL Access content directly
Conference Papers Year :

Learning Interactions and Relationships between Movie Characters

(1) , (2) , (2)


Interactions between people are often governed by their relationships. On the flip side, social relationships are built upon several interactions. Two strangers are more likely to greet and introduce themselves while becoming friends over time. We are fascinated by this interplay between interactions and relationships, and believe that it is an important aspect of understanding social situations. In this work, we propose neural models to learn and jointly predict interactions, relationships, and the pair of characters that are involved. We note that interactions are informed by a mixture of visual and dialog cues, and present a multimodal architecture to extract meaningful information from them. Localizing the pair of interacting characters in video is a time-consuming process, instead, we train our model to learn from clip-level weak labels. We evaluate our models on the MovieGraphs dataset and show the impact of modalities, use of longer temporal context for predicting relationships, and achieve encouraging performance using weak labels as compared with ground-truth labels. Code is online.
Fichier principal
Vignette du fichier
2003.13158.pdf (4.13 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03017606 , version 1 (21-11-2020)



Anna Kukleva, Makarand Tapaswi, Ivan Laptev. Learning Interactions and Relationships between Movie Characters. CVPR 2020- IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2020, Virtual, United States. ⟨10.1109/CVPR42600.2020.00987⟩. ⟨hal-03017606⟩
23 View
86 Download



Gmail Facebook Twitter LinkedIn More