Skip to Main content Skip to Navigation
Conference papers

Learning Interactions and Relationships between Movie Characters

Abstract : Interactions between people are often governed by their relationships. On the flip side, social relationships are built upon several interactions. Two strangers are more likely to greet and introduce themselves while becoming friends over time. We are fascinated by this interplay between interactions and relationships, and believe that it is an important aspect of understanding social situations. In this work, we propose neural models to learn and jointly predict interactions, relationships, and the pair of characters that are involved. We note that interactions are informed by a mixture of visual and dialog cues, and present a multimodal architecture to extract meaningful information from them. Localizing the pair of interacting characters in video is a time-consuming process, instead, we train our model to learn from clip-level weak labels. We evaluate our models on the MovieGraphs dataset and show the impact of modalities, use of longer temporal context for predicting relationships, and achieve encouraging performance using weak labels as compared with ground-truth labels. Code is online.
Document type :
Conference papers
Complete list of metadatas

https://hal.inria.fr/hal-03017606
Contributor : Makarand Tapaswi <>
Submitted on : Saturday, November 21, 2020 - 5:40:09 AM
Last modification on : Thursday, January 14, 2021 - 3:14:12 PM

File

2003.13158.pdf
Files produced by the author(s)

Identifiers

Collections

Citation

Anna Kukleva, Makarand Tapaswi, Ivan Laptev. Learning Interactions and Relationships between Movie Characters. CVPR 2020- IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2020, Virtual, United States. ⟨10.1109/CVPR42600.2020.00987⟩. ⟨hal-03017606⟩

Share

Metrics

Record views

17

Files downloads

98