Actor and Observer: Joint Modeling of First and Third-Person Videos

Abstract : Several theories in cognitive neuroscience suggest that when people interact with the world, or simulate interactions, they do so from a first-person egocentric perspective, and seamlessly transfer knowledge between third-person (observer) and first-person (actor). Despite this, learning such models for human action recognition has not been achievable due to the lack of data. This paper takes a step in this direction, with the introduction of Charades-Ego, a large-scale dataset of paired first-person and third-person videos, involving 112 people, with 4000 paired videos. This enables learning the link between the two, actor and observer perspectives. Thereby, we address one of the biggest bottlenecks facing egocentric vision research, providing a link from first-person to the abundant third-person data on the web. We use this data to learn a joint representation of first and third-person videos, with only weak supervision, and show its effectiveness for transferring knowledge from the third-person to the first-person domain.
Type de document :
Communication dans un congrès
CVPR 2018 - IEEE Conference on Computer Vision & Pattern Recognition, Jun 2018, Salt Lake City, Utah, United States. IEEE, pp.1-6, 2018
Liste complète des métadonnées

Littérature citée [44 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01755547
Contributeur : Karteek Alahari <>
Soumis le : vendredi 30 mars 2018 - 16:35:00
Dernière modification le : mardi 30 octobre 2018 - 09:42:34

Fichier

Sigurdsson18.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01755547, version 1
  • ARXIV : 1804.09627

Collections

Citation

Gunnar Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, Karteek Alahari. Actor and Observer: Joint Modeling of First and Third-Person Videos. CVPR 2018 - IEEE Conference on Computer Vision & Pattern Recognition, Jun 2018, Salt Lake City, Utah, United States. IEEE, pp.1-6, 2018. 〈hal-01755547〉

Partager

Métriques

Consultations de la notice

610

Téléchargements de fichiers

641