TubeDETR: Spatio-Temporal Video Grounding with Transformers - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

TubeDETR: Spatio-Temporal Video Grounding with Transformers

Résumé

We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query. This is a challenging task that requires the joint and efficient modeling of temporal, spatial and multi-modal interactions. To address this task, we propose TubeDETR, a transformer-based architecture inspired by the recent success of such models for text-conditioned object detection. Our model notably includes: (i) an efficient video and text encoder that models spatial multi-modal interactions over sparsely sampled frames and (ii) a space-time decoder that jointly performs spatio-temporal localization. We demonstrate the advantage of our proposed components through an extensive ablation study. We also evaluate our full approach on the spatio-temporal video grounding task and demonstrate improvements over the state of the art on the challenging VidSTG and HC-STVG benchmarks.
Fichier principal
Vignette du fichier
TubeDETRv2.pdf (4.91 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03625586 , version 1 (31-03-2022)
hal-03625586 , version 2 (09-06-2022)

Identifiants

Citer

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, Cordelia Schmid. TubeDETR: Spatio-Temporal Video Grounding with Transformers. CVPR 2022 - IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2022, New Orleans, United States. ⟨hal-03625586v2⟩
193 Consultations
69 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More