Towards Weakly-Supervised Action Localization

Philippe Weinzaepfel 1 Xavier Martin 1 Cordelia Schmid 1
1 Thoth - Apprentissage de modèles à partir de données massives
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann
Abstract : This paper presents a novel approach for weakly-supervised action localization, i.e., that does not require per-frame spatial annotations for training. We first introduce an effective method for extracting human tubes by combining a state-of-the-art human detector with a tracking-by-detection approach. Our tube extraction leverages the large amount of annotated humans available today and outperforms the state of the art by an order of magnitude: with less than 5 tubes per video, we obtain a recall of 95% on the UCF-Sports and J-HMDB datasets. Given these human tubes, we perform weakly-supervised selection based on multi-fold Multiple Instance Learning (MIL) with improved dense trajectories and achieve excellent results. We obtain a mAP of 84% on UCF-Sports, 54% on J-HMDB and 45% on UCF-101, which outperforms the state of the art for weakly-supervised action localization and is close to the performance of the best fully-supervised approaches. The second contribution of this paper is a new realistic dataset for action localization, named DALY (Daily Action Localization in YouTube). It contains high quality temporal and spatial annotations for 10 actions in 31 hours of videos (3.3M frames), which is an order of magnitude larger than standard action localization datasets. On the DALY dataset, our tubes have a spatial recall of 82%, but the detection task is extremely challenging, we obtain 10.8% mAP.
Type de document :
Pré-publication, Document de travail
Liste complète des métadonnées
Contributeur : Thoth Team <>
Soumis le : mercredi 18 mai 2016 - 15:18:44
Dernière modification le : mardi 26 juillet 2016 - 15:49:45
Document(s) archivé(s) le : jeudi 17 novembre 2016 - 13:26:16


Fichiers produits par l'(les) auteur(s)


  • HAL Id : hal-01317558, version 1
  • ARXIV : 1605.05197



Philippe Weinzaepfel, Xavier Martin, Cordelia Schmid. Towards Weakly-Supervised Action Localization. 2016. <hal-01317558>



Consultations de
la notice


Téléchargements du document