Towards Weakly-Supervised Action Localization

Philippe Weinzaepfel 1 Xavier Martin 1 Cordelia Schmid 1
1 Thoth - Apprentissage de modèles à partir de données massives
LJK - Laboratoire Jean Kuntzmann, Inria Grenoble - Rhône-Alpes
Abstract : This paper presents a novel approach for weakly-supervised action localization, i.e., that does not require per-frame spatial annotations for training. We first introduce an effective method for extracting human tubes by combining a state-of-the-art human detector with a tracking-by-detection approach. Our tube extraction leverages the large amount of annotated humans available today and outperforms the state of the art by an order of magnitude: with less than 5 tubes per video, we obtain a recall of 95% on the UCF-Sports and J-HMDB datasets. Given these human tubes, we perform weakly-supervised selection based on multi-fold Multiple Instance Learning (MIL) with improved dense trajectories and achieve excellent results. We obtain a mAP of 84% on UCF-Sports, 54% on J-HMDB and 45% on UCF-101, which outperforms the state of the art for weakly-supervised action localization and is close to the performance of the best fully-supervised approaches. The second contribution of this paper is a new realistic dataset for action localization, named DALY (Daily Action Localization in YouTube). It contains high quality temporal and spatial annotations for 10 actions in 31 hours of videos (3.3M frames), which is an order of magnitude larger than standard action localization datasets. On the DALY dataset, our tubes have a spatial recall of 82%, but the detection task is extremely challenging, we obtain 10.8% mAP.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas

Cited literature [41 references]  Display  Hide  Download
Contributor : Thoth Team <>
Submitted on : Wednesday, May 18, 2016 - 3:18:44 PM
Last modification on : Monday, April 30, 2018 - 3:02:01 PM
Long-term archiving on: Thursday, November 17, 2016 - 1:26:16 PM


Files produced by the author(s)


  • HAL Id : hal-01317558, version 1
  • ARXIV : 1605.05197


Philippe Weinzaepfel, Xavier Martin, Cordelia Schmid. Towards Weakly-Supervised Action Localization. 2016. ⟨hal-01317558v1⟩



Record views


Files downloads