Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Is attention to bounding boxes all you need for pedestrian action prediction?

Lina Achaji 1 Julien Moreau 1 Thibault Fouqueray 1 Francois Aioun 1 François Charpillet 2 
2 LARSEN - Lifelong Autonomy and interaction skills for Robots in a Sensing ENvironment
Inria Nancy - Grand Est, LORIA - AIS - Department of Complex Systems, Artificial Intelligence & Robotics
Abstract : The human driver is no longer the only one concerned with the complexity of the driving scenarios. Autonomous vehicles (AV) are similarly becoming involved in the process. Nowadays, the development of AV in urban places underpins essential safety concerns for vulnerable road users (VRUs) such as pedestrians. Therefore, to make the roads safer, it is critical to classify and predict their future behavior. In this paper, we present a framework based on multiple variations of the Transformer models to reason attentively about the dynamic evolution of the pedestrians' past trajectory and predict its future actions of crossing or not crossing the street. We proved that using only bounding-boxes as input to our model can outperform the previous state-of-the-art models and reach a prediction accuracy of 91% and an F1-score of 0.83 on the PIE dataset up to two seconds ahead in the future. In addition, we introduced a large-size simulated dataset (CP2A) using CARLA for action prediction. Our model has similarly reached high accuracy (91 %) and F1-score (0.91) on this dataset. Interestingly, we showed that pre-training our Transformer model on the simulated dataset and then fine-tuning it on the real dataset can be very effective for the action prediction task.
Document type :
Preprints, Working Papers, ...
Complete list of metadata
Contributor : lina Achaji Connect in order to contact the contributor
Submitted on : Friday, July 16, 2021 - 3:56:03 PM
Last modification on : Thursday, March 31, 2022 - 4:12:24 AM
Long-term archiving on: : Sunday, October 17, 2021 - 7:05:14 PM


Files produced by the author(s)


  • HAL Id : hal-03288986, version 1



Lina Achaji, Julien Moreau, Thibault Fouqueray, Francois Aioun, François Charpillet. Is attention to bounding boxes all you need for pedestrian action prediction?. 2021. ⟨hal-03288986⟩



Record views


Files downloads