HAL will be down for maintenance from Friday, June 10 at 4pm through Monday, June 13 at 9am. More information
Skip to Main content Skip to Navigation
Conference papers

What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

Marcin Andrychowicz 1 Anton Raichuk 1 Piotr Stańczyk 1 Manu Orsini 1 Sertan Girgin 1 Raphaël Marinier 1 Léonard Hussenot 1, 2 Matthieu Geist 1 Olivier Pietquin 1 Marcin Michalski 1 Sylvain Gelly 1 Olivier Bachem 1
2 Scool - Scool
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189
Abstract : In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20]. As a step towards filling that gap, we implement >50 such ``choices'' in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.
Document type :
Conference papers
Complete list of metadata

Contributor : Léonard Hussenot Connect in order to contact the contributor
Submitted on : Monday, March 8, 2021 - 3:43:53 PM
Last modification on : Thursday, March 24, 2022 - 3:42:40 AM
Long-term archiving on: : Wednesday, June 9, 2021 - 7:15:06 PM


Files produced by the author(s)


  • HAL Id : hal-03162554, version 1
  • ARXIV : 2006.05990


Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, et al.. What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study. ICLR 2021 - Ninth International Conference on Learning Representations, May 2021, Vienna / Virtual, Austria. ⟨hal-03162554⟩



Record views


Files downloads