High-Dimensional Control Using Generalized Auxiliary Tasks

Yannis Flet-Berliac 1, 2, 3 Philippe Preux 1, 2, 3
1 SEQUEL - Sequential Learning
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189
Abstract : A long-standing challenge in reinforcement learning is the design of function approximations and efficient learning algorithms that provide agents with fast training, robust learning, and high performance in complex environments. To this end, the use of prior knowledge, while promising, is often costly and, in essence, challenging to scale up. In contrast, we consider problem knowledge signals, that are any relevant indicator useful to solve a task, e.g., metrics of uncertainty or proactive prediction of future states. Our framework consists of predicting such complementary quantities associated with self-performance assessment and accurate expectations. Therefore, policy and value functions are no longer only optimized for a reward but are learned using environment-agnostic quantities. We propose a generally applicable framework for structuring reinforcement learning by injecting problem knowledge in policy gradient updates. In this paper: (a) We introduce MERL, our multi-head reinforcement learning framework for generalized auxiliary tasks. (b) We conduct experiments across a variety of standard benchmark environments. Our results show that MERL improves performance for on- and off-policy methods. (c) We show that MERL also improves transfer learning on a set of challenging tasks. (d) We investigate how our approach addresses the problem of reward sparsity and pushes the function approximations into a better-constrained parameter configuration.
Complete list of metadatas

Contributor : Yannis Flet-Berliac <>
Submitted on : Friday, November 29, 2019 - 3:32:42 PM
Last modification on : Saturday, November 30, 2019 - 1:44:11 AM


Files produced by the author(s)


  • HAL Id : hal-02295705, version 3



Yannis Flet-Berliac, Philippe Preux. High-Dimensional Control Using Generalized Auxiliary Tasks. 2019. ⟨hal-02295705v3⟩



Record views


Files downloads