MERL: Multi-Head Reinforcement Learning - Archive ouverte HAL Access content directly
Conference Papers Year :

(1, 2, 3) , (1, 2, 3)
1
2
3
Yannis Flet-Berliac
Philippe Preux

#### Abstract

A common challenge in reinforcement learning is how to convert the agent's interactions with an environment into fast and robust learning. For instance, earlier work makes use of domain knowledge to improve existing reinforcement learning algorithms in complex tasks. While promising, previously acquired knowledge is often costly and challenging to scale up. Instead, we decide to consider problem knowledge with signals from quantities relevant to solve any task, e.g., self-performance assessment and accurate expectations. $\mathcal{V}^{ex}$ is such a quantity. It is the fraction of variance explained by the value function $V$ and measures the discrepancy between $V$ and the returns. Taking advantage of $\mathcal{V}^{ex}$, we propose MERL, a general framework for structuring reinforcement learning by injecting problem knowledge into policy gradient updates. As a result, the agent is not only optimized for a reward but learns using problem-focused quantities provided by MERL, applicable out-of-the-box to any task. In this paper: (a) We introduce and define MERL, the multi-head reinforcement learning framework we use throughout this work. (b) We conduct experiments across a variety of standard benchmark environments, including 9 continuous control tasks, where results show improved performance. (c) We demonstrate that MERL also improves transfer learning on a set of challenging pixel-based tasks. (d) We ponder how MERL tackles the problem of reward sparsity and better conditions the feature space of reinforcement learning agents.

### Dates and versions

hal-02305105 , version 1 (03-10-2019)
hal-02305105 , version 2 (13-10-2019)
hal-02305105 , version 3 (29-11-2019)

### Identifiers

• HAL Id : hal-02305105 , version 3
• ARXIV :

### Cite

Yannis Flet-Berliac, Philippe Preux. MERL: Multi-Head Reinforcement Learning. Deep Reinforcement Learning Workshop, NeurIPS, Dec 2019, Vancouver, Canada. ⟨hal-02305105v3⟩

### Export

BibTeX TEI Dublin Core DC Terms EndNote Datacite

250 View