Skip to Main content Skip to Navigation
Conference papers

Inferring DQN structure for high-dimensional continuous control

Abstract : Despite recent advancements in the field of Deep Reinforcement Learning, Deep Q-network (DQN) models still show lackluster performance on problems with high-dimensional action spaces. The problem is even more pronounced for cases with high-dimensional continuous action spaces due to combinatorial increase in the number of the outputs. Recent works approach the problem by dividing the network into multiple parallel or sequential (action) modules responsible for different discretized actions. However there are drawbacks to both the parallel and the sequential approaches, i.e. parallel module architectures lack coordination between action modules, leading to extra complexity in the task, while a sequential structure can result in the vanishing gradients problem and exploding parameter space. In this work we show that the compositional structure of the action modules has a significant impact on the model performance, we propose a novel approach to infer the network structure for DQN models operating with high-dimensional continuous actions. Our method is based on uncertainty estimation techniques and yields substantially higher scores for MuJoCo environments with high-dimensional continuous action spaces, as well as a realistic AAA sailing simulator game.
Complete list of metadatas

Cited literature [25 references]  Display  Hide  Download

https://hal.inria.fr/hal-02984307
Contributor : Chedy Raïssi <>
Submitted on : Friday, October 30, 2020 - 6:43:00 PM
Last modification on : Tuesday, November 3, 2020 - 3:30:15 AM

File

6024-Paper.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02984307, version 1

Collections

Citation

Andrey Sakryukin, Chedy Raïssi, Mohan Kankanhalli. Inferring DQN structure for high-dimensional continuous control. 2020 International Conference on Machine Learning, Jul 2020, Vienna, Austria. ⟨hal-02984307⟩

Share

Metrics

Record views

21

Files downloads

91