Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning - Archive ouverte HAL Access content directly
Conference Papers Year :

Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning

(1) , (2, 3) , (4, 5) , (1) , (1, 3) , (1)
1
2
3
4
5
Nino Vieillard
  • Function : Author
  • PersonId : 1090625
Olivier Pietquin
  • Function : Author
  • PersonId : 1090627
Rémi Munos
  • Function : Author
  • PersonId : 1090628
Matthieu Geist
  • Function : Author
  • PersonId : 1090629

Abstract

Recent Reinforcement Learning (RL) algorithms making use of Kullback-Leibler (KL) regularization as a core component have shown outstanding performance. Yet, only little is understood theoretically about why KL regularization helps, so far. We study KL regularization within an approximate value iteration scheme and show that it implicitly averages q-values. Leveraging this insight, we provide a very strong performance bound, the very first to combine two desirable aspects: a linear dependency to the horizon (instead of quadratic) and an error propagation term involving an averaging effect of the estimation errors (instead of an accumulation effect). We also study the more general case of an additional entropy regularizer. The resulting abstract scheme encompasses many existing RL algorithms. Some of our assumptions do not hold with neural networks, so we complement this theoretical analysis with an extensive empirical study.
Fichier principal
Vignette du fichier
NeurIPS-2020-leverage-the-average-an-analysis-of-kl-regularization-in-reinforcement-learning-Supplemental.pdf (4.25 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03137351 , version 1 (10-02-2021)

Identifiers

  • HAL Id : hal-03137351 , version 1

Cite

Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, et al.. Leverage the Average: an Analysis of KL Regularization in Reinforcement Learning. NeurIPS - 34th Conference on Neural Information Processing Systems, Dec 2020, Vancouver / Online, Canada. ⟨hal-03137351⟩
179 View
252 Download

Share

Gmail Facebook Twitter LinkedIn More