Samples are not all useful: Denoising policy gradient updates using variance

Yannis Flet-Berliac 1, 2, 3 Philippe Preux 1, 2, 3
1 SEQUEL - Sequential Learning
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189
Abstract : Policy gradient algorithms in reinforcement learning rely on efficiently sampling an environment. Most sampling procedures are based solely on sampling the agent's policy. However, other measures made available through these algorithms could be used in order to improve the sampling prior to each policy update. Following this line of thoughts, we propose a method where a transition is used in the gradient update if it meets a particular criterion, and rejected otherwise. This criterion is the fraction of variance explained ($\mathcal{V}^{ex}$), a measure of the discrepancy between a model and actual samples. $\mathcal{V}^{ex}$ can be used to evaluate the impact each transition will have on the learning. This criterion refines sampling and improves the policy gradient algorithm. In this paper: (1) We introduce and explore $\mathcal{V}^{ex}$, the selection criterion used to improve the sampling procedure. (2) We conduct experiments across a variety of standard benchmark environments, including continuous control problems. Our results show better performance than if we did not use the $\mathcal{V}^{ex}$ criterion for the policy gradient update. (3) We investigate why $\mathcal{V}^{ex}$ gives a good evaluation for the selection of samples that will positively impact the learning. (4) We show how this criterion can be interpreted as a dynamic way to adjust the ratio between exploration and exploitation.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas
Contributor : Yannis Flet-Berliac <>
Submitted on : Wednesday, April 10, 2019 - 12:56:26 PM
Last modification on : Tuesday, April 16, 2019 - 1:31:38 AM


Files produced by the author(s)


  • HAL Id : hal-02091547, version 2



Yannis Flet-Berliac, Philippe Preux. Samples are not all useful: Denoising policy gradient updates using variance. 2019. ⟨hal-02091547v2⟩



Record views


Files downloads