| HAL-Inria | Publications, software ... of Inria's scientists |
Conference papers
Stochastic Variance-Reduced Policy Gradient
Abstract : In this paper, we propose a novel reinforcement-learning algorithm consisting in a stochastic variance-reduced version of policy gradient for solving Markov Decision Processes (MDPs). Stochastic variance-reduced gradient (SVRG) methods have proven to be very successful in supervised learning. However, their adaptation to policy gradient is not straightforward and needs to account for I) a non-concave objective function; II) approximations in the full gradient computation; and III) a non-stationary sampling process. The result is SVRPG, a stochastic variance-reduced policy gradient algorithm that leverages on importance weights to preserve the unbiasedness of the gradient estimate. Under standard assumptions on the MDP, we provide convergence guarantees for SVRPG with a convergence rate that is linear under increasing batch sizes. Finally, we suggest practical variants of SVRPG, and we empirically evaluate them on continuous MDPs.
https://hal.inria.fr/hal-01940394
Contributor : Matteo Pirotta Connect in order to contact the contributor
Submitted on : Friday, November 30, 2018 - 11:31:35 AM
Last modification on : Tuesday, January 4, 2022 - 5:32:48 AM
Long-term archiving on: : Friday, March 1, 2019 - 2:18:15 PM
Contributor : Matteo Pirotta Connect in order to contact the contributor
Submitted on : Friday, November 30, 2018 - 11:31:35 AM
Last modification on : Tuesday, January 4, 2022 - 5:32:48 AM
Long-term archiving on: : Friday, March 1, 2019 - 2:18:15 PM
File
supplementary.pdf
Files produced by the author(s)