Natural Actor-Critic Algorithms

Shalabh Bhatnagar 1 Richard Sutton 2 Mohammad Ghavamzadeh 3 Mark Lee 2
3 SEQUEL - Sequential Learning
LIFL - Laboratoire d'Informatique Fondamentale de Lille, Inria Lille - Nord Europe, LAGIS - Laboratoire d'Automatique, Génie Informatique et Signal
Abstract : We present four new reinforcement learning algorithms based on actor-critic, function approximation, and natural gradient ideas, and we provide their convergence proofs. Actor-critic reinforcement learning methods are online approximations to policy iteration in which the value function parameters are estimated using temporal difference learning and the policy parameters are updated by stochastic gradient descent. Methods based on policy gradients in this way are of special interest because of their compatibility with function approximation methods, which are needed to handle large or infinite state spaces. The use of temporal difference learning in this way is of special interest because in many applications it dramatically reduces the variance of the gradient estimates. The use of the natural gradient is of interest because it can produce better conditioned parameterizations and has been shown to further reduce variance in some cases. Our results extend prior two-timescale convergence results for actor-critic methods by Konda and Tsitsiklis by using temporal difference learning in the actor and by incorporating natural gradients. Our results extend prior empirical studies of natural actor-critic methods by Peters, Vijayakumar and Schaal by providing the first convergence proofs and the first fully incremental algorithms. We present empirical results verifying the convergence of our algorithms.
Type de document :
Article dans une revue
Automatica, Elsevier, 2009, 45 (11), 〈10.1016/j.automatica.2009.07.008〉
Liste complète des métadonnées

Littérature citée [73 références]  Voir  Masquer  Télécharger
Contributeur : Mohammad Ghavamzadeh <>
Soumis le : mardi 2 juillet 2013 - 15:26:26
Dernière modification le : jeudi 11 janvier 2018 - 06:22:13
Document(s) archivé(s) le : jeudi 3 octobre 2013 - 10:40:06


Fichiers produits par l'(les) auteur(s)




Shalabh Bhatnagar, Richard Sutton, Mohammad Ghavamzadeh, Mark Lee. Natural Actor-Critic Algorithms. Automatica, Elsevier, 2009, 45 (11), 〈10.1016/j.automatica.2009.07.008〉. 〈hal-00840470〉



Consultations de la notice


Téléchargements de fichiers