Local Policy Search in a Convex Space and Conservative Policy Iteration as Boosted Policy Search

Bruno Scherrer 1 Matthieu Geist 2
1 MAIA - Autonomous intelligent machine
Inria Nancy - Grand Est, LORIA - AIS - Department of Complex Systems, Artificial Intelligence & Robotics
Abstract : Local Policy Search is a popular reinforcement learning approach for handling large state spaces. Formally, it searches locally in a parameterized policy space in order to maximize the associated value function averaged over some pre-defined distribution. The best one can hope in general from such an approach is to get a local optimum of this criterion. The first contribution of this article is the following surprising result: if the policy space is convex, any (approximate) local optimum enjoys a global performance guarantee. Unfortunately, the convexity assumption is strong: it is not satisfied by commonly used parameterizations and designing a parameterization that induces this property seems hard. A natural so-lution to alleviate this issue consists in deriving an algorithm that solves the local policy search problem using a boosting approach (constrained to the convex hull of the policy space). The resulting algorithm turns out to be a slight generalization of conservative policy iteration; thus, our second contribution is to highlight an original connection between local policy search and approximate dynamic pro-gramming.
Liste complète des métadonnées

Littérature citée [25 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01091079
Contributeur : Bruno Scherrer <>
Soumis le : lundi 8 décembre 2014 - 15:30:55
Dernière modification le : jeudi 5 avril 2018 - 12:30:11
Document(s) archivé(s) le : lundi 9 mars 2015 - 10:10:49

Fichiers

lps_guarantee_ecml14_final.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

Citation

Bruno Scherrer, Matthieu Geist. Local Policy Search in a Convex Space and Conservative Policy Iteration as Boosted Policy Search. ECML, Sep 2014, Nancy, France. pp.35 - 50, 2014, 〈10.1007/978-3-662-44845-8_3〉. 〈hal-01091079〉

Partager

Métriques

Consultations de la notice

383

Téléchargements de fichiers

289