Policy Search: Any Local Optimum Enjoys a Global Performance Guarantee

Bruno Scherrer 1 Matthieu Geist 2
1 MAIA - Autonomous intelligent machine
Inria Nancy - Grand Est, LORIA - AIS - Department of Complex Systems, Artificial Intelligence & Robotics
2 MALIS - MAchine Learning and Interactive Systems
SUPELEC-Campus Metz, CentraleSupélec
Abstract : Local Policy Search is a popular reinforcement learning approach for handling large state spaces. Formally, it searches locally in a paramet erized policy space in order to maximize the associated value function averaged over some predefined distribution. It is probably commonly b elieved that the best one can hope in general from such an approach is to get a local optimum of this criterion. In this article, we show th e following surprising result: \emph{any} (approximate) \emph{local optimum} enjoys a \emph{global performance guarantee}. We compare this g uarantee with the one that is satisfied by Direct Policy Iteration, an approximate dynamic programming algorithm that does some form of Poli cy Search: if the approximation error of Local Policy Search may generally be bigger (because local search requires to consider a space of s tochastic policies), we argue that the concentrability coefficient that appears in the performance bound is much nicer. Finally, we discuss several practical and theoretical consequences of our analysis.
Liste complète des métadonnées

Littérature citée [20 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-00829548
Contributeur : Bruno Scherrer <>
Soumis le : jeudi 6 juin 2013 - 15:56:37
Dernière modification le : jeudi 5 avril 2018 - 12:30:24
Document(s) archivé(s) le : samedi 7 septembre 2013 - 03:05:08

Fichiers

report.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-00829548, version 1
  • ARXIV : 1306.1520

Citation

Bruno Scherrer, Matthieu Geist. Policy Search: Any Local Optimum Enjoys a Global Performance Guarantee. 2013. 〈hal-00829548〉

Partager

Métriques

Consultations de la notice

412

Téléchargements de fichiers

151