Boosted Bellman Residual Minimization Handling Expert Demonstrations - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2014

Boosted Bellman Residual Minimization Handling Expert Demonstrations

Résumé

This paper addresses the problem of batch Reinforcement Learning with Expert Demonstrations (RLED). In RLED, the goal is to find an optimal policy of a Markov Decision Process (MDP), using a data set of fixed sampled transitions of the MDP as well as a data set of fixed expert demonstrations. This is slightly different from the batch Reinforcement Learning (RL) framework where only fixed sampled transitions of the MDP are available. Thus, the aim of this article is to propose algorithms that leverage those expert data. The idea proposed here differs from the Approximate Dynamic Programming methods in the sense that we minimize the Optimal Bellman Residual (OBR), where the minimization is guided by constraints defined by the expert demonstrations. This choice is motivated by the the fact that controlling the OBR implies controlling the distance between the estimated and optimal quality functions. However, this method presents some difficulties as the criterion to minimize is non-convex, non-differentiable and biased. Those difficulties are overcome via the embedding of distributions in a Reproducing Kernel Hilbert Space (RKHS) and a boosting technique which allows obtaining non-parametric algorithms. Finally, our algorithms are compared to the only state of the art algorithm, Approximate Policy Iteration with Demonstrations (APID) algorithm, in different experimental settings.
Fichier principal
Vignette du fichier
supelec885.pdf (481.12 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01060953 , version 1 (04-09-2014)

Identifiants

Citer

Bilal Piot, Matthieu Geist, Olivier Pietquin. Boosted Bellman Residual Minimization Handling Expert Demonstrations. European Conference, ECML PKDD 2014, Sep 2014, Nancy, France. pp.549-564, ⟨10.1007/978-3-662-44851-9_35⟩. ⟨hal-01060953⟩
528 Consultations
742 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More