Anti Imitation-Based Policy Learning

Michèle Sebag 1, 2 Riad Akrour 1, 2 Basile Mayeur 1, 2 Marc Schoenauer 1, 2
2 TAO - Machine Learning and Optimisation
CNRS - Centre National de la Recherche Scientifique : UMR8623, Inria Saclay - Ile de France, UP11 - Université Paris-Sud - Paris 11, LRI - Laboratoire de Recherche en Informatique
Abstract : The Anti Imitation-based Policy Learning (AIPoL) approach, taking inspiration from the Energy-based learning framework (LeCun et al. 2006), aims at a pseudo-value function such that it induces the same order on the state space as a (nearly optimal) value function. By construction , the greedification of such a pseudo-value induces the same policy as the value function itself. The approach assumes that, thanks to prior knowledge, not-to-be-imitated demonstrations can easily be generated. For instance, applying a random policy on a good initial state (e.g., a bicycle in equilibrium) will on average lead to visit states with decreasing values (the bicycle ultimately falls down). Such a demonstration , that is, a sequence of states with decreasing values, is used along a standard learning-to-rank approach to define a pseudo-value function. If the model of the environment is known, this pseudo-value directly induces a policy by greedification. Otherwise, the bad demonstrations are exploited together with off-policy learning to learn a pseudo-Q-value function and likewise thence derive a policy by greedification. To our best knowledge the use of bad demonstrations to achieve policy learning is original. The theoretical analysis shows that the loss of optimality of the pseudo value-based policy is bounded under mild assumptions, and the empirical validation of AIPoL on the mountain car, the bicycle and the swing-up pendulum problems demonstrates the simplicity and the merits of the approach.
Type de document :
Communication dans un congrès
Paolo Frasconi ; Niels Landwehr; Giuseppe Manco; Jilles Vreeken. Machine Learning and Knowledge Discovery in Databases - European Conference, ECML-PKDD 2016, Sep 2016, Riva del Garda, Afghanistan. pp.559 - 575, 2016, LNCS. 〈10.1007/978-3-319-46227-1_35〉
Liste complète des métadonnées

https://hal.inria.fr/hal-01387652
Contributeur : Marc Schoenauer <>
Soumis le : mercredi 26 octobre 2016 - 18:41:57
Dernière modification le : jeudi 11 janvier 2018 - 06:22:14

Fichiers

mainDIVA.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

Citation

Michèle Sebag, Riad Akrour, Basile Mayeur, Marc Schoenauer. Anti Imitation-Based Policy Learning. Paolo Frasconi ; Niels Landwehr; Giuseppe Manco; Jilles Vreeken. Machine Learning and Knowledge Discovery in Databases - European Conference, ECML-PKDD 2016, Sep 2016, Riva del Garda, Afghanistan. pp.559 - 575, 2016, LNCS. 〈10.1007/978-3-319-46227-1_35〉. 〈hal-01387652〉

Partager

Métriques

Consultations de la notice

340

Téléchargements de fichiers

169