Anti Imitation-Based Policy Learning

Michèle Sebag 1, 2 Riad Akrour 1, 2 Basile Mayeur 1, 2 Marc Schoenauer 1, 2
2 TAO - Machine Learning and Optimisation
LRI - Laboratoire de Recherche en Informatique, UP11 - Université Paris-Sud - Paris 11, Inria Saclay - Ile de France, CNRS - Centre National de la Recherche Scientifique : UMR8623
Abstract : The Anti Imitation-based Policy Learning (AIPoL) approach, taking inspiration from the Energy-based learning framework (LeCun et al. 2006), aims at a pseudo-value function such that it induces the same order on the state space as a (nearly optimal) value function. By construction , the greedification of such a pseudo-value induces the same policy as the value function itself. The approach assumes that, thanks to prior knowledge, not-to-be-imitated demonstrations can easily be generated. For instance, applying a random policy on a good initial state (e.g., a bicycle in equilibrium) will on average lead to visit states with decreasing values (the bicycle ultimately falls down). Such a demonstration , that is, a sequence of states with decreasing values, is used along a standard learning-to-rank approach to define a pseudo-value function. If the model of the environment is known, this pseudo-value directly induces a policy by greedification. Otherwise, the bad demonstrations are exploited together with off-policy learning to learn a pseudo-Q-value function and likewise thence derive a policy by greedification. To our best knowledge the use of bad demonstrations to achieve policy learning is original. The theoretical analysis shows that the loss of optimality of the pseudo value-based policy is bounded under mild assumptions, and the empirical validation of AIPoL on the mountain car, the bicycle and the swing-up pendulum problems demonstrates the simplicity and the merits of the approach.
Document type :
Conference papers
Complete list of metadatas

https://hal.inria.fr/hal-01387652
Contributor : Marc Schoenauer <>
Submitted on : Wednesday, October 26, 2016 - 6:41:57 PM
Last modification on : Thursday, April 5, 2018 - 12:30:12 PM

Files

mainDIVA.pdf
Files produced by the author(s)

Identifiers

Citation

Michèle Sebag, Riad Akrour, Basile Mayeur, Marc Schoenauer. Anti Imitation-Based Policy Learning. Machine Learning and Knowledge Discovery in Databases - European Conference, ECML-PKDD 2016, Sep 2016, Riva del Garda, Afghanistan. pp.559 - 575, ⟨10.1007/978-3-319-46227-1_35⟩. ⟨hal-01387652⟩

Share

Metrics

Record views

476

Files downloads

376