Skip to Main content Skip to Navigation
Conference papers

Anti Imitation-Based Policy Learning

Michèle Sebag 1, 2 Riad Akrour 1, 2 Basile Mayeur 1, 2 Marc Schoenauer 1, 2
2 TAO - Machine Learning and Optimisation
CNRS - Centre National de la Recherche Scientifique : UMR8623, Inria Saclay - Ile de France, UP11 - Université Paris-Sud - Paris 11, LRI - Laboratoire de Recherche en Informatique
Abstract : The Anti Imitation-based Policy Learning (AIPoL) approach, taking inspiration from the Energy-based learning framework (LeCun et al. 2006), aims at a pseudo-value function such that it induces the same order on the state space as a (nearly optimal) value function. By construction , the greedification of such a pseudo-value induces the same policy as the value function itself. The approach assumes that, thanks to prior knowledge, not-to-be-imitated demonstrations can easily be generated. For instance, applying a random policy on a good initial state (e.g., a bicycle in equilibrium) will on average lead to visit states with decreasing values (the bicycle ultimately falls down). Such a demonstration , that is, a sequence of states with decreasing values, is used along a standard learning-to-rank approach to define a pseudo-value function. If the model of the environment is known, this pseudo-value directly induces a policy by greedification. Otherwise, the bad demonstrations are exploited together with off-policy learning to learn a pseudo-Q-value function and likewise thence derive a policy by greedification. To our best knowledge the use of bad demonstrations to achieve policy learning is original. The theoretical analysis shows that the loss of optimality of the pseudo value-based policy is bounded under mild assumptions, and the empirical validation of AIPoL on the mountain car, the bicycle and the swing-up pendulum problems demonstrates the simplicity and the merits of the approach.
Document type :
Conference papers
Complete list of metadatas
Contributor : Marc Schoenauer <>
Submitted on : Wednesday, October 26, 2016 - 6:41:57 PM
Last modification on : Friday, November 20, 2020 - 4:20:21 PM


Files produced by the author(s)




Michèle Sebag, Riad Akrour, Basile Mayeur, Marc Schoenauer. Anti Imitation-Based Policy Learning. Machine Learning and Knowledge Discovery in Databases - European Conference, ECML-PKDD 2016, Sep 2016, Riva del Garda, Afghanistan. pp.559 - 575, ⟨10.1007/978-3-319-46227-1_35⟩. ⟨hal-01387652⟩



Record views


Files downloads