Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view

Bruno Scherrer 1
1 MAIA - Autonomous intelligent machine
INRIA Lorraine, LORIA - Laboratoire Lorrain de Recherche en Informatique et ses Applications
Abstract : We investigate projection methods, for evaluating a linear approximation of the value function of a policy in a Markov Decision Process context. We consider two popular approaches, the one-step Temporal Difference fix-point computation (TD(0)) and the Bellman Residual (BR) minimization. We describe examples, where each method outperforms the other. We highlight a simple relation between the objective function they minimize, and show that while BR enjoys a performance guarantee, TD(0) does not in general. We then propose a unified view in terms of oblique projections of the Bellman equation, which substantially simplifies and extends the characterization of (schoknecht,2002) and the recent analysis of (Yu & Bertsekas, 2008). Eventually, we describe some simulations that suggest that if the TD(0) solution is usually slightly better than the BR solution, its inherent numerical instability makes it very bad in some cases, and thus worse on average.
Type de document :
Communication dans un congrès
27th International Conference on Machine Learning - ICML 2010, Jun 2010, Haïfa, Israel. 2010
Liste complète des métadonnées

Littérature citée [9 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/inria-00537403
Contributeur : Bruno Scherrer <>
Soumis le : vendredi 19 novembre 2010 - 08:48:28
Dernière modification le : jeudi 11 janvier 2018 - 06:19:51
Document(s) archivé(s) le : jeudi 30 juin 2011 - 13:37:27

Fichiers

Identifiants

  • HAL Id : inria-00537403, version 1
  • ARXIV : 1011.4362

Collections

Citation

Bruno Scherrer. Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view. 27th International Conference on Machine Learning - ICML 2010, Jun 2010, Haïfa, Israel. 2010. 〈inria-00537403〉

Partager

Métriques

Consultations de la notice

268

Téléchargements de fichiers

559