Biasing Approximate Dynamic Programming with a Lower Discount Factor

Marek Petrik 1 Bruno Scherrer 2
2 MAIA - Autonomous intelligent machine
INRIA Lorraine, LORIA - Laboratoire Lorrain de Recherche en Informatique et ses Applications
Abstract : Most algorithms for solving Markov decision processes rely on a discount factor, which ensures their convergence. It is generally assumed that using an artificially low discount factor will improve the convergence rate, while sacrificing the solution quality. We however demonstrate that using an artificially low discount factor may significantly improve the solution quality, when used in approximate dynamic programming. We propose two explanations of this phenomenon. The first justification follows directly from the standard approximation error bounds: using a lower discount factor may decrease the approximation error bounds. However, we also show that these bounds are loose, thus their decrease does not entirely justify the improved solution quality. We thus propose another justification: when the rewards are received only sporadically (as in the case of Tetris), we can derive tighter bounds, which support a significant improvement in the solution quality with a decreased discount factor.
Document type :
Conference papers
Complete list of metadatas

Cited literature [8 references]  Display  Hide  Download

https://hal.inria.fr/inria-00337652
Contributor : Bruno Scherrer <>
Submitted on : Friday, November 7, 2008 - 3:41:37 PM
Last modification on : Tuesday, August 6, 2019 - 11:38:23 AM
Long-term archiving on : Tuesday, October 9, 2012 - 3:11:47 PM

File

finaldiscount.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : inria-00337652, version 1

Collections

Citation

Marek Petrik, Bruno Scherrer. Biasing Approximate Dynamic Programming with a Lower Discount Factor. Twenty-Second Annual Conference on Neural Information Processing Systems -NIPS 2008, Dec 2008, Vancouver, Canada. ⟨inria-00337652⟩

Share

Metrics

Record views

331

Files downloads

494