HAL - INRIA :: [inria-00116936, version 2] A Unified View of TD Algorithms; Introducing Full-Gradient TD and Equi-Gradient Descent TD

inria-00116936, version 2

A Unified View of TD Algorithms; Introducing Full-Gradient TD and Equi-Gradient Descent TD

Manuel Loth ()^a^, 1, 2, Philippe Preux ()^b^, 1, 2, Manuel Davy^c^, 1, 3

European Symposium on Artificial Neural Networks (2007)

Abstract: This paper addresses the issue of policy evaluation in Markov Decision Processes, using linear function approximation. It provides a unified view of algorithms such as TD(lambda), LSTD(lambda), iLSTD, residual-gradient TD. It is asserted that they all consist in minimizing a gradient function and differ by the form of this function and their means of minimizing it. Two new schemes are introduced in that framework: Full-gradient TD which uses a generalization of the principle introduced in iLSTD, and EGD TD, which reduces the gradient by successive equi-gradient descents. These three algorithms form a new intermediate family with the interesting property of making much better use of the samples than TD while keeping a gradient descent scheme, which is useful for complexity issues and optimistic policy iteration.

a – Université des Sciences et Technologie de Lille - Lille I
b – Université Charles de Gaulle - Lille III
c – Ecole Centrale de Lille
1: SEQUEL (INRIA Futurs)
INRIA – CNRS : UMR8022 – CNRS : UMR8146 – Université des Sciences et Technologies de Lille - Lille I – Université Charles de Gaulle - Lille III – Ecole Centrale de Lille
2: GRAPPA (LIFL)
CNRS : UMR8022 – Université Charles de Gaulle - Lille III – Université des Sciences et Technologies de Lille - Lille I
3: Laboratoire d'Automatique, Génie Informatique et Signal (LAGIS)
CNRS : UMR8146 – Université des Sciences et Technologies de Lille - Lille I – Ecole Centrale de Lille

inria-00116936, version 2
http://hal.inria.fr/inria-00116936
oai:hal.inria.fr:inria-00116936
From: Manuel Loth <>
Submitted on: Wednesday, 29 November 2006 10:12:47
Updated on: Monday, 28 May 2007 19:56:53

See detailed view

Associated documents

PS :

PDF :

arXiv: cs.LG/0611145

Export

Bibtex EndNote TEI RefWorks

<?xml version="1.0" encoding="UTF-8"?>
<listBibl xmlns="http://www.tei-c.org/ns/1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:mml="http://www.w3.org/1998/Math/MathML">
<listBibl type="inproceedings"><head>Peer-reviewed conferences/proceedings</head>
<biblStruct type="inproceedings" xml:id="inria-00116936"><analytic>
<author><persName><forename>Manuel</forename><surname>Loth</surname></persName>
<affiliation>
<orgName type="laboratory">INRIA Futurs</orgName>
<orgName type="team">SEQUEL</orgName>
<country>FR</country>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">CNRS : UMR8022</orgName>
<orgName type="institution">CNRS : UMR8146</orgName>
<orgName type="institution">Université des Sciences et Technologies de Lille - Lille I</orgName>
<orgName type="institution">Université Charles de Gaulle - Lille III</orgName>
<orgName type="institution">Ecole Centrale de Lille</orgName>
</affiliation>
<affiliation>
<orgName type="laboratory">LIFL</orgName>
<orgName type="team">GRAPPA</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR8022</orgName>
<orgName type="institution">Université Charles de Gaulle - Lille III</orgName>
<orgName type="institution">Université des Sciences et Technologies de Lille - Lille I</orgName>
</affiliation>
</author>
<author><persName><forename>Philippe</forename><surname>Preux</surname></persName>
<affiliation>
<orgName type="laboratory">INRIA Futurs</orgName>
<orgName type="team">SEQUEL</orgName>
<country>FR</country>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">CNRS : UMR8022</orgName>
<orgName type="institution">CNRS : UMR8146</orgName>
<orgName type="institution">Université des Sciences et Technologies de Lille - Lille I</orgName>
<orgName type="institution">Université Charles de Gaulle - Lille III</orgName>
<orgName type="institution">Ecole Centrale de Lille</orgName>
</affiliation>
<affiliation>
<orgName type="laboratory">LIFL</orgName>
<orgName type="team">GRAPPA</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR8022</orgName>
<orgName type="institution">Université Charles de Gaulle - Lille III</orgName>
<orgName type="institution">Université des Sciences et Technologies de Lille - Lille I</orgName>
</affiliation>
</author>
<author><persName><forename>Manuel</forename><surname>Davy</surname></persName>
<affiliation>
<orgName type="laboratory">INRIA Futurs</orgName>
<orgName type="team">SEQUEL</orgName>
<country>FR</country>
<orgName type="institution">INRIA</orgName>
<orgName type="institution">CNRS : UMR8022</orgName>
<orgName type="institution">CNRS : UMR8146</orgName>
<orgName type="institution">Université des Sciences et Technologies de Lille - Lille I</orgName>
<orgName type="institution">Université Charles de Gaulle - Lille III</orgName>
<orgName type="institution">Ecole Centrale de Lille</orgName>
</affiliation>
<affiliation>
<orgName type="laboratory">LAGIS</orgName>
<orgName type="team">Laboratoire d'Automatique, Génie Informatique et Signal</orgName>
<country>FR</country>
<orgName type="institution">CNRS : UMR8146</orgName>
<orgName type="institution">Université des Sciences et Technologies de Lille - Lille I</orgName>
<orgName type="institution">Ecole Centrale de Lille</orgName>
</affiliation>
</author>
<title type="main" level="a">A Unified View of TD Algorithms; Introducing Full-Gradient TD and Equi-Gradient Descent TD</title></analytic><monogr><meeting>European Symposium on Artificial Neural Networks</meeting><imprint><pubPlace><settlement>Bruges</settlement><country> Belgium</country></pubPlace><biblScope type="publicationDate">2007-04-25</biblScope></imprint></monogr><idno type="HALid">http://hal.inria.fr/inria-00116936</idno><idno type="HALFile">http://hal.inria.fr/inria-00116936/PDF/unified.pdf</idno></biblStruct>
</listBibl>
</listBibl>