Reinforcement Learning of POMDPs using Spectral Methods

Kamyar Azizzadenesheli 1 Alessandro Lazaric 2, 3 Animashree Anandkumar 1
1 UCI - University of California [Irvine]
2 SEQUEL - Sequential Learning
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189
3 CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189
Abstract : We propose a new reinforcement learning algorithm for partially observable Markov decision processes (POMDP) based on spectral decomposition methods. While spectral methods have been previously employed for consistent learning of (passive) latent variable models such as hidden Markov models, POMDPs are more challenging since the learner interacts with the environment and possibly changes the future observations in the process. We devise a learning algorithm running through episodes, in each episode we employ spectral techniques to learn the POMDP parameters from a trajectory generated by a fixed policy. At the end of the episode, an optimization oracle returns the optimal memoryless planning policy which maximizes the expected reward based on the estimated POMDP model. We prove an order-optimal regret bound w.r.t. the optimal memoryless policy and efficient scaling with respect to the dimensionality of observation and action spaces.
Keywords : Spectral Methods Method of Moments Partially Observable Markov Decision Pro-cess Latent Variable Model Upper Confidence Reinforcement Learning
Type de document :
Communication dans un congrès
Proceedings of the 29th Annual Conference on Learning Theory (COLT2016), Jun 2016, New York City, United States. 2016
Liste complète des métadonnées

Littérature citée [27 références]  Voir  Masquer  Télécharger
https://hal.inria.fr/hal-01322207
Contributeur : Alessandro Lazaric <>
Soumis le : jeudi 26 mai 2016 - 17:34:15
Dernière modification le : mardi 3 juillet 2018 - 11:46:51
Document(s) archivé(s) le : samedi 27 août 2016 - 11:03:09

Fichier

master.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01322207, version 1

Collections

INRIA | INSTITUT-TELECOM | CRISTAL-SEQUEL | CRISTAL | UNIV-LILLE

Citation

Kamyar Azizzadenesheli, Alessandro Lazaric, Animashree Anandkumar. Reinforcement Learning of POMDPs using Spectral Methods. Proceedings of the 29th Annual Conference on Learning Theory (COLT2016), Jun 2016, New York City, United States. 2016. 〈hal-01322207〉

Exporter

BibTeX TEI DC DCterms EndNote

Partager

Métriques

Consultations de la notice

224

Téléchargements de fichiers

155

Contact


archive-ouverte@inria.fr