Rotting bandits are not harder than stochastic ones

Julien Seznec 1, 2 Andrea Locatelli 3 Alexandra Carpentier 3 Alessandro Lazaric 1, 4 Michal Valko 1
1 SEQUEL - Sequential Learning
Inria Lille - Nord Europe, CRIStAL - Centre de Recherche en Informatique, Signal et Automatique de Lille (CRIStAL) - UMR 9189
Abstract : In bandits, arms' distributions are stationary. This is often violated in practice, where rewards change over time. In applications as recommendation systems, online advertising, and crowdsourcing, the changes may be triggered by the pulls, so that the arms' rewards change as a function of the number of pulls. In this paper, we consider the specific case of non-parametric rotting bandits, where the expected reward of an arm may decrease every time it is pulled. We introduce the filtering on expanding window average (FEWA) algorithm that at each round constructs moving averages of increasing windows to identify arms that are more likely to return high rewards when pulled once more. We prove that, without any knowledge on the decreasing behavior of the arms, FEWA achieves similar anytime problem-dependent, O(log(KT)), and problem-independent, O(sqrtKT), regret bounds of near-optimal stochastic algorithms as UCB1 of Auer et al. (2002a). This result substantially improves the prior result of Levine et al. (2017) which needed knowledge of the horizon and decaying parameters to achieve problem-independent bound of only O(K1/3T2/3). Finally, we report simulations confirming the theoretical improvements of FEWA.
Document type :
Conference papers
Complete list of metadatas

https://hal.inria.fr/hal-01936894
Contributor : Julien Seznec <>
Submitted on : Tuesday, November 27, 2018 - 4:57:38 PM
Last modification on : Friday, March 22, 2019 - 1:37:08 AM

File

seznec2019rotting_arxiv.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01936894, version 1

Citation

Julien Seznec, Andrea Locatelli, Alexandra Carpentier, Alessandro Lazaric, Michal Valko. Rotting bandits are not harder than stochastic ones. International Conference on Artificial Intelligence and Statistics, 2019, Naha, Japan. ⟨hal-01936894⟩

Share

Metrics

Record views

160

Files downloads

124