Skip to Main content Skip to Navigation
Reports

Whittle Index Policy for Crawling Ephemeral Content

Abstract : We consider a task of scheduling a crawler to retrieve content from several sites with ephemeral content. A user typically loses interest in ephemeral content, like news or posts at social network groups, after several days or hours. Thus, development of timely crawling policy for such ephemeral information sources is very important. We first formulate this problem as an optimal control problem with average reward. The reward can be measured in the number of clicks or relevant search requests. The problem in its initial formulation suffers from the curse of dimensionality and quickly becomes intractable even with moderate number of information sources. Fortunately, this problem admits a Whittle index, which leads to problem decomposition and to a very simple and efficient crawling policy. We derive the Whittle index and provide its theoretical justification.
Complete list of metadata

Cited literature [27 references]  Display  Hide  Download

https://hal.inria.fr/hal-01136651
Contributor : Konstantin Avrachenkov <>
Submitted on : Friday, March 27, 2015 - 5:05:50 PM
Last modification on : Wednesday, October 14, 2020 - 3:57:38 AM
Long-term archiving on: : Tuesday, April 18, 2017 - 2:26:57 AM

Files

RR-8702.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01136651, version 1
  • ARXIV : 1503.08558

Collections

Citation

Konstantin Avrachenkov, Vivek Borkar. Whittle Index Policy for Crawling Ephemeral Content. [Research Report] RR-8702, Inria Sophia Antipolis; INRIA. 2015. ⟨hal-01136651⟩

Share

Metrics

Record views

548

Files downloads

342