A Fully Decentralized Autoscaling Algorithm for Stream Processing Applications

Mehdi Belkhiria 1 Cédric Tedeschi 1
1 MYRIADS - Design and Implementation of Autonomous Distributed Systems
Inria Rennes – Bretagne Atlantique , IRISA-D1 - SYSTÈMES LARGE ÉCHELLE
Abstract : Stream Processing deals with the efficient, real-time processing of continuous streams of data. Stream Processing engines ease the development and deployment of such applications which are commonly pipelines of operators to be traversed by each data item. Due to the varying velocity of the streams, autoscaling is needed to dynamically adapt the number of instances of each operator. With the advent of geographically-dispersed computing platforms such as Fog platforms, operators are dispersed accordingly, and autoscaling needs to be decentralized as well. In this paper, we propose an algorithm allowing for scaling decisions to be taken and enforced in a fully-decentralized way. In particular, in spite of scaling actions being triggered concurrently, each operator maintains a view of its neighbours in the graph so as no data message is lost. The protocol is detailed and its correctness discussed. Its performance is captured through early simulation experiments.
Complete list of metadatas

Cited literature [20 references]  Display  Hide  Download

https://hal.inria.fr/hal-02171172
Contributor : Cédric Tedeschi <>
Submitted on : Tuesday, July 2, 2019 - 4:50:57 PM
Last modification on : Thursday, September 19, 2019 - 12:52:15 PM

File

autodasp2019.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02171172, version 1

Citation

Mehdi Belkhiria, Cédric Tedeschi. A Fully Decentralized Autoscaling Algorithm for Stream Processing Applications. Auto-DaSP 2019 - Third International Workshop on Autonomic Solutions for Parallel and Distributed Data Stream Processing, Aug 2019, Göttingen, Germany. pp.1-12. ⟨hal-02171172⟩

Share

Metrics

Record views

82

Files downloads

857