Dynamic Load Balancing Techniques for Distributed Complex Event Processing Systems

Abstract : Applying real-time, cost-effective Complex Event processing (CEP) in the cloud has been an important goal in recent years. Distributed Stream Processing Systems (DSPS) have been widely adopted by major computing companies such as Facebook and Twitter for performing scalable event processing in streaming data. However, dynamically balancing the load of the DSPS’ components can be particularly challenging due to the high volume of data, the components’ state management needs, and the low latency processing requirements. Systems should be able to cope with these challenges and adapt to dynamic and unpredictable load changes in real-time. Our approach makes the following contributions: (i) we formulate the load balancing problem in distributed CEP systems as an instance of the job-shop scheduling problem, and (ii) we present a novel framework that dynamically balances the load of CEP engines in real-time and adapts to sudden changes in the volume of streaming data by exploiting two balancing policies. Our detailed experimental evaluation using data from the Twitter social network indicates the benefits of our approach in the system’s throughput.
Complete list of metadatas

Cited literature [15 references]  Display  Hide  Download

https://hal.inria.fr/hal-01434793
Contributor : Hal Ifip <>
Submitted on : Friday, January 13, 2017 - 2:02:30 PM
Last modification on : Friday, January 13, 2017 - 2:05:49 PM
Long-term archiving on : Friday, April 14, 2017 - 8:23:08 PM

File

416479_1_En_14_Chapter.pdf
Files produced by the author(s)

Licence


Distributed under a Creative Commons Attribution 4.0 International License

Identifiers

Citation

Nikos Zacheilas, Nikolas Zygouras, Nikolaos Panagiotou, Vana Kalogeraki, Dimitrios Gunopulos. Dynamic Load Balancing Techniques for Distributed Complex Event Processing Systems. 16th IFIP WG 6.1 International Conference on Distributed Applications and Interoperable Systems (DAIS), Jun 2016, Heraklion, Crete, Greece. pp.174-188, ⟨10.1007/978-3-319-39577-7_14⟩. ⟨hal-01434793⟩

Share

Metrics

Record views

89

Files downloads

127