Skip to Main content Skip to Navigation
Conference papers

SpecK: Composition of Stream Processing Applications over Fog Environments

Davaadorj Battulga 1, 2 Daniele Miorandi 1 Cédric Tedeschi 2 
2 MYRIADS - Design and Implementation of Autonomous Distributed Systems
Inria Rennes – Bretagne Atlantique , IRISA-D1 - SYSTÈMES LARGE ÉCHELLE
Abstract : Stream Processing (SP), i.e., the processing of data in motion, as soon as it becomes available, is a hot topic in cloud computing. Various SP stacks exist today, with applications ranging from IoT analytics to processing of payment transactions. The backbone of said stacks are Stream Processing Engines (SPEs), software packages offering a high-level programming model and scalable execution of data stream processing pipelines. SPEs have been traditionally developed to work inside a single datacenter, and optimised for speed. With the advent of Fog computing, however, the processing of data streams needs to be carried out over multiple geographically distributed computing sites: Data gets typically pre-processed close to where they are generated, then aggregated at intermediate nodes, and finally globally and persistently stored in the Cloud. SPEs were not designed to address these new scenarios. In this paper, we argue that large scale Fog-based stream processing should rely on the coordinated composition of geographically dispersed SPE instances. We propose an architecture based on the composition of multiple SPE instances and their communication via distributed message brokers. We introduce SpecK, a tool to automate the deployment and adaptation of pipelines over a Fog computing platform. Given a description of the pipeline, SpecKcovers all the operations needed to deploy a stream processing computation over the different SPE instances targeted, using their own APIs and establishing the required communication channels to forward data among them. A prototypical implementation of SpecK is presented, and its performance is evaluated over Grid'5000, a large-scale, distributed experimental facility.
Complete list of metadata
Contributor : Guillaume Pierre Connect in order to contact the contributor
Submitted on : Monday, June 14, 2021 - 3:41:09 PM
Last modification on : Thursday, May 5, 2022 - 10:42:47 AM
Long-term archiving on: : Thursday, September 16, 2021 - 8:23:17 AM


Files produced by the author(s)



Davaadorj Battulga, Daniele Miorandi, Cédric Tedeschi. SpecK: Composition of Stream Processing Applications over Fog Environments. DAIS 2021 - 21st International Conference on Distributed Applications and Interoperable Systems, Jun 2021, Valetta, Malta. pp.38-54, ⟨10.1007/978-3-030-78198-9_3⟩. ⟨hal-03259975⟩



Record views


Files downloads