Locality-Aware Routing in Stateful Streaming Applications

Matthieu Caneill 1 Ahmed El Rheddane 1 Vincent Leroy 2 Noël De Palma 1
1 ERODS - Efficient and Robust Distributed Systems
LIG - Laboratoire d'Informatique de Grenoble, UJF - Université Joseph Fourier - Grenoble 1
Abstract : Distributed stream processing engines continuously execute series of operators on data streams. Horizontal scaling is achieved by deploying multiple instances of each operator in order to process data tuples in parallel. As the application is distributed on an increasingly high number of servers, the likelihood that the stream is sent to a different server for each operator increases. This is particularly important in the case of stateful applications that rely on keys to deterministically route messages to a specific instance of an operator. Since network is a bottleneck for many stream applications, this behavior significantly degrades their performance. Our objective is to improve stream locality for stateful stream processing applications. We propose to analyse traces of the application to uncover correlations between the keys used in successive routing operations. By assigning correlated keys to instances hosted on the same server, we significantly reduce network consumption and increase performance while preserving load balance. Furthermore, this approach is executed online, so that the assignment can automatically adapt to changes in the characteristics of the data. Data migration is handled seamlessly with each routing configuration update. We implemented and evaluated our protocol using Apache Storm, with a real workload consisting of geo-tagged Flickr pictures as well as Twitter publications. Our results show a significant improvement in throughput.
Type de document :
Communication dans un congrès
Middleware'16 - 17th International Middleware Conference, Dec 2016, Trento, Italy. Middleware'16, pp.1 - 13, 2016, 〈http://2016.middleware-conference.org/〉. 〈10.1145/2988336.2988340〉
Liste complète des métadonnées

Littérature citée [19 références]  Voir  Masquer  Télécharger

Contributeur : Matthieu Caneill <>
Soumis le : lundi 12 décembre 2016 - 16:03:31
Dernière modification le : jeudi 11 octobre 2018 - 08:48:05
Document(s) archivé(s) le : mardi 21 mars 2017 - 06:52:11


Accord explicite pour ce dépôt




Matthieu Caneill, Ahmed El Rheddane, Vincent Leroy, Noël De Palma. Locality-Aware Routing in Stateful Streaming Applications. Middleware'16 - 17th International Middleware Conference, Dec 2016, Trento, Italy. Middleware'16, pp.1 - 13, 2016, 〈http://2016.middleware-conference.org/〉. 〈10.1145/2988336.2988340〉. 〈hal-01407457〉



Consultations de la notice


Téléchargements de fichiers