Towards a Unified Storage and Ingestion Architecture for Stream Processing

Abstract : Big Data applications are rapidly moving from a batch-oriented execution model to a streaming execution model in order to extract value from the data in real-time. However, processing live data alone is often not enough: in many cases, such applications need to combine the live data with previously archived data to increase the quality of the extracted insights. Current streaming-oriented runtimes and middlewares are not flexible enough to deal with this trend, as they address ingestion (collection and pre-processing of data streams) and persistent storage (archival of intermediate results) using separate services. This separation often leads to I/O redundancy (e.g., write data twice to disk or transfer data twice over the network) and interference (e.g., I/O bottlenecks when collecting data streams and writing archival data simultaneously). In this position paper, we argue for a unified ingestion and storage architecture for streaming data that addresses the aforementioned challenge. We identify a set of constraints and benefits for such a unified model, while highlighting the important architectural aspects required to implement it in real life. Based on these aspects, we briefly sketch our plan for future work that develops the position defended in this paper.
Type de document :
Communication dans un congrès
Second Workshop on Real-time & Stream Analytics in Big Data Colocates with the 2017 IEEE International Conference on Big Data, Dec 2017, Boston, United States. IEEE, pp.1-6, 2017, 〈https://workshop.euranova.eu/bigdata17.html〉
Liste complète des métadonnées

Littérature citée [26 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01649207
Contributeur : Ovidiu-Cristian Marcu <>
Soumis le : lundi 27 novembre 2017 - 12:35:53
Dernière modification le : mercredi 16 mai 2018 - 11:24:13

Fichier

bare_conf.pdf
Fichiers produits par l'(les) auteur(s)

Licence


Copyright (Tous droits réservés)

Identifiants

  • HAL Id : hal-01649207, version 1

Citation

Ovidiu-Cristian Marcu, Alexandru Costan, Gabriel Antoniu, María S. Pérez-Hernández, Radu Tudoran, et al.. Towards a Unified Storage and Ingestion Architecture for Stream Processing. Second Workshop on Real-time & Stream Analytics in Big Data Colocates with the 2017 IEEE International Conference on Big Data, Dec 2017, Boston, United States. IEEE, pp.1-6, 2017, 〈https://workshop.euranova.eu/bigdata17.html〉. 〈hal-01649207〉

Partager

Métriques

Consultations de la notice

467

Téléchargements de fichiers

331