Dual-Paradigm Stream Processing - Archive ouverte HAL Access content directly
Conference Papers Year :

Dual-Paradigm Stream Processing

(1) , (1) , (2, 3) , (1) , (1) , (1)


Existing stream processing frameworks operate either under data stream paradigm processing data record by record to favor low latency, or under operation stream paradigm processing data in micro-batches to desire high throughput. For complex and mutable data processing requirements, this dilemma brings the selection and deployment of stream processing frameworks into an embarrass- ing situation. Moreover, current data stream or operation stream paradigms cannot handle data burst e ciently, which probably re- sults in noticeable performance degradation. This paper introduces a dual-paradigm stream processing, called DO (Data and Operation) that can adapt to stream data volatility. It enables data to be pro- cessed in micro-batches (i.e., operation stream) when data burst occurs to achieve high throughput, while data is processed record by record (i.e., data stream) in the remaining time to sustain low latency. DO embraces a method to detect data bursts, identify the main operations a ected by the data burst and switch paradigms accordingly. Our insight behind DO’s design is that the trade-o between latency and throughput of stream processing frameworks can be dynamically achieved according to data communication among operations in a ne-grained manner (i.e., operation level) instead of framework level. We implement a prototype stream pro- cessing framework that adopts DO. Our experimental results show that our framework with DO can achieve 5x speedup over operation stream under low data stream sizes, and outperforms data stream on throughput by 2.1x to 3.2x under data burst.
Fichier principal
Vignette du fichier
ICPP2018-Stream.pdf (5.2 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-01834668 , version 1 (21-09-2018)



Song Wu, Zhiyi Liu, Shadi Ibrahim, Lin Gu, Hai Jin, et al.. Dual-Paradigm Stream Processing. ICPP 2018 - 47th International Conference on Parallel Processing, Aug 2018, Eugene, United States. pp.Article No. 83, ⟨10.1145/3225058.3225120⟩. ⟨hal-01834668⟩
356 View
261 Download



Gmail Facebook Twitter LinkedIn More