Skip to Main content Skip to Navigation
New interface
Journal articles

Taming System Dynamics on Resource Optimization for Data Processing Workflows: A Probabilistic Approach

Abstract : In many data-intensive applications, workflow is often used as an important model for organizing data processing tasks and resource provisioning is an important and challenging problem for improving the performance of workflows. Recently, system variations in the cloud and large-scale clusters, such as those in I/O and network performances and failure events, have been observed to greatly affect the performance of workflows. Traditional resource provisioning methods, which overlook these variations, can lead to suboptimal resource provisioning results. In this article, we provide a general solution for workflow performance optimizations considering system variations. Specifically, we model system dynamics as time-dependent random variables and take their probability distributions as optimization input. Despite its effectiveness, this solution involves heavy computation overhead. Thus, we propose three pruning techniques to simplify workflow structure and reduce the probability evaluation overhead. We implement our techniques in a runtime library, which allows users to incorporate efficient probabilistic optimization into existing resource provisioning methods. Experiments show that probabilistic solutions can improve the performance by up to 65 percent compared to state-of-the-art static solutions, and our pruning techniques can greatly reduce the overhead of our probabilistic approach.
Document type :
Journal articles
Complete list of metadata

https://hal.inria.fr/hal-03363176
Contributor : Shadi Ibrahim Connect in order to contact the contributor
Submitted on : Sunday, October 3, 2021 - 2:05:33 PM
Last modification on : Friday, August 5, 2022 - 2:54:52 PM

Identifiers

Citation

Amelie Chi Zhou, Weilin Xue, Yao Xiao, Bingsheng He, Shadi Ibrahim, et al.. Taming System Dynamics on Resource Optimization for Data Processing Workflows: A Probabilistic Approach. IEEE Transactions on Parallel and Distributed Systems, 2021, 33 (1), pp.231-248. ⟨10.1109/TPDS.2021.3091400⟩. ⟨hal-03363176⟩

Share

Metrics

Record views

45