On-the-fly scheduling vs. reservation-based scheduling for unpredictable workflows - Archive ouverte HAL Access content directly
Journal Articles International Journal of High Performance Computing Applications Year : 2019

On-the-fly scheduling vs. reservation-based scheduling for unpredictable workflows

(1) , (1) , (2) , (1) , (1) , (1)
1
2

Abstract

Scientific insights in the coming decade will clearly depend on the effective processing of large datasets generated by dynamic heterogeneous applications typical of workflows in large data centers or of emerging fields like neuroscience. In this paper, we show how these big data workflows have a unique set of characteristics that pose challenges for leveraging HPC methodologies, particularly in scheduling. Our findings indicate that execution times for these workflows are highly unpredictable and are not correlated with the size of the dataset involved or the precise functions used in the analysis. We characterize this inherent variability and sketch the need for new scheduling approaches by quantifying significant gaps in achievable performance. Through simulations, we show how on-the-fly scheduling approaches can deliver benefits in both system-level and user-level performance measures. On average, we find improvements of up to 35% in system utilization and up to 45% in average stretch of the applications, illustrating the potential of increasing performance through new scheduling approaches.
Fichier principal
Vignette du fichier
ijhpca_CR.pdf (3.12 Mo) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-02058290 , version 1 (05-03-2019)

Identifiers

Cite

Ana Gainaru, Hongyang Sun, Guillaume Aupy, Yuankai Huo, Bennett A Landman, et al.. On-the-fly scheduling vs. reservation-based scheduling for unpredictable workflows. International Journal of High Performance Computing Applications, In press, ⟨10.1177/1094342019841681⟩. ⟨hal-02058290⟩

Collections

CNRS INRIA INRIA2
58 View
270 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More