Speculative Scheduling for Stochastic HPC Applications - Archive ouverte HAL Access content directly
Conference Papers Year :

Speculative Scheduling for Stochastic HPC Applications

(1) , (2) , (1) , (1)


New emerging fields are developing a growing number of large-scale applications with heterogeneous, dynamic and data-intensive requirements that put a high emphasis on productivity and thus are not tuned to run efficiently on today's high performance computing (HPC) systems. Some of these applications, such as neuroscience workloads and those that use adaptive numerical algorithms, develop modeling and simulation workflows with stochastic execution times and unpredictable resource requirements. When they are deployed on current HPC systems using existing resource management solutions, it can result in loss of efficiency for the users and decrease in effective system utilization for the platform providers. In this paper, we consider the current HPC scheduling model and describe the challenge it poses for stochastic applications due to the strict requirement in its job deployment policies. To address the challenge, we present speculative scheduling techniques that adapt the resource requirements of a stochastic application on-the-fly, based on its past execution behavior instead of relying on estimates given by the user. We focus on improving the overall system utilization and application response time without disrupting the current HPC scheduling model or the application development process. Our solution can operate alongside existing HPC batch schedulers without interfering with their usage modes. We show that speculative scheduling can improve the system utilization and average application response time by 25-30% compared to the classical HPC approach.
Fichier principal
Vignette du fichier
icpp19_speculative.pdf (1.25 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-02158598 , version 1 (18-06-2019)



Ana Gainaru, Guillaume Pallez, Hongyang Sun, Padma Raghavan. Speculative Scheduling for Stochastic HPC Applications. ICPP 2019 - 48th International Conference on Parallel Processing, Aug 2019, Kyoto, Japan. ⟨10.1145/nnnnnnn.nnnnnnn⟩. ⟨hal-02158598⟩


103 View
203 Download



Gmail Facebook Twitter LinkedIn More