Skip to Main content Skip to Navigation
Conference papers

Speculative Scheduling for Stochastic HPC Applications

Abstract : New emerging fields are developing a growing number of large-scale applications with heterogeneous, dynamic and data-intensive requirements that put a high emphasis on productivity and thus are not tuned to run efficiently on today's high performance computing (HPC) systems. Some of these applications, such as neuroscience workloads and those that use adaptive numerical algorithms, develop modeling and simulation workflows with stochastic execution times and unpredictable resource requirements. When they are deployed on current HPC systems using existing resource management solutions, it can result in loss of efficiency for the users and decrease in effective system utilization for the platform providers. In this paper, we consider the current HPC scheduling model and describe the challenge it poses for stochastic applications due to the strict requirement in its job deployment policies. To address the challenge, we present speculative scheduling techniques that adapt the resource requirements of a stochastic application on-the-fly, based on its past execution behavior instead of relying on estimates given by the user. We focus on improving the overall system utilization and application response time without disrupting the current HPC scheduling model or the application development process. Our solution can operate alongside existing HPC batch schedulers without interfering with their usage modes. We show that speculative scheduling can improve the system utilization and average application response time by 25-30% compared to the classical HPC approach.
Complete list of metadata

Cited literature [27 references]  Display  Hide  Download
Contributor : Guillaume Pallez (aupy) Connect in order to contact the contributor
Submitted on : Tuesday, June 18, 2019 - 10:58:12 AM
Last modification on : Friday, January 21, 2022 - 3:11:48 AM


Files produced by the author(s)




Ana Gainaru, Guillaume Pallez, Hongyang Sun, Padma Raghavan. Speculative Scheduling for Stochastic HPC Applications. ICPP 2019 - 48th International Conference on Parallel Processing, Aug 2019, Kyoto, Japan. ⟨10.1145/nnnnnnn.nnnnnnn⟩. ⟨hal-02158598⟩



Les métriques sont temporairement indisponibles