Skip to Main content Skip to Navigation
Reports

Limiting the memory footprint when dynamically scheduling DAGs on shared-memory platforms

Loris Marchal 1 Bertrand Simon 1 Frédéric Vivien 1
1 ROMA - Optimisation des ressources : modèles, algorithmes et ordonnancement
Inria Grenoble - Rhône-Alpes, LIP - Laboratoire de l'Informatique du Parallélisme
Abstract : Scientific workflows are frequently modeled as Directed Acyclic Graphs (DAGs) of tasks, which represent computational modules and their dependences in the form of data pro- duced by a task and used by another one. This formulation allows the use of runtime systems which dynamically allocate tasks onto the resources of increasingly complex computing plat- forms. However, for some workflows, such a dynamic schedule may run out of memory by processing too many tasks simultaneously. This paper focuses on the problem of transform- ing such a DAG to prevent memory shortage, and concentrates on shared memory platforms. We first propose a simple model of DAGs which is expressive enough to emulate complex mem- ory behaviors. We then exhibit a polynomial-time algorithm that computes the maximum peak memory of a DAG, that is, the maximum memory needed by any parallel schedule. We consider the problem of reducing this maximum peak memory to make it smaller than a given bound. Our solution consists in adding new fictitious edges, while trying to minimize the critical path of the graph. After proving that this problem is NP-complete, we provide an ILP solution as well as several heuristic strategies that are thoroughly compared by simulation on synthetic DAGs modeling actual computational workflows. We show that on most instances we are able to de- crease the maximum peak memory at the cost of a small increase in the critical path, thus with little impact on the quality of the final parallel schedule.
Complete list of metadata

Cited literature [37 references]  Display  Hide  Download

https://hal.inria.fr/hal-01948462
Contributor : Equipe Roma <>
Submitted on : Sunday, December 9, 2018 - 8:06:29 PM
Last modification on : Monday, November 16, 2020 - 9:48:06 AM
Long-term archiving on: : Sunday, March 10, 2019 - 12:11:24 PM

File

RR-9231.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01948462, version 1

Collections

Citation

Loris Marchal, Bertrand Simon, Frédéric Vivien. Limiting the memory footprint when dynamically scheduling DAGs on shared-memory platforms. [Research Report] RR-9231, Inria Grenoble Rhône-Alpes. 2018, pp.1-41. ⟨hal-01948462⟩

Share

Metrics

Record views

214

Files downloads

368