Skip to Main content Skip to Navigation
Conference papers

Pipelined Model Parallelism: Complexity Results and Memory Considerations

Olivier Beaumont 1, 2 Lionel Eyraud-Dubois 1, 2 Alena Shilova 1, 2
1 HiePACS - High-End Parallel Algorithms for Challenging Numerical Simulations
LaBRI - Laboratoire Bordelais de Recherche en Informatique, Inria Bordeaux - Sud-Ouest
Abstract : The training phase in Deep Neural Networks has become an important source of computing resource usage and the resulting volume of computation makes it crucial to perform efficiently on parallel architectures. Data parallelism is the most widely used method, but it requires to replicate the network weights on all processors, and to perform collective communications of the network weights. In this context, model parallelism is an attractive alternative, in which the different layers of the network are distributed over the computing processors. Indeed, it is expected to better distribute weights (to cope with memory problems) and it eliminates the need for large collective communications since only forward activations are communicated. However, to be efficient, it must be combined with a pipelined approach, which in turn induces new memory costs. In this paper, our goal is to formalize pipelined model parallelism as a scheduling problem, to establish its complexity, and to analyze the consequences of the assumptions that are typically performed in practical solutions such as Pipedream.
Complete list of metadata
Contributor : Alena Shilova Connect in order to contact the contributor
Submitted on : Thursday, February 18, 2021 - 3:29:54 PM
Last modification on : Friday, January 21, 2022 - 3:23:11 AM


Files produced by the author(s)


  • HAL Id : hal-02968802, version 3



Olivier Beaumont, Lionel Eyraud-Dubois, Alena Shilova. Pipelined Model Parallelism: Complexity Results and Memory Considerations. Europar 2021, Aug 2021, Lisbon, Portugal. ⟨hal-02968802v3⟩



Les métriques sont temporairement indisponibles