Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Pipelined Model Parallelism: Complexity Results and Memory Considerations

Olivier Beaumont 1, 2 Lionel Eyraud-Dubois 1, 2 Alena Shilova 1, 2
1 HiePACS - High-End Parallel Algorithms for Challenging Numerical Simulations
LaBRI - Laboratoire Bordelais de Recherche en Informatique, Inria Bordeaux - Sud-Ouest
Abstract : The training phase in Deep Neural Networks has become an important source of computing resource usage and because of the resulting volume of computation, it is crucial to perform it efficiently on parallel architectures. Even today, data parallelism is the most widely used method, but the associated requirement to replicate all the weights on the totality of computation resources poses problems of memory at the level of each node and of collective communications at the level of the platform. In this context, the model parallelism, which consists in distributing the different layers of the network over the computing nodes, is an attractive alternative. Indeed, it is expected to better distribute weights (to cope with memory problems) and it does not imply large collective communications since only forward activations are communicated. However, to be efficient, it must be combined with a pipelined / streaming approach, which leads in turn to new memory costs. The goal of this paper is to model these memory costs in detail, to analyze the complexity of the associated throughput optimization problem under memory constraints and to show that it is possible to formalize this optimization problem as an Integer Linear Program (ILP).
Complete list of metadatas

Cited literature [42 references]  Display  Hide  Download
Contributor : Lionel Eyraud-Dubois <>
Submitted on : Friday, October 16, 2020 - 6:00:26 PM
Last modification on : Tuesday, October 20, 2020 - 3:37:24 AM


Files produced by the author(s)


  • HAL Id : hal-02968802, version 2



Olivier Beaumont, Lionel Eyraud-Dubois, Alena Shilova. Pipelined Model Parallelism: Complexity Results and Memory Considerations. 2020. ⟨hal-02968802v2⟩



Record views


Files downloads