HAL will be down for maintenance from Friday, June 10 at 4pm through Monday, June 13 at 9am. More information
Skip to Main content Skip to Navigation
Conference papers

Pipelined Model Parallelism: Complexity Results and Memory Considerations

Olivier Beaumont 1, 2 Lionel Eyraud-Dubois 1, 2 Alena Shilova 1, 2
1 HiePACS - High-End Parallel Algorithms for Challenging Numerical Simulations
LaBRI - Laboratoire Bordelais de Recherche en Informatique, Inria Bordeaux - Sud-Ouest
Abstract : The training phase in Deep Neural Networks has become an important source of computing resource usage and because of the resulting volume of computation, it is crucial to perform it efficiently on parallel architectures. Even today, data parallelism is the most widely used method, but the associated requirement to replicate all the weights on the totality of computation resources poses problems of memory at the level of each node and of collective communications at the level of the platform. In this context, the model parallelism, which consists in distributing the different layers of the network over the computing nodes, is an attractive alternative. Indeed, it is expected to better distribute weights (to cope with memory problems) and it does not imply large collective communications since only forward activations are communicated. However, to be efficient, it must be combined with a pipelined / streaming approach, which leads in turn to new memory costs. The goal of this paper is to model these memory costs in detail, to analyze the complexity of the associated throughput optimization problem under memory constraints and to show that it is possible to formalize this optimization problem as an Integer Linear Program (ILP).
Complete list of metadata

Cited literature [42 references]  Display  Hide  Download

https://hal.inria.fr/hal-02968802
Contributor : Lionel Eyraud-Dubois Connect in order to contact the contributor
Submitted on : Friday, October 16, 2020 - 10:03:03 AM
Last modification on : Monday, December 20, 2021 - 4:50:14 PM

File

paperILP.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02968802, version 1

Citation

Olivier Beaumont, Lionel Eyraud-Dubois, Alena Shilova. Pipelined Model Parallelism: Complexity Results and Memory Considerations. 27th International European Conference on Parallel and Distributed Computing, Aug 2021, Lisbon, Portugal. ⟨hal-02968802v1⟩

Share

Metrics

Record views

314

Files downloads

309