Skip to Main content Skip to Navigation
Journal articles

Optimal Memory-aware Backpropagation of Deep Join Networks

Olivier Beaumont 1 Julien Herrmann 2 Guillaume Pallez 2 Alena Shilova 1
1 HiePACS - High-End Parallel Algorithms for Challenging Numerical Simulations
LaBRI - Laboratoire Bordelais de Recherche en Informatique, Inria Bordeaux - Sud-Ouest
2 TADAAM - Topology-Aware System-Scale Data Management for High-Performance Computing
LaBRI - Laboratoire Bordelais de Recherche en Informatique, Inria Bordeaux - Sud-Ouest
Abstract : Deep Learning training memory needs can prevent the user to consider large models and large batch sizes. In this work, we propose to use techniques from memory-aware scheduling and Automatic Differentiation (AD) to execute a backpropagation graph with a bounded memory requirement at the cost of extra recomputations. The case of a single homogeneous chain, i.e. the case of a network whose all stages are identical and form a chain, is well understood and optimal solutions have been proposed in the AD literature. The networks encountered in practice in the context of Deep Learning are much more diverse, both in terms of shape and heterogeneity. In this work, we define the class of backpropagation graphs, and extend those on which one can compute in polynomial time a solution that minimizes the total number of recomputations. In particular we consider join graphs which correspond to models such as Siamese or Cross Modal Networks.
Complete list of metadatas

Cited literature [14 references]  Display  Hide  Download

https://hal.inria.fr/hal-02401105
Contributor : Guillaume Pallez (aupy) <>
Submitted on : Monday, December 9, 2019 - 6:10:59 PM
Last modification on : Thursday, January 9, 2020 - 9:56:10 AM
Document(s) archivé(s) le : Tuesday, March 10, 2020 - 11:03:53 PM

File

rsta_rev2.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02401105, version 1

Collections

Citation

Olivier Beaumont, Julien Herrmann, Guillaume Pallez, Alena Shilova. Optimal Memory-aware Backpropagation of Deep Join Networks. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Royal Society, The, In press. ⟨hal-02401105⟩

Share

Metrics

Record views

36

Files downloads

118