Skip to Main content Skip to Navigation
Conference papers

DeepFreeze: Towards Scalable Asynchronous Checkpointing of Deep Learning Models

Abstract : In the age of big data, deep learning has emerged as a powerful tool to extract insight and exploit its value, both in industry and scientific applications. One common pattern emerging in such applications is frequent checkpointing of the state of the learning model during training, needed in a variety of scenarios: analysis of intermediate states to explain features and correlations with training data, exploration strategies involving alternative models that share a common ancestor, knowledge transfer, resilience, etc. However, with increasing size of the learning models and popularity of distributed data-parallel training approaches, simple checkpointing techniques used so far face several limitations: low serialization performance, blocking I/O, stragglers due to the fact that only a single process is involved in checkpointing. This paper proposes a checkpointing technique specifically designed to address the aforementioned limitations, introducing efficient asynchronous techniques to hide the overhead of serialization and I/O, and distribute the load over all participating processes. Experiments with two deep learning applications (CANDLE and ResNet) on a pre-Exascale HPC platform (Theta) shows significant improvement over state-of-art, both in terms of checkpointing duration and runtime overhead.
Complete list of metadatas

Cited literature [33 references]  Display  Hide  Download

https://hal.inria.fr/hal-02543977
Contributor : Bogdan Nicolae <>
Submitted on : Wednesday, April 15, 2020 - 7:02:44 PM
Last modification on : Thursday, April 16, 2020 - 3:53:13 PM

File

paper.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02543977, version 1

Citation

Bogdan Nicolae, Jiali Li, Justin Wozniak, George Bosilca, Matthieu Dorier, et al.. DeepFreeze: Towards Scalable Asynchronous Checkpointing of Deep Learning Models. CCGrid'20: 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing, Nov 2020, Melbourne, Australia. ⟨hal-02543977⟩

Share

Metrics

Record views

298

Files downloads

289