Large scale in transit computation of quantiles for ensemble runs

Abstract : While estimating the uncertainties of numerical simulation model outputs, quantiles are important statistical quantities that can be used in risk analysis, outlier detection or computation of confidence intervals. Quantiles being order statistics, the classical approach for their computation requires availability of the full sample before ranking it. In numerical simulation, this approach is not suitable at exascale as large ensembles of model runs would need to gather a prohibitively large amount of data. This paper solves this problem by using an iterative approach based on the stochastic quan-tile algorithm of Robbins-Monro whose parameters are finely tuned in order to gain robustness. The computational part of the approach relies on the Melissa framework, a file avoiding, adaptive, fault tolerant and elastic framework. Quantiles are updated on-the-fly as soon as the in transit parallel server receives results from one of the running simulations. We validate our approach on a use case based on 3000 fluid dynamics parallel simulations of 6M hexahedra and 100 time-steps. This validation case was executed on two supercomputers, avoiding 11 TB of file storage per execution. Ubiquitous spatio-temporal maps of quantiles and inter-quantile based intervals are then produced via our robustly tuned Robbins-Monro algorithm.
Document type :
Preprints, Working Papers, ...
Complete list of metadatas

Cited literature [44 references]  Display  Hide  Download

https://hal.inria.fr/hal-02016828
Contributor : Bertrand Iooss <>
Submitted on : Tuesday, February 12, 2019 - 10:41:05 PM
Last modification on : Thursday, October 24, 2019 - 1:46:09 PM
Long-term archiving on : Monday, May 13, 2019 - 6:30:21 PM

File

Melissa_quantiles2019.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02016828, version 1

Citation

Alejandro Ribes, Théophile Terraz, Bertrand Iooss, Yvan Fournier, Bruno Raffin. Large scale in transit computation of quantiles for ensemble runs. 2019. ⟨hal-02016828v1⟩

Share

Metrics

Record views

92

Files downloads

286