Large scale in transit computation of quantiles for ensemble runs

Abstract : While estimating the uncertainties of numerical simulation model outputs, quantiles are important statistical quantities that can be used in risk analysis, outlier detection or computation of confidence intervals. Quantiles being order statistics, the classical approach for their computation requires availability of the full sample before ranking it. In numerical simulation, this approach is not suitable at exascale as large ensembles of model runs would need to gather a prohibitively large amount of data. This paper solves this problem by using an iterative approach based on the stochastic quan-tile algorithm of Robbins-Monro whose parameters are finely tuned in order to gain robustness. The computational part of the approach relies on the Melissa framework, a file avoiding, adaptive, fault tolerant and elastic framework. Quantiles are updated on-the-fly as soon as the in transit parallel server receives results from one of the running simulations. We validate our approach on a use case based on 3000 fluid dynamics parallel simulations of 6M hexahedra and 100 time-steps. This validation case was executed on two supercomputers, avoiding 11 TB of file storage per execution. Ubiquitous spatio-temporal maps of quantiles and inter-quantile based intervals are then produced via our robustly tuned Robbins-Monro algorithm.
Type de document :
Pré-publication, Document de travail
2019
Liste complète des métadonnées

https://hal.inria.fr/hal-02016828
Contributeur : Bertrand Iooss <>
Soumis le : mardi 12 février 2019 - 22:41:05
Dernière modification le : lundi 25 février 2019 - 16:34:22

Fichier

Melissa_quantiles2019.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-02016828, version 1

Citation

Alejandro Ribes, Théophile Terraz, Bertrand Iooss, Yvan Fournier, Bruno Raffin. Large scale in transit computation of quantiles for ensemble runs. 2019. 〈hal-02016828〉

Partager

Métriques

Consultations de la notice

47

Téléchargements de fichiers

100