Modeling High-throughput Applications for in situ Analytics

Guillaume Aupy 1 Brice Goglin 1 Valentin Honoré 1, 2 Bruno Raffin 3
1 TADAAM - Topology-Aware System-Scale Data Management for High-Performance Computing
LaBRI - Laboratoire Bordelais de Recherche en Informatique, Inria Bordeaux - Sud-Ouest
3 DATAMOVE - Data Aware Large Scale Computing
Inria Grenoble - Rhône-Alpes, LIG - Laboratoire d'Informatique de Grenoble
Abstract : With the goal of performing exascale computing, the importance of I/O management becomes more and more critical to maintain system performance. While the computing capacities of machines are getting higher, the I/O capa- bilities of systems do not increase as fast. We are able to generate more data but unable to manage them eciently due to variability of I/O performance. Limiting the requests to the Parallel File System (PFS) becomes necessary. To address this issue, new strategies are being developed such as online in situ analysis. The idea is to overcome the limitations of basic post-mortem data analysis where the data have to be stored on PFS rst and processed later. There are several software solutions that allow users to speci cally dedicate nodes for analysis of data and distribute the computation tasks over di er- ent sets of nodes. Thus far, they rely on a manual resource partitioning and allocation by the user of tasks (simulations, analysis). In this work, we propose a memory-constraint modelization for in situ anal- ysis. We use this model to provide di erent scheduling policies to determine both the number of resources that should be dedicated to analysis functions, and that schedule eciently these functions. We evaluate them and show the importance of considering memory constraints in the model. Finally, we discuss the di erent challenges that have to be addressed in order to build automatic tools for in situ analytics.
Complete list of metadatas
Contributor : Valentin Honoré <>
Submitted on : Friday, April 5, 2019 - 3:58:16 PM
Last modification on : Thursday, May 16, 2019 - 3:40:34 PM


Files produced by the author(s)


  • HAL Id : hal-02091340, version 1


Guillaume Aupy, Brice Goglin, Valentin Honoré, Bruno Raffin. Modeling High-throughput Applications for in situ Analytics. International Journal of High Performance Computing Applications, SAGE Publications, In press, pp.1-44. ⟨hal-02091340⟩



Record views


Files downloads