Skip to Main content Skip to Navigation
Journal articles

Modeling High-throughput Applications for in situ Analytics

Guillaume Aupy 1 Brice Goglin 1 Valentin Honoré 1 Bruno Raffin 2
1 TADAAM - Topology-Aware System-Scale Data Management for High-Performance Computing
LaBRI - Laboratoire Bordelais de Recherche en Informatique, Inria Bordeaux - Sud-Ouest
2 DATAMOVE - Data Aware Large Scale Computing
Inria Grenoble - Rhône-Alpes, LIG - Laboratoire d'Informatique de Grenoble
Abstract : With the goal of performing exascale computing, the importance of I/Omanagement becomes more and more critical to maintain system performance.While the computing capacities of machines are getting higher, the I/O capa-bilities of systems do not increase as fast. We are able to generate more databut unable to manage them eciently due to variability of I/O performance.Limiting the requests to the Parallel File System (PFS) becomes necessary. Toaddress this issue, new strategies are being developed such as online in situanalysis. The idea is to overcome the limitations of basic post-mortem dataanalysis where the data have to be stored on PFS rst and processed later.There are several software solutions that allow users to specically dedicatenodes for analysis of data and distribute the computation tasks over dier-ent sets of nodes. Thus far, they rely on a manual resource partitioning andallocation by the user of tasks (simulations, analysis).In this work, we propose a memory-constraint modelization for in situ anal-ysis. We use this model to provide dierent scheduling policies to determineboth the number of resources that should be dedicated to analysis functions,and that schedule eciently these functions. We evaluate them and show theimportance of considering memory constraints in the model. Finally, we discussthe dierent challenges that have to be addressed in order to build automatictools for in situ analytics.
Complete list of metadata

Cited literature [50 references]  Display  Hide  Download
Contributor : Valentin Honoré Connect in order to contact the contributor
Submitted on : Friday, April 5, 2019 - 3:58:16 PM
Last modification on : Wednesday, January 12, 2022 - 8:39:11 AM
Long-term archiving on: : Saturday, July 6, 2019 - 2:37:56 PM


Files produced by the author(s)



Guillaume Aupy, Brice Goglin, Valentin Honoré, Bruno Raffin. Modeling High-throughput Applications for in situ Analytics. International Journal of High Performance Computing Applications, SAGE Publications, In press, 33 (6), pp.1185-1200. ⟨10.1177/1094342019847263⟩. ⟨hal-02091340⟩



Les métriques sont temporairement indisponibles