Managing Hot Metadata for Scientific Workflows on Multisite Clouds

Luis Pineda-Morales 1, 2 Ji Liu 2, 3, 4 Alexandru Costan 1 Esther Pacitti 3, 4 Gabriel Antoniu 1 Patrick Valduriez 3, 4 Marta Mattoso 5
1 KerData - Scalable Storage for Clouds and Beyond
Inria Rennes – Bretagne Atlantique , IRISA-D1 - SYSTÈMES LARGE ÉCHELLE
3 ZENITH - Scientific Data Management
LIRMM - Laboratoire d'Informatique de Robotique et de Microélectronique de Montpellier, CRISAM - Inria Sophia Antipolis - Méditerranée
Abstract : Large-scale scientific applications are often expressed as workflows that help defining data dependencies between their different components. Several such workflows have huge storage and computation requirements, and so they need to be processed in multiple (cloud-federated) datacenters. It has been shown that efficient metadata handling plays a key role in the performance of computing systems. However, most of this evidence concern only single-site, HPC systems to date. In this paper, we present a hybrid decentralized/distributed model for handling hot metadata (frequently accessed metadata) in multisite architectures. We couple our model with a scientific workflow management system (SWfMS) to validate and tune its applicability to different real-life scientific scenarios. We show that efficient management of hot metadata improves the performance of SWfMS, reducing the workflow execution time up to 50% for highly parallel jobs and avoiding unnecessary cold metadata operations.
Type de document :
Communication dans un congrès
BIGDATA 2016 - 2016 IEEE International Conference on Big Data, Dec 2016, Washington, United States. 2016
Liste complète des métadonnées

Littérature citée [30 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01395715
Contributeur : Gabriel Antoniu <>
Soumis le : vendredi 11 novembre 2016 - 15:14:47
Dernière modification le : jeudi 26 octobre 2017 - 13:44:06
Document(s) archivé(s) le : jeudi 16 mars 2017 - 11:32:17

Fichier

BIGDATA2016-final.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01395715, version 1

Citation

Luis Pineda-Morales, Ji Liu, Alexandru Costan, Esther Pacitti, Gabriel Antoniu, et al.. Managing Hot Metadata for Scientific Workflows on Multisite Clouds. BIGDATA 2016 - 2016 IEEE International Conference on Big Data, Dec 2016, Washington, United States. 2016. 〈hal-01395715〉

Partager

Métriques

Consultations de
la notice

503

Téléchargements du document

170