Managing Hot Metadata for Scientific Workflows on Multisite Clouds - Archive ouverte HAL Access content directly
Conference Papers Year : 2016

Managing Hot Metadata for Scientific Workflows on Multisite Clouds

(1, 2) , (2, 3, 4) , (1) , (3, 4) , (1) , (3, 4) , (5)
1
2
3
4
5

Abstract

Large-scale scientific applications are often expressed as workflows that help defining data dependencies between their different components. Several such workflows have huge storage and computation requirements, and so they need to be processed in multiple (cloud-federated) datacenters. It has been shown that efficient metadata handling plays a key role in the performance of computing systems. However, most of this evidence concern only single-site, HPC systems to date. In this paper, we present a hybrid decentralized/distributed model for handling hot metadata (frequently accessed metadata) in multisite architectures. We couple our model with a scientific workflow management system (SWfMS) to validate and tune its applicability to different real-life scientific scenarios. We show that efficient management of hot metadata improves the performance of SWfMS, reducing the workflow execution time up to 50% for highly parallel jobs and avoiding unnecessary cold metadata operations.
Fichier principal
Vignette du fichier
BIGDATA2016-final.pdf (513.91 Ko) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-01395715 , version 1 (11-11-2016)

Identifiers

Cite

Luis Pineda-Morales, Ji Liu, Alexandru Costan, Esther Pacitti, Gabriel Antoniu, et al.. Managing Hot Metadata for Scientific Workflows on Multisite Clouds. Big Data, Dec 2016, Washington, DC, United States. pp.390-397, ⟨10.1109/BigData.2016.7840628⟩. ⟨hal-01395715⟩
818 View
428 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More