Adaptive File Management for Scientific Workflows on the Azure Cloud

Radu Tudoran 1 Alexandru Costan 1 Rad Ramin Rezai 2 Goetz Brasche 2 Gabriel Antoniu 1
1 KerData - Scalable Storage for Clouds and Beyond
Inria Rennes – Bretagne Atlantique , IRISA-D1 - SYSTÈMES LARGE ÉCHELLE
2 Cloud Team
EMIC - European Microsoft Innovation Center
Abstract : Scientific workflows typically communicate data between tasks using files. Currently, on public clouds, this is achieved by using the cloud storage services, which are unable to exploit the workflow semantics and are subject to low throughput and high latencies. To overcome these limitations, we propose an alternative leveraging data locality through direct file transfers between the compute nodes. We rely on the observation that workflows generate a set of common data access patterns that our solution exploits in conjunction with context information to self-adapt, choose the most adequate transfer protocol and expose the data layout within the virtual machines to the workflow engines. This file management system was integrated within the Microsoft Generic Worker workflow engine and was validated using synthetic benchmarks and a real-life application on the Azure cloud. The results show it can bring significant performance gains: up to 5x file transfer speedup compared to solutions based on standard cloud storage and over 25% application timespan reduction compared to Hadoop on Azure.
Type de document :
Communication dans un congrès
IEEE Big Data, Oct 2013, Santa Clara, United States. IEEE, pp.273 - 281, 2013
Liste complète des métadonnées

Littérature citée [22 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-00926748
Contributeur : Radu Tudoran <>
Soumis le : vendredi 10 janvier 2014 - 10:57:26
Dernière modification le : mercredi 16 mai 2018 - 11:23:28
Document(s) archivé(s) le : jeudi 10 avril 2014 - 22:26:54

Fichier

bare_conf.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-00926748, version 1

Citation

Radu Tudoran, Alexandru Costan, Rad Ramin Rezai, Goetz Brasche, Gabriel Antoniu. Adaptive File Management for Scientific Workflows on the Azure Cloud. IEEE Big Data, Oct 2013, Santa Clara, United States. IEEE, pp.273 - 281, 2013. 〈hal-00926748〉

Partager

Métriques

Consultations de la notice

957

Téléchargements de fichiers

356