Chronos: Failure-Aware Scheduling in Shared Hadoop Clusters

Orcun Yildiz 1 Shadi Ibrahim 1 Tran Anh Phuong 1 Gabriel Antoniu 1
1 KerData - Scalable Storage for Clouds and Beyond
Inria Rennes – Bretagne Atlantique , IRISA-D1 - SYSTÈMES LARGE ÉCHELLE
Abstract : Hadoop emerged as the de facto state-of-the-art system for MapReduce-based data analytics. The reliability of Hadoop systems depends in part on how well they handle failures. Currently, Hadoop handles machine failures by re-executing all the tasks of the failed machines (i.e., executing recovery tasks). Unfortunately, this elegant solution is entirely entrusted to the core of Hadoop and hidden from Hadoop schedulers. The unawareness of failures therefore may prevent Hadoop schedulers from operating correctly towards meeting their objectives (e.g., fairness, job priority) and can significantly impact the performance of MapReduce applications. This paper presents Chronos, a failure-aware scheduling strategy that enables an early yet smart action for fast failure recovery while still operating within a specific scheduler objective. Upon failure detection, rather than waiting an uncertain amount of time to get resources for recovery tasks, Chronos leverages a lightweight preemption technique to carefully allocate these resources. In addition, Chronos considers data locality when scheduling recovery tasks to further improve the performance. We demonstrate the utility of Chronos by combining it with Fifo and Fair schedulers. The experimental results show that Chronos recovers to a correct scheduling behavior within a couple of seconds only and reduces the job completion times by up to 55% compared to state-of-the-art schedulers.
Type de document :
Communication dans un congrès
BigData'15-The 2015 IEEE International Conference on Big Data, Oct 2015, Santa Clara, CA, United States
Liste complète des métadonnées

Littérature citée [12 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01203001
Contributeur : Shadi Ibrahim <>
Soumis le : mardi 22 septembre 2015 - 10:26:20
Dernière modification le : mercredi 11 avril 2018 - 02:01:09
Document(s) archivé(s) le : mardi 29 décembre 2015 - 07:01:00

Fichier

IEEEBigData2015.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01203001, version 1

Citation

Orcun Yildiz, Shadi Ibrahim, Tran Anh Phuong, Gabriel Antoniu. Chronos: Failure-Aware Scheduling in Shared Hadoop Clusters. BigData'15-The 2015 IEEE International Conference on Big Data, Oct 2015, Santa Clara, CA, United States. 〈hal-01203001〉

Partager

Métriques

Consultations de la notice

646

Téléchargements de fichiers

285