An Eye on the Elephant in the Wild: A Performance Evaluation of Hadoop's Schedulers Under Failures

Shadi Ibrahim 1 Tran Anh Phuong 1 Gabriel Antoniu 1
1 KerData - Scalable Storage for Clouds and Beyond
Inria Rennes – Bretagne Atlantique , IRISA-D1 - SYSTÈMES LARGE ÉCHELLE
Abstract : Large-scale data analysis has increasingly come to rely on MapReduce and its open-source implementation Hadoop. Recently, Hadoop has not only been used for running single batch jobs but it has also been optimized to simultaneously support the execution of multiple jobs belonging to multiple concurrent users. Several schedulers (i.e., Fifo, Fair, and Capacity schedulers) have been proposed to optimize locality executions of tasks but do not consider failures, although, evidence in the literature shows that faults do occur and can probably result in performance problems. In this paper, we have designed a set of experiments to evaluate the performance of Hadoop under failure when applying several schedulers (i.e., explore the conflict between job scheduling, exposing locality executions, and failures). Our results reveal several drawbacks of current Hadoop's mechanism in prioritizing failed tasks. By trying to launch failed tasks as soon as possible regardless of locality, it significantly increases the execution time of jobs with failed tasks, due to two reasons: 1) available resources might not be freed up as quickly as expected and 2) failed tasks might be re-executed on machines with no data on it, introducing extra cost for data transferring through network, which is normally the most scarce resource in today's data-centers. Our preliminary study with Hadoop not only helps us to understand the interplay between fault-tolerance and job scheduling, but also offers useful insights into optimizing the current schedulers to be more efficient in case of failures.
Type de document :
Communication dans un congrès
ARMS-CC'15-The second workshop on Adaptive Resource Management and Scheduling for Cloud Computing, held in conjunction with PODC 2015,, Jul 2015, Donostia-San Sebastián, Spain
Liste complète des métadonnées

Littérature citée [24 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01184236
Contributeur : Shadi Ibrahim <>
Soumis le : jeudi 13 août 2015 - 15:03:20
Dernière modification le : vendredi 16 novembre 2018 - 01:39:20
Document(s) archivé(s) le : samedi 14 novembre 2015 - 10:31:59

Fichier

ARMS-CC2015.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01184236, version 1

Citation

Shadi Ibrahim, Tran Anh Phuong, Gabriel Antoniu. An Eye on the Elephant in the Wild: A Performance Evaluation of Hadoop's Schedulers Under Failures. ARMS-CC'15-The second workshop on Adaptive Resource Management and Scheduling for Cloud Computing, held in conjunction with PODC 2015,, Jul 2015, Donostia-San Sebastián, Spain. 〈hal-01184236〉

Partager

Métriques

Consultations de la notice

679

Téléchargements de fichiers

241