Skip to Main content Skip to Navigation
New interface
Conference papers

An Eye on the Elephant in the Wild: A Performance Evaluation of Hadoop's Schedulers Under Failures

Shadi Ibrahim 1 Tran Anh Phuong 1 Gabriel Antoniu 1 
1 KerData - Scalable Storage for Clouds and Beyond
Inria Rennes – Bretagne Atlantique , IRISA-D1 - SYSTÈMES LARGE ÉCHELLE
Abstract : Large-scale data analysis has increasingly come to rely on MapReduce and its open-source implementation Hadoop. Recently, Hadoop has not only been used for running single batch jobs but it has also been optimized to simultaneously support the execution of multiple jobs belonging to multiple concurrent users. Several schedulers (i.e., Fifo, Fair, and Capacity schedulers) have been proposed to optimize locality executions of tasks but do not consider failures, although, evidence in the literature shows that faults do occur and can probably result in performance problems. In this paper, we have designed a set of experiments to evaluate the performance of Hadoop under failure when applying several schedulers (i.e., explore the conflict between job scheduling, exposing locality executions, and failures). Our results reveal several drawbacks of current Hadoop's mechanism in prioritizing failed tasks. By trying to launch failed tasks as soon as possible regardless of locality, it significantly increases the execution time of jobs with failed tasks, due to two reasons: 1) available resources might not be freed up as quickly as expected and 2) failed tasks might be re-executed on machines with no data on it, introducing extra cost for data transferring through network, which is normally the most scarce resource in today's data-centers. Our preliminary study with Hadoop not only helps us to understand the interplay between fault-tolerance and job scheduling, but also offers useful insights into optimizing the current schedulers to be more efficient in case of failures.
Complete list of metadata

Cited literature [24 references]  Display  Hide  Download
Contributor : Shadi Ibrahim Connect in order to contact the contributor
Submitted on : Thursday, August 13, 2015 - 3:03:20 PM
Last modification on : Thursday, January 20, 2022 - 5:33:22 PM
Long-term archiving on: : Saturday, November 14, 2015 - 10:31:59 AM


Files produced by the author(s)


  • HAL Id : hal-01184236, version 1


Shadi Ibrahim, Tran Anh Phuong, Gabriel Antoniu. An Eye on the Elephant in the Wild: A Performance Evaluation of Hadoop's Schedulers Under Failures. ARMS-CC'15-The second workshop on Adaptive Resource Management and Scheduling for Cloud Computing, held in conjunction with PODC 2015,, Jul 2015, Donostia-San Sebastián, Spain. ⟨hal-01184236⟩



Record views


Files downloads