Skip to Main content Skip to Navigation

Preserving Fairness in Shared Hadoop Cluster: A Study on the Impact of (Non-) Preemptive Approaches

Abstract : Recently, MapReduce and its open-source implementation Hadoop have emerged as prevalent tools for big data analysis in the cloud. Fair resource allocation in-between jobs and users is an important issue, especially in multi-tenant environments such as clouds. Thus several scheduling policies have been developed to preserve fairness in multi-tenant Hadoop clusters. At the core of these schedulers, simple (non-) preemptive approaches are employed to free resources for tasks belonging to jobs with less-share. For example, Hadoop Fair Scheduler is equipped with two approaches: wait and kill. While wait may introduce a serious violation in fairness, kill may result in a huge waste of resources. Yet, recently some works have introduced new preemption approaches (e.g., pause-resume) in shared Hadoop clusters. To this end, in this work, we closely examine three approaches including wait, kill and pause-resume when Hadoop Fair Scheduler is employed for ensuring fair execution between multiple concurrent jobs. We perform extensive experiments to assess the impact of these approaches on performance and resource utilization while ensuring fairness. Our experimental results bring out the differences between these approaches and illustrate that these approaches are only sub-optimal for different workloads and cluster configurations: the efficiency of achieving fairness and the overall performance varies with the workload composition, resource availability and the cost of the adopted preemption technique.
Complete list of metadata
Contributor : Shadi Ibrahim Connect in order to contact the contributor
Submitted on : Monday, January 4, 2021 - 8:16:45 PM
Last modification on : Wednesday, November 3, 2021 - 8:09:35 AM
Long-term archiving on: : Monday, April 5, 2021 - 6:42:58 PM


Files produced by the author(s)


  • HAL Id : hal-03091371, version 1


Orcun Yildiz, Shadi Ibrahim. Preserving Fairness in Shared Hadoop Cluster: A Study on the Impact of (Non-) Preemptive Approaches. [Research Report] RR-9384, Inria Rennes - Bretagne Atlantique. 2020. ⟨hal-03091371⟩



Les métriques sont temporairement indisponibles