Skip to Main content Skip to Navigation
Conference papers

FLEX: A Slot Allocation Scheduling Optimizer for MapReduce Workloads

Abstract : Originally, MapReduce implementations such as Hadoop employed First In First Out (fifo) scheduling, but such simple schemes cause job starvation. The Hadoop Fair Scheduler (hfs) is a slot-based MapReduce scheme designed to ensure a degree of fairness among the jobs, by guaranteeing each job at least some minimum number of allocated slots. Our prime contribution in this paper is a different, flexible scheduling allocation scheme, known as flex. Our goal is to optimize any of a variety of standard scheduling theory metrics (response time, stretch, makespan and Service Level Agreements (slas), among others) while ensuring the same minimum job slot guarantees as in hfs, and maximum job slot guarantees as well. The flex allocation scheduler can be regarded as an add-on module that works synergistically with hfs. We describe the mathematical basis for flex, and compare it with fifo and hfs in a variety of experiments.
Document type :
Conference papers
Complete list of metadata

Cited literature [17 references]  Display  Hide  Download

https://hal.inria.fr/hal-01055274
Contributor : Hal Ifip <>
Submitted on : Tuesday, August 12, 2014 - 11:47:14 AM
Last modification on : Wednesday, August 16, 2017 - 5:20:50 PM
Long-term archiving on: : Wednesday, November 26, 2014 - 10:41:48 PM

File

middleware10-cr.pdf
Files produced by the author(s)

Licence


Distributed under a Creative Commons Attribution 4.0 International License

Identifiers

Citation

Joel Wolf, Deepak Rajan, Kirsten Hildrum, Rohit Khandekar, Vibhore Kumar, et al.. FLEX: A Slot Allocation Scheduling Optimizer for MapReduce Workloads. ACM/IFIP/USENIX 11th International Middleware Conference (MIDDLEWARE), Nov 2010, Bangalore, India. pp.1-20, ⟨10.1007/978-3-642-16955-7_1⟩. ⟨hal-01055274⟩

Share

Metrics

Record views

348

Files downloads

877