Resource-Aware Adaptive Scheduling for MapReduce Clusters

Abstract : We present a resource-aware scheduling technique for MapReduce multi-job workloads that aims at improving resource utilization across machines while observing completion time goals. Existing MapReduce schedulers define a static number of slots to represent the capacity of a cluster, creating a fixed number of execution slots per machine. This abstraction works for homogeneous workloads, but fails to capture the different resource requirements of individual jobs in multi-user environments. Our technique leverages job profiling information to dynamically adjust the number of slots on each machine, as well as workload placement across them, to maximize the resource utilization of the cluster. In addition, our technique is guided by user-provided completion time goals for each job. Source code of our prototype is available at [1].
Type de document :
Communication dans un congrès
Fabio Kon; Anne-Marie Kermarrec. 12th International Middleware Conference (MIDDLEWARE), Dec 2011, Lisbon, Portugal. Springer, Lecture Notes in Computer Science, LNCS-7049, pp.187-207, 2011, Middleware 2011. 〈10.1007/978-3-642-25821-3_10〉
Liste complète des métadonnées

Littérature citée [16 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01597795
Contributeur : Hal Ifip <>
Soumis le : jeudi 28 septembre 2017 - 17:19:38
Dernière modification le : jeudi 28 septembre 2017 - 17:20:03
Document(s) archivé(s) le : vendredi 29 décembre 2017 - 15:57:55

Fichier

978-3-642-25821-3_10_Chapter.p...
Fichiers produits par l'(les) auteur(s)

Licence


Distributed under a Creative Commons Paternité 4.0 International License

Identifiants

Citation

Jordà Polo, Claris Castillo, David Carrera, Yolanda Becerra, Ian Whalley, et al.. Resource-Aware Adaptive Scheduling for MapReduce Clusters. Fabio Kon; Anne-Marie Kermarrec. 12th International Middleware Conference (MIDDLEWARE), Dec 2011, Lisbon, Portugal. Springer, Lecture Notes in Computer Science, LNCS-7049, pp.187-207, 2011, Middleware 2011. 〈10.1007/978-3-642-25821-3_10〉. 〈hal-01597795〉

Partager

Métriques

Consultations de la notice

97

Téléchargements de fichiers

61