Self-Balancing Job Parallelism and Throughput in Hadoop

Abstract : In Hadoop cluster, the performance and the resource consumption of MapReduce jobs do not only depend on the characteristics of these applications and workloads, but also on the appropriate setting of Hadoop configuration parameters. However, when the job workloads are not known a priori or they evolve over time, a static configuration may quickly lead to a waste of computing resources and consequently to a performance degradation. In this paper, we therefore propose an on-line approach that dynamically reconfigures Hadoop at runtime. Concretely, we focus on balancing the job parallelism and throughput by adjusting Hadoop capacity scheduler memory configuration. Our evaluation shows that the approach outperforms vanilla Hadoop deployments by up to 40% and the best statically profiled configurations by up to 13%.
Type de document :
Communication dans un congrès
Márk Jelasity; Evangelia Kalyvianaki. 16th IFIP WG 6.1 International Conference on Distributed Applications and Interoperable Systems (DAIS), Jun 2016, Heraklion, Crete, Greece. Springer, Lecture Notes in Computer Science, LNCS-9687, pp.129-143, Distributed Applications and Interoperable Systems. 〈http://2016.discotec.org〉. 〈10.1007/978-3-319-39577-7_11〉
Liste complète des métadonnées

Littérature citée [22 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01294834
Contributeur : Romain Rouvoy <>
Soumis le : mardi 14 juin 2016 - 09:19:19
Dernière modification le : vendredi 17 novembre 2017 - 08:50:20
Document(s) archivé(s) le : jeudi 15 septembre 2016 - 10:23:57

Fichier

zhang-dais16.pdf
Fichiers produits par l'(les) auteur(s)

Licence


Distributed under a Creative Commons Paternité 4.0 International License

Identifiants

Citation

Bo Zhang, Filip Křikava, Romain Rouvoy, Lionel Seinturier. Self-Balancing Job Parallelism and Throughput in Hadoop. Márk Jelasity; Evangelia Kalyvianaki. 16th IFIP WG 6.1 International Conference on Distributed Applications and Interoperable Systems (DAIS), Jun 2016, Heraklion, Crete, Greece. Springer, Lecture Notes in Computer Science, LNCS-9687, pp.129-143, Distributed Applications and Interoperable Systems. 〈http://2016.discotec.org〉. 〈10.1007/978-3-319-39577-7_11〉. 〈hal-01294834〉

Partager

Métriques

Consultations de la notice

635

Téléchargements de fichiers

179