Resource Provisioning Framework for MapReduce Jobs with Performance Goals

Abstract : Many companies are increasingly using MapReduce for efficient large scale data processing such as personalized advertising, spam detection, and different data mining tasks. Cloud computing offers an attractive option for businesses to rent a suitable size Hadoop cluster, consume resources as a service, and pay only for resources that were utilized. One of the open questions in such environments is the amount of resources that a user should lease from the service provider. Often, a user targets specific performance goals and the application needs to complete data processing by a certain time deadline. However, currently, the task of estimating required resources to meet application performance goals is solely the users’ responsibility. In this work, we introduce a novel framework and technique to address this problem and to offer a new resource sizing and provisioning service in MapReduce environments. For a MapReduce job that needs to be completed within a certain time, the job profile is built from the job past executions or by executing the application on a smaller data set using an automated profiling tool. Then, by applying scaling rules combined with a fast and efficient capacity planning model, we generate a set of resource provisioning options. Moreover, we design a model for estimating the impact of node failures on a job completion time to evaluate worst case scenarios. We validate the accuracy of our models using a set of realistic applications. The predicted completion times of generated resource provisioning options are within 10% of the measured times in our 66-node Hadoop cluster.
Type de document :
Communication dans un congrès
Fabio Kon; Anne-Marie Kermarrec. 12th International Middleware Conference (MIDDLEWARE), Dec 2011, Lisbon, Portugal. Springer, Lecture Notes in Computer Science, LNCS-7049, pp.165-186, 2011, Middleware 2011. 〈10.1007/978-3-642-25821-3_9〉
Liste complète des métadonnées

Littérature citée [26 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01597764
Contributeur : Hal Ifip <>
Soumis le : jeudi 28 septembre 2017 - 17:11:34
Dernière modification le : jeudi 28 septembre 2017 - 17:16:52
Document(s) archivé(s) le : vendredi 29 décembre 2017 - 16:14:29

Fichier

978-3-642-25821-3_9_Chapter.pd...
Fichiers produits par l'(les) auteur(s)

Licence


Distributed under a Creative Commons Paternité 4.0 International License

Identifiants

Citation

Abhishek Verma, Ludmila Cherkasova, Roy Campbell. Resource Provisioning Framework for MapReduce Jobs with Performance Goals. Fabio Kon; Anne-Marie Kermarrec. 12th International Middleware Conference (MIDDLEWARE), Dec 2011, Lisbon, Portugal. Springer, Lecture Notes in Computer Science, LNCS-7049, pp.165-186, 2011, Middleware 2011. 〈10.1007/978-3-642-25821-3_9〉. 〈hal-01597764〉

Partager

Métriques

Consultations de la notice

25

Téléchargements de fichiers

8