HybridMR: a New Approach for Hybrid MapReduce Combining Desktop Grid and Cloud Infrastructures

Abstract : This paper introduces HybridMR, a novel model for the execution of MapReduce (MR) computation on hybrid computing environment. Using this model, high performance cloud resources and heterogeneous desktop personal computers (PCs) in Internet or Intranet can be integrated to form a hybrid computing environment. Thanks to HybridMR, the computation and storage capability of large scale desktop PCs can be fully utilized to process large scale datasets. HybridMR relies on two innovative solutions to enable such large scale data-intensive computation. The first one is HybridDFS, which is a hybrid distributed file system. HybridDFS features reliable distributed storage that alleviates the volatility of desktop PCs, thanks to fault tolerance and file replication mechanism. The second innovation is a new node priority-based fair scheduling (NPBFS) algorithm has been developed in HybridMR to achieve both data storage balance and job assignment balance by assigning each node a priority through quantifying CPU speed, memory size, and input and output capacity. In this paper, we describe the HybridMR, HybridDFS, and NPBFS. We report on performance evaluation results, which show that the proposed HybridMR not only achieves reliable MR computation, reduces task response time, and improves the performance of MR, but also reduces the computation cost and achieves a greener computing mode.
Type de document :
Article dans une revue
Concurrency and Computation: Practice and Experience, Wiley, 2015, 27 (16), pp.16. 〈10.1002/cpe.3515〉
Liste complète des métadonnées

Littérature citée [31 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01239299
Contributeur : Gilles Fedak <>
Soumis le : lundi 7 décembre 2015 - 15:59:15
Dernière modification le : mardi 16 janvier 2018 - 15:30:19
Document(s) archivé(s) le : mardi 8 mars 2016 - 14:45:30

Fichier

mapreduce_cpe_15.pdf
Fichiers produits par l'(les) auteur(s)

Licence


Distributed under a Creative Commons Paternité - Pas d'utilisation commerciale - Partage selon les Conditions Initiales 4.0 International License

Identifiants

Collections

Citation

Bing Tang, Haiwu He, Gilles Fedak. HybridMR: a New Approach for Hybrid MapReduce Combining Desktop Grid and Cloud Infrastructures. Concurrency and Computation: Practice and Experience, Wiley, 2015, 27 (16), pp.16. 〈10.1002/cpe.3515〉. 〈hal-01239299〉

Partager

Métriques

Consultations de la notice

545

Téléchargements de fichiers

199