Skip to Main content Skip to Navigation
Journal articles

HybridMR: a New Approach for Hybrid MapReduce Combining Desktop Grid and Cloud Infrastructures

Abstract : This paper introduces HybridMR, a novel model for the execution of MapReduce (MR) computation on hybrid computing environment. Using this model, high performance cloud resources and heterogeneous desktop personal computers (PCs) in Internet or Intranet can be integrated to form a hybrid computing environment. Thanks to HybridMR, the computation and storage capability of large scale desktop PCs can be fully utilized to process large scale datasets. HybridMR relies on two innovative solutions to enable such large scale data-intensive computation. The first one is HybridDFS, which is a hybrid distributed file system. HybridDFS features reliable distributed storage that alleviates the volatility of desktop PCs, thanks to fault tolerance and file replication mechanism. The second innovation is a new node priority-based fair scheduling (NPBFS) algorithm has been developed in HybridMR to achieve both data storage balance and job assignment balance by assigning each node a priority through quantifying CPU speed, memory size, and input and output capacity. In this paper, we describe the HybridMR, HybridDFS, and NPBFS. We report on performance evaluation results, which show that the proposed HybridMR not only achieves reliable MR computation, reduces task response time, and improves the performance of MR, but also reduces the computation cost and achieves a greener computing mode.
Complete list of metadata

Cited literature [31 references]  Display  Hide  Download
Contributor : Gilles Fedak Connect in order to contact the contributor
Submitted on : Monday, December 7, 2015 - 3:59:15 PM
Last modification on : Saturday, September 11, 2021 - 3:16:57 AM
Long-term archiving on: : Tuesday, March 8, 2016 - 2:45:30 PM


Files produced by the author(s)


Distributed under a Creative Commons Attribution - NonCommercial - ShareAlike 4.0 International License




Bing Tang, Haiwu He, Gilles Fedak. HybridMR: a New Approach for Hybrid MapReduce Combining Desktop Grid and Cloud Infrastructures. Concurrency and Computation: Practice and Experience, Wiley, 2015, 27 (16), pp.16. ⟨10.1002/cpe.3515⟩. ⟨hal-01239299⟩



Les métriques sont temporairement indisponibles