HybridMR: a New Approach for Hybrid MapReduce Combining Desktop Grid and Cloud Infrastructures - Archive ouverte HAL Access content directly
Journal Articles Concurrency and Computation: Practice and Experience Year : 2015

HybridMR: a New Approach for Hybrid MapReduce Combining Desktop Grid and Cloud Infrastructures

Abstract

This paper introduces HybridMR, a novel model for the execution of MapReduce (MR) computation on hybrid computing environment. Using this model, high performance cloud resources and heterogeneous desktop personal computers (PCs) in Internet or Intranet can be integrated to form a hybrid computing environment. Thanks to HybridMR, the computation and storage capability of large scale desktop PCs can be fully utilized to process large scale datasets. HybridMR relies on two innovative solutions to enable such large scale data-intensive computation. The first one is HybridDFS, which is a hybrid distributed file system. HybridDFS features reliable distributed storage that alleviates the volatility of desktop PCs, thanks to fault tolerance and file replication mechanism. The second innovation is a new node priority-based fair scheduling (NPBFS) algorithm has been developed in HybridMR to achieve both data storage balance and job assignment balance by assigning each node a priority through quantifying CPU speed, memory size, and input and output capacity. In this paper, we describe the HybridMR, HybridDFS, and NPBFS. We report on performance evaluation results, which show that the proposed HybridMR not only achieves reliable MR computation, reduces task response time, and improves the performance of MR, but also reduces the computation cost and achieves a greener computing mode.
Fichier principal
Vignette du fichier
mapreduce_cpe_15.pdf (13.42 Mo) Télécharger le fichier
Origin : Files produced by the author(s)
Loading...

Dates and versions

hal-01239299 , version 1 (07-12-2015)

Licence

Attribution - NonCommercial - ShareAlike - CC BY 4.0

Identifiers

Cite

Bing Tang, Haiwu He, Gilles Fedak. HybridMR: a New Approach for Hybrid MapReduce Combining Desktop Grid and Cloud Infrastructures. Concurrency and Computation: Practice and Experience, 2015, 27 (16), pp.16. ⟨10.1002/cpe.3515⟩. ⟨hal-01239299⟩
1239 View
218 Download

Altmetric

Share

Gmail Facebook Twitter LinkedIn More