mpCache: Accelerating MapReduce with Hybrid Storage System on Many-Core Clusters

Abstract : As a widely used programming model and implementation for processing large data sets, MapReduce does not scale well on many-core clusters, which, unfortunately, are common in current data centers. To deal with the problem, this paper: 1) analyzes the causes of poor scalability of MapReduce on many-core clusters and identifies the key one as the underlying low-speed storage (hard disk) can not meet the requirements of frequent IO operations, and 2) proposes mpCache, a SSD based hybrid storage system that caches both Input Data and Localized Data, and dynamically tunes the cache space allocation between them to make full use of the space. mpCache has been incorporated into Hadoop and evaluated on a 7-node cluster by 13 benchmarks. The experimental results show that mpCache gains an average speedup of 2.09 when compared with the original Hadoop, and achieves an average speedup of 1.79 when compared with PACMan, the latest in-memory optimization of MapReduce.
Type de document :
Communication dans un congrès
Ching-Hsien Hsu; Xuanhua Shi; Valentina Salapura. 11th IFIP International Conference on Network and Parallel Computing (NPC), Sep 2014, Ilan, Taiwan. Springer, Lecture Notes in Computer Science, LNCS-8707, pp.220-233, 2014, Network and Parallel Computing. 〈10.1007/978-3-662-44917-2_19〉
Liste complète des métadonnées

Littérature citée [20 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01403087
Contributeur : Hal Ifip <>
Soumis le : vendredi 25 novembre 2016 - 14:31:06
Dernière modification le : vendredi 1 décembre 2017 - 01:10:05
Document(s) archivé(s) le : lundi 20 mars 2017 - 17:31:37

Fichier

978-3-662-44917-2_19_Chapter.p...
Fichiers produits par l'(les) auteur(s)

Licence


Distributed under a Creative Commons Paternité 4.0 International License

Identifiants

Citation

Bo Wang, Jinlei Jiang, Guangwen Yang. mpCache: Accelerating MapReduce with Hybrid Storage System on Many-Core Clusters. Ching-Hsien Hsu; Xuanhua Shi; Valentina Salapura. 11th IFIP International Conference on Network and Parallel Computing (NPC), Sep 2014, Ilan, Taiwan. Springer, Lecture Notes in Computer Science, LNCS-8707, pp.220-233, 2014, Network and Parallel Computing. 〈10.1007/978-3-662-44917-2_19〉. 〈hal-01403087〉

Partager

Métriques

Consultations de la notice

55

Téléchargements de fichiers

10