Assessing MapReduce for Internet Computing: a Comparison of Hadoop and BitDew-MapReduce

Lu Lu 1 Hai Jin 1 Xuanhua Shi 1 Gilles Fedak 2
1 Cluster and Grid Computing Lab
2 AVALON - Algorithms and Software Architectures for Distributed and HPC Platforms
Inria Grenoble - Rhône-Alpes, LIP - Laboratoire de l'Informatique du Parallélisme
Abstract : MapReduce is emerging as an important programming model for data-intensive application. Adapting this model to desktop grid would allow taking advantage of the vast amount of computing power and distributed storage to execute new range of application able to process enormous amount of data. In 2010, we have presented the first implementation of MapReduce dedicated to Internet Desktop Grid based on the BitDew middleware. In this paper, we present new optimizations to BitDew-MapReduce (BitDew-MR): aggressive task backup, intermediate result backup, task re-execution mitigation and network failure hiding. We propose a new experimental framework which emulates key fundamental aspects of Internet Desktop Grid. Using the framework, we compare BitDew-MR and the open-source Hadoop middleware on Grid5000. Our experimental results show that 1) BitDew-MR successfully passes all the stress-tests of the framework while Hadoop is unable to work in typical wide-area network topology which includes PC hidden behind firewall and NAT; 2) BitDew-MR outperforms Hadoop performances on several aspects: scalability, fairness, resilience to node failures, and network disconnections.
Type de document :
Communication dans un congrès
GRID 2012 - 13th ACM/IEEE International Conference on Grid Computing, Sep 2012, Beijing, China. IEEE, Grid Computing (GRID), 2012 ACM/IEEE 13th International Conference on, pp.76-84, 2012, 〈10.1109/Grid.2012.31〉
Liste complète des métadonnées
https://hal.inria.fr/hal-00757070
Contributeur : Gilles Fedak <>
Soumis le : lundi 26 novembre 2012 - 11:17:15
Dernière modification le : lundi 13 août 2018 - 13:40:06

Lien texte intégral

openaccess

Identifiants

Collections

ENS-LYON | INRIA | LIP | PRIMES_WP5 | PRIMES | UNIV-LYON1

Citation

Lu Lu, Hai Jin, Xuanhua Shi, Gilles Fedak. Assessing MapReduce for Internet Computing: a Comparison of Hadoop and BitDew-MapReduce. GRID 2012 - 13th ACM/IEEE International Conference on Grid Computing, Sep 2012, Beijing, China. IEEE, Grid Computing (GRID), 2012 ACM/IEEE 13th International Conference on, pp.76-84, 2012, 〈10.1109/Grid.2012.31〉. 〈hal-00757070〉

Exporter

BibTeX TEI DC DCterms EndNote

Partager

Métriques

Consultations de la notice

249

Contact


archive-ouverte@inria.fr