Optimization of Real-World MapReduce Applications With Flame-MR: Practical Use Cases - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Article Dans Une Revue IEEE Access Année : 2018

Optimization of Real-World MapReduce Applications With Flame-MR: Practical Use Cases

Résumé

Apache Hadoop is a widely used MapReduce framework for storing and processing large amounts of data. However, it presents some performance issues that hinder its utilization in many practical use cases. Although existing alternatives like Spark or Hama can outperform Hadoop, they require to rewrite the source code of the applications due to API incompatibilities. This paper studies the use of Flame-MR, an in-memory processing architecture for MapReduce applications, to improve the performance of real-world use cases in a transparent way while keeping application compatibility. Flame-MR adapts to the characteristics of the workloads, managing efficiently the use of custom data formats and iterative computations, while also reducing workload imbalance. The experimental evaluation, conducted in high performance clusters and the Microsoft Azure cloud, shows a clear outperformance of Flame-MR over Hadoop. In most cases, Flame-MR reduces the execution times by more than a half.
Fichier principal
Vignette du fichier
2_FINAL_Article-2.pdf (606.5 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01955503 , version 1 (14-12-2018)

Identifiants

Citer

Jorge Veiga, Roberto R Expósito, Bruno Raffin, Juan Tourino. Optimization of Real-World MapReduce Applications With Flame-MR: Practical Use Cases. IEEE Access, 2018, 6, pp.69750-69762. ⟨10.1109/ACCESS.2018.2880842⟩. ⟨hal-01955503⟩
131 Consultations
147 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More