On Achieving Efficient Data Transfer for Graph Processing in Geo-Distributed Datacenters

Amelie Zhou 1 Shadi Ibrahim 1, 2 Bingsheng He 3
1 ASCOLA - Aspect and Composition Languages
Inria Rennes – Bretagne Atlantique , LS2N - Laboratoire des Sciences du Numérique de Nantes
Abstract : Graph partitioning is important for optimizing the performance and communication cost of large graph processing jobs. Recently, many graph applications such as social networks store their data on geo-distributed datacenters (DCs) to provide services worldwide with low latency. This raises new challenges to existing graph partitioning methods, due to the costly Wide Area Network (WAN) usage and the multi-levels of network heterogeneities in geo-distributed DCs. In this paper, we propose a geo-aware graph partitioning method named G-Cut, which aims at minimizing the inter-DC data transfer time of graph processing jobs in geo-distributed DCs while satisfying the WAN usage budget. G-Cut adopts two novel optimization phases which address the two challenges in WAN usage and network heterogeneities separately. G-Cut can be also applied to partition dynamic graphs thanks to its lightweight runtime overhead. We evaluate the effectiveness and efficiency of G-Cut using real-world graphs with both real geo-distributed DCs and simulations. Evaluation results show that G-Cut can reduce the inter-DC data transfer time by up to 58% and reduce the WAN usage by up to 70% compared to state-of-the-art graph partitioning methods with a low runtime overhead.
Type de document :
Communication dans un congrès
ICDCS'17- The 37th IEEE International Conference on Distributed Computing Systems (ICDCS 2017), Jun 2017, Atlanta, United States
Liste complète des métadonnées

Littérature citée [26 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01560187
Contributeur : Shadi Ibrahim <>
Soumis le : mardi 11 juillet 2017 - 12:13:13
Dernière modification le : mardi 4 septembre 2018 - 11:04:01
Document(s) archivé(s) le : mercredi 24 janvier 2018 - 20:26:08

Fichier

ICDCS-GraphPartitioning.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01560187, version 1

Citation

Amelie Zhou, Shadi Ibrahim, Bingsheng He. On Achieving Efficient Data Transfer for Graph Processing in Geo-Distributed Datacenters. ICDCS'17- The 37th IEEE International Conference on Distributed Computing Systems (ICDCS 2017), Jun 2017, Atlanta, United States. 〈hal-01560187〉

Partager

Métriques

Consultations de la notice

459

Téléchargements de fichiers

142