An RDMA Middleware for Asynchronous Multi-stage Shuffling in Analytical Processing

Abstract : A key component in large scale distributed analytical processing is shuffling, the distribution of data to multiple nodes such that the computation can be done in parallel. In this paper we describe the design and implementation of a communication middleware to support data shuffling for executing multi-stage analytical processing operations in parallel. The middleware relies on RDMA (Remote Direct Memory Access) to provide basic operations to asynchronously exchange data among multiple machines. Experimental results show that the RDMA-based middleware developed can provide a 75 % reduction of the costs of communication operations on parallel analytical processing tasks, when compared with a sockets middleware.
Complete list of metadatas

Cited literature [16 references]  Display  Hide  Download

https://hal.inria.fr/hal-01434800
Contributor : Hal Ifip <>
Submitted on : Friday, January 13, 2017 - 2:02:44 PM
Last modification on : Friday, January 13, 2017 - 2:05:48 PM
Long-term archiving on : Friday, April 14, 2017 - 7:12:43 PM

File

416479_1_En_5_Chapter.pdf
Files produced by the author(s)

Licence


Distributed under a Creative Commons Attribution 4.0 International License

Identifiers

Citation

Rui Gonçalves, José Pereira, Ricardo Jiménez-Peris. An RDMA Middleware for Asynchronous Multi-stage Shuffling in Analytical Processing. 16th IFIP WG 6.1 International Conference on Distributed Applications and Interoperable Systems (DAIS), Jun 2016, Heraklion, Crete, Greece. pp.61-74, ⟨10.1007/978-3-319-39577-7_5⟩. ⟨hal-01434800⟩

Share

Metrics

Record views

49

Files downloads

74