A Performance Evaluation of Erasure Coding Libraries for Cloud-Based Data Stores

Abstract : Erasure codes have been widely used over the last decade to implement reliable data stores. They offer interesting trade-offs between efficiency, reliability, and storage overhead. Indeed, a distributed data store holding encoded data blocks can tolerate the failure of multiple nodes while requiring only a fraction of the space necessary for plain replication, albeit at an increased encoding and decoding cost. There exists nowadays a number of libraries implementing several variations of erasure codes, which notably differ in terms of complexity and implementation-specific optimizations.Seven years ago, Plank et al. [14] have conducted a comprehensive performance evaluation of open-source erasure coding libraries available at the time to compare their raw performance and measure the impact of different parameter configurations. In the present experimental study, we take a fresh perspective at the state of the art of erasure coding libraries. Not only do we cover a wider set of libraries running on modern hardware, but we also consider their efficiency when used in realistic settings for cloud-based storage, namely when deployed across several nodes in a data centre. Our measurements therefore account for the end-to-end costs of data accesses over several distributed nodes, including the encoding and decoding costs, and shed light on the performance one can expect from the various libraries when deployed in a real system. Our results reveal important differences in the efficiency of the different libraries, notably due to the type of coding algorithm and the use of hardware-specific optimizations.
Complete list of metadatas

Cited literature [18 references]  Display  Hide  Download

https://hal.inria.fr/hal-01434792
Contributor : Hal Ifip <>
Submitted on : Friday, January 13, 2017 - 2:02:29 PM
Last modification on : Wednesday, November 28, 2018 - 2:48:22 PM
Long-term archiving on : Friday, April 14, 2017 - 7:46:41 PM

File

416479_1_En_13_Chapter.pdf
Files produced by the author(s)

Licence


Distributed under a Creative Commons Attribution 4.0 International License

Identifiers

Citation

Dorian Burihabwa, Pascal Felber, Hugues Mercier, Valerio Schiavoni. A Performance Evaluation of Erasure Coding Libraries for Cloud-Based Data Stores. 16th IFIP WG 6.1 International Conference on Distributed Applications and Interoperable Systems (DAIS), Jun 2016, Heraklion, Crete, Greece. pp.160-173, ⟨10.1007/978-3-319-39577-7_13⟩. ⟨hal-01434792⟩

Share

Metrics

Record views

108

Files downloads

76