OpenBenchmark: Repeatable and Reproducible Internet of Things Experimentation on Testbeds

Abstract : Experimentation on testbeds with Internet of Things (IoT) devices is hard. The tedious firmware development, the lack of user interfaces, the stochastic nature of the radio channel, the testbed learning curve, are some of the factors that make the evaluation process error prone. The impact of such errors on published results can be quite unfortunate, leading to misconclusions and false common wisdom. The selection of experiment conditions or performance metrics to evaluate one's own proposal may not lead to perfectly fair comparisons with state-of-the-art, either. Our research community is well aware of these problems and is actively working on solutions. We present OpenBenchmark, a cloud-based, reproducible, repeatable and comparable IoT benchmarking service. OpenBenchmark facilitates and improves the IoT experimentation workflow: it runs the experiments on supported testbeds, instruments the supported firmware according to the industry-relevant test scenarios, and collects and processes the experiment data to produce Key Performance Indicators (KPIs). This paper introduces the OpenBenchmark platform, discusses its applicability, design and implementation.
Complete list of metadatas

Cited literature [17 references]  Display  Hide  Download

https://hal.inria.fr/hal-02266556
Contributor : Thomas Watteyne <>
Submitted on : Wednesday, August 14, 2019 - 7:32:29 PM
Last modification on : Wednesday, October 2, 2019 - 10:35:31 AM
Long-term archiving on: Thursday, January 9, 2020 - 3:34:17 PM

File

vucinic19openbenchmark.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02266556, version 1

Collections

Citation

Malisa Vucinic, Bozidar Skrbic, Enis Kocan, Milica Pejanovic-Djurisic, Thomas Watteyne. OpenBenchmark: Repeatable and Reproducible Internet of Things Experimentation on Testbeds. CNERT - IEEE INFOCOM Workshop on Computer and Networking Experimental Research using Testbeds, Apr 2019, Paris, France. ⟨hal-02266556⟩

Share

Metrics

Record views

79

Files downloads

301