Skip to Main content Skip to Navigation
New interface
Conference papers

OpenBenchmark: Repeatable and Reproducible Internet of Things Experimentation on Testbeds

Abstract : Experimentation on testbeds with Internet of Things (IoT) devices is hard. The tedious firmware development, the lack of user interfaces, the stochastic nature of the radio channel, the testbed learning curve, are some of the factors that make the evaluation process error prone. The impact of such errors on published results can be quite unfortunate, leading to misconclusions and false common wisdom. The selection of experiment conditions or performance metrics to evaluate one's own proposal may not lead to perfectly fair comparisons with state-of-the-art, either. Our research community is well aware of these problems and is actively working on solutions. We present OpenBenchmark, a cloud-based, reproducible, repeatable and comparable IoT benchmarking service. OpenBenchmark facilitates and improves the IoT experimentation workflow: it runs the experiments on supported testbeds, instruments the supported firmware according to the industry-relevant test scenarios, and collects and processes the experiment data to produce Key Performance Indicators (KPIs). This paper introduces the OpenBenchmark platform, discusses its applicability, design and implementation.
Complete list of metadata

Cited literature [17 references]  Display  Hide  Download
Contributor : Thomas Watteyne Connect in order to contact the contributor
Submitted on : Wednesday, August 14, 2019 - 7:32:29 PM
Last modification on : Tuesday, October 25, 2022 - 4:16:56 PM
Long-term archiving on: : Thursday, January 9, 2020 - 3:34:17 PM


Files produced by the author(s)


  • HAL Id : hal-02266556, version 1



Malisa Vucinic, Bozidar Skrbic, Enis Kocan, Milica Pejanovic-Djurisic, Thomas Watteyne. OpenBenchmark: Repeatable and Reproducible Internet of Things Experimentation on Testbeds. CNERT - IEEE INFOCOM Workshop on Computer and Networking Experimental Research using Testbeds, Apr 2019, Paris, France. ⟨hal-02266556⟩



Record views


Files downloads