Optimal Minimisation of Pairwise-covering Test Configurations Using Constraint Programming

Aymeric Hervieu 1 Dusica Marijan 2 Arnaud Gotlieb 2 Benoit Baudry 1
1 DiverSe - Diversity-centric Software Engineering
Inria Rennes – Bretagne Atlantique , IRISA-D4 - LANGAGE ET GÉNIE LOGICIEL
Abstract : Context: Testing highly-configurable software systems is challenging due to a large number of test configurations that have to be carefully selected in order to reduce the testing effort as much as possible, while maintaining high software quality. Finding the smallest set of valid test configurations that ensure sufficient coverage of the system's feature interactions is thus the objective of validation engineers, especially when the execution of test configurations is costly or time-consuming. However, this problem is NP-hard in general and approximation algorithms have often been used to address it in practice. Objective: In this paper, we explore an alternative approach based on constraint programming that will allow engineers to increase the effectiveness of configuration testing while keeping the number of configurations as low as possible. Method: Our approach consists in using a (time-aware) minimisation algorithm based on constraint programming. Given the amount of time, our solution generates a minimised set of valid test configurations that ensure coverage of all pairs of feature values (a.k.a. pairwise coverage). The approach has been implemented in a tool called PACOGEN. Results: PACOGEN was evaluated on 224 feature models in comparison with the two existing tools that are based on a greedy algorithm. For 79% of 224 feature models, PACOGEN generated up to 60% fewer test configurations than the competitor tools. We further evaluated PACOGEN in the case study of large industrial highly-configurable video conferencing software with a feature model of 169 features, and found 60% fewer configurations compared with the manual approach followed by test engineers. The set of test configurations generated by PACOGEN decreased the time required by test engineers in manual test configuration by 85%, increasing the feature-pairs coverage at the same time. Conclusion: Extensive evaluation concluded that optimal minimisation of pairwise-covering test configurations is efficiently addressed using constraint programming techniques.
Complete list of metadatas

Cited literature [41 references]  Display  Hide  Download

https://hal.inria.fr/hal-01352831
Contributor : Benoit Baudry <>
Submitted on : Wednesday, August 10, 2016 - 9:48:08 AM
Last modification on : Thursday, October 3, 2019 - 3:36:02 PM
Long-term archiving on : Friday, November 11, 2016 - 10:30:22 AM

File

IST_journal.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01352831, version 1

Citation

Aymeric Hervieu, Dusica Marijan, Arnaud Gotlieb, Benoit Baudry. Optimal Minimisation of Pairwise-covering Test Configurations Using Constraint Programming. Information and Software Technology, Elsevier, 2016, 71, pp.129-146. ⟨hal-01352831⟩

Share

Metrics

Record views

1105

Files downloads

282