Code Coverage and Postrelease Defects: A Large-Scale Study on Open Source Projects

Pavneet Singh Kochhar 1 David Lo 1 Julia Lawall 2 Nachiappan Nagappan 3
1 SIS - Singapore Management University
2 Whisper - Well Honed Infrastructure Software for Programming Environments and Runtimes
LIP6 - Laboratoire d'Informatique de Paris 6, Inria de Paris
3 Microsoft Research
Abstract : Testing is a pivotal activity in ensuring the quality of software. Code coverage is a common metric used as a yardstick to measure the efficacy and adequacy of testing. However, does higher coverage actually lead to a decline in post-release bugs? Do files that have higher test coverage actually have fewer bug reports? The direct relationship between code coverage and actual bug reports has not yet been analysed via a comprehensive empirical study on real bugs. Past studies only involve a few software systems or artificially injected bugs (mutants). In this empirical study, we examine these questions in the context of open-source software projects based on their actual reported bugs. We analyze 100 large open-source Java projects and measure the code coverage of the test cases that come along with these projects. We collect real bugs logged in the issue tracking system after the release of the software and analyse the correlations between code coverage and these bugs. We also collect other metrics such as cyclomatic complexity and lines of code, which are used to normalize the number of bugs and coverage to correlate with other metrics as well as use these metrics in regression analysis. Our results show that coverage has an insignificant correlation with the number of bugs that are found after the release of the software at the project level, and no such correlation at the file level.
Keywords : open-source sofware testing post-release defects code coverage Empirical study
Type de document :
Article dans une revue
IEEE Transactions on Reliability, Institute of Electrical and Electronics Engineers, 2017, 66 (4), pp.1213 - 1228. 〈10.1109/TR.2017.2727062〉
Liste complète des métadonnées

Littérature citée [40 références]  Voir  Masquer  Télécharger
https://hal.inria.fr/hal-01653728
Contributeur : Julia Lawall <>
Soumis le : vendredi 1 décembre 2017 - 18:18:41
Dernière modification le : vendredi 31 août 2018 - 09:25:58

Fichier

coverage.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

Collections

UPMC | INRIA | LIP6 | INRIA2017

Citation

Pavneet Singh Kochhar, David Lo, Julia Lawall, Nachiappan Nagappan. Code Coverage and Postrelease Defects: A Large-Scale Study on Open Source Projects. IEEE Transactions on Reliability, Institute of Electrical and Electronics Engineers, 2017, 66 (4), pp.1213 - 1228. 〈10.1109/TR.2017.2727062〉. 〈hal-01653728〉

Exporter

BibTeX TEI DC DCterms EndNote

Partager

Métriques

Consultations de la notice

135

Téléchargements de fichiers

92

Contact


archive-ouverte@inria.fr