Skip to Main content Skip to Navigation
Journal articles

Threats of a replication crisis in empirical computer science

Abstract : Many areas of computer science research (e.g., performance analysis, software engineering, artificial intelligence, and human-computer interaction) validate research claims by using statistical significance as the standard of evidence. A loss of confidence in statistically significant findings is plaguing other empirical disciplines, yet there has been relatively little debate of this issue and its associated 'replication crisis' in computer science. We review factors that have contributed to the crisis in other disciplines, with a focus on problems stemming from an over-reliance on-and misuse of-null hypothesis significance testing. Computer science research can be greatly improved by following the steps taken by other disciplines, such as using more sophisticated evidentiary criteria, and showing greater openness and transparency through experimental preregistration and data/artifact repositories.
Document type :
Journal articles
Complete list of metadata

Cited literature [42 references]  Display  Hide  Download

https://hal.inria.fr/hal-02907143
Contributor : Pierre Dragicevic <>
Submitted on : Monday, July 27, 2020 - 1:37:18 PM
Last modification on : Saturday, May 1, 2021 - 3:47:00 AM
Long-term archiving on: : Tuesday, December 1, 2020 - 7:23:48 AM

File

Computer_Science_Research_and_...
Files produced by the author(s)

Identifiers

Citation

Andy Cockburn, Pierre Dragicevic, Lonni Besançon, Carl Gutwin. Threats of a replication crisis in empirical computer science. Communications of the ACM, Association for Computing Machinery, 2020, 63 (8), pp.70-79. ⟨10.1145/3360311⟩. ⟨hal-02907143⟩

Share

Metrics

Record views

166

Files downloads

818