Service interruption on Monday 11 July from 12:30 to 13:00: all the sites of the CCSD (HAL, EpiSciences, SciencesConf, AureHAL) will be inaccessible (network hardware connection).
Skip to Main content Skip to Navigation
Journal articles

Threats of a replication crisis in empirical computer science

Abstract : Many areas of computer science research (e.g., performance analysis, software engineering, artificial intelligence, and human-computer interaction) validate research claims by using statistical significance as the standard of evidence. A loss of confidence in statistically significant findings is plaguing other empirical disciplines, yet there has been relatively little debate of this issue and its associated 'replication crisis' in computer science. We review factors that have contributed to the crisis in other disciplines, with a focus on problems stemming from an over-reliance on-and misuse of-null hypothesis significance testing. Computer science research can be greatly improved by following the steps taken by other disciplines, such as using more sophisticated evidentiary criteria, and showing greater openness and transparency through experimental preregistration and data/artifact repositories.
Document type :
Journal articles
Complete list of metadata

Cited literature [42 references]  Display  Hide  Download
Contributor : Pierre Dragicevic Connect in order to contact the contributor
Submitted on : Monday, July 27, 2020 - 1:37:18 PM
Last modification on : Sunday, June 26, 2022 - 2:52:27 AM
Long-term archiving on: : Tuesday, December 1, 2020 - 7:23:48 AM


Files produced by the author(s)



Andy Cockburn, Pierre Dragicevic, Lonni Besançon, Carl Gutwin. Threats of a replication crisis in empirical computer science. Communications of the ACM, Association for Computing Machinery, 2020, 63 (8), pp.70-79. ⟨10.1145/3360311⟩. ⟨hal-02907143⟩



Record views


Files downloads