Skip to Main content Skip to Navigation
Conference papers

Is there a reproducibility crisis around here? Maybe not, but we still need to change

Alex Holcombe 1 Charles Ludowici 1 Steve Haroz 2
2 AVIZ - Analysis and Visualization
LRI - Laboratoire de Recherche en Informatique, Inria Saclay - Ile de France
Abstract : Those of us who study large effects may believe ourselves to be unaffected by the reproducibility problems that plague other areas. However, we will argue that initiatives to address the reproducibility crisis, such as preregistration and data sharing, are worth adopting even under optimistic scenarios of high rates of replication success. We searched the text of articles published in the Journal of Vision from January through October of 2018 for URLs (our code is here: and examined them for raw data, experiment code, analysis code, and preregistrations. We also reviewed the articles' supplemental material. Of the 165 articles, approximately 12% provide raw data, 4% provide experiment code, and 5% provide analysis code. Only one article contained a preregistration. When feasible, preregistration is important because p-values are not interpretable unless the number of comparisons performed is known, and selective reporting appears to be common across fields. In the absence of preregistration, then, and in the context of the low rates of successful replication found across multiple fields, many claims in vision science are shrouded by uncertain credence. Sharing de-identified data, experiment code, and data analysis code not only increases credibility and ameliorates the negative impact of errors, it also accelerates science. Open practices allow researchers to build on others’ work more quickly and with more confidence. Given our results and the broader context of concern by funders, evident in the recent NSF statement that “transparency is a necessary condition when designing scientifically valid research” and “pre-registration… can help ensure the integrity and transparency of the proposed research”, there is much to discuss.
Document type :
Conference papers
Complete list of metadata
Contributor : Steve Haroz Connect in order to contact the contributor
Submitted on : Thursday, December 6, 2018 - 11:30:07 AM
Last modification on : Thursday, July 8, 2021 - 3:50:40 AM


Distributed under a Creative Commons Attribution 4.0 International License


  • HAL Id : hal-01944513, version 1


Alex Holcombe, Charles Ludowici, Steve Haroz. Is there a reproducibility crisis around here? Maybe not, but we still need to change. VSS 2019 - Vision Sciences Society, May 2019, St. Pete Beach, Florida, United States. ⟨hal-01944513⟩



Record views