Skip to Main content Skip to Navigation
Journal articles

Code review poses extra challenges

Nicolas Rougier 1 Konrad Hinsen 2, 3
1 Mnemosyne - Mnemonic Synergy
LaBRI - Laboratoire Bordelais de Recherche en Informatique, Inria Bordeaux - Sud-Ouest, IMN - Institut des Maladies Neurodégénératives [Bordeaux]
Abstract : As computational scientists who are unsatisfied with the current level of quality control concerning computational work, we applaud Nature's recently announced initiative to encourage the submission of code accompanying an article, and have it checked by reviewers. We hope that more journals will follow Nature's lead. As editors-in-chief of ReScience (, a journal dedicated to computational replications, we regularly supervise code reviews. Our experience is that they require significant changes to the traditional reviewing process. An effective code review is possible only if reviewers can both run and inspect the software. We therefore consider the submission of source code mandatory. Moreover, inspection of the source code requires that reviewers have experience with software development in addition to scientific domain expertise. ReScience selects reviewers both according to their scientific discipline and according to their familiarity with programming languages. The first step of a code review consists of reproducing the figures and/or tables from the submitted article using the suppled software. This requires that the authors provide all input files and datasets, and clear instructions. Nevertheless, in our experience, technical problems in installing and running scientific software are inevitable. They are resolved most efficiently if authors and reviewers can communicate rapidly and interactively, without the intervention of the editorial staff. ReScience uses an open reviewing approach conducted on the GitHub platform, which has proven to be very effective. A more detailed account of our experience with publishing computational replications in ReScience can be found in PeerJ Computer Science. 2017 Dec 18;3:e142 (doi:10.7717/peerjcs.142). The currently ongoing reviews at ReScience can be viewed at
Document type :
Journal articles
Complete list of metadata
Contributor : Nicolas P. Rougier Connect in order to contact the contributor
Submitted on : Monday, April 23, 2018 - 11:59:44 AM
Last modification on : Tuesday, October 12, 2021 - 5:20:13 PM

Links full text



Nicolas Rougier, Konrad Hinsen. Code review poses extra challenges. Nature, Nature Publishing Group, 2018, 556 (7701), pp.309. ⟨10.1038/d41586-018-04628-w⟩. ⟨hal-01773411⟩



Record views