Skip to Main content Skip to Navigation
Journal articles

Design and Analysis of the NIPS 2016 Review Process

Abstract : Neural Information Processing Systems (NIPS) is a top-tier annual conference in machine learning. The 2016 edition of the conference comprised more than 2,400 paper submissions, 3,000 reviewers, and 8,000 attendees. This represents a growth of nearly 40% in terms of submissions, 96% in terms of reviewers, and over 100% in terms of attendees as compared to the previous year. The massive scale as well as rapid growth of the conference calls for a thorough quality assessment of the peer-review process and novel means of improvement. In this paper, we analyze several aspects of the data collected during the review process, including an experiment investigating the efficacy of collecting ordinal rankings from reviewers. We make a number of key observations, provide suggestions that may be useful for subsequent conferences, and discuss open problems towards the goal of improving peer review.
Document type :
Journal articles
Complete list of metadata

Cited literature [39 references]  Display  Hide  Download

https://hal.inria.fr/hal-01991580
Contributor : Marc Schoenauer <>
Submitted on : Wednesday, January 23, 2019 - 10:32:09 PM
Last modification on : Sunday, May 2, 2021 - 3:30:11 AM

File

17-511.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01991580, version 1

Citation

Nihar Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, Ulrike von Luxburg. Design and Analysis of the NIPS 2016 Review Process. Journal of Machine Learning Research, Microtome Publishing, 2018, 19, pp.1 - 34. ⟨hal-01991580⟩

Share

Metrics

Record views

66

Files downloads

73