Design and Analysis of the NIPS 2016 Review Process

Nihar B. Shah 1 Behzad Tabibian 2 Krikamol Muandet 3 Isabelle Guyon 4 Ulrike von Luxburg 5
4 TAU - TAckling the Underspecified
LRI - Laboratoire de Recherche en Informatique, Inria Saclay - Ile de France, CNRS - Centre National de la Recherche Scientifique : UMR8623, UP11 - Université Paris-Sud - Paris 11
Abstract : Neural Information Processing Systems (NIPS) is a top-tier annual conference in machine learning. The 2016 edition of the conference comprised more than 2,400 paper submissions, 3,000 reviewers, and 8,000 attendees, representing a growth of nearly 40% in terms of submissions, 96% in terms of reviewers, and over 100% in terms of attendees as compared to the previous year. In this report, we analyze several aspects of the data collected during the review process, including an experiment investigating the efficacy of collecting ordinal rankings from reviewers (vs. usual scores aka cardinal rankings). Our goal is to check the soundness of the review process we implemented and, in going so, provide insights that may be useful in the design of the review process of subsequent conferences. We introduce a number of metrics that could be used for monitoring improvements when new ideas are introduced.
Complete list of metadatas

https://hal.inria.fr/hal-01668377
Contributor : Isabelle Guyon <>
Submitted on : Wednesday, December 20, 2017 - 8:09:20 AM
Last modification on : Thursday, February 21, 2019 - 2:58:02 PM

Links full text

Identifiers

  • HAL Id : hal-01668377, version 1
  • ARXIV : 1708.09794

Citation

Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, Ulrike von Luxburg. Design and Analysis of the NIPS 2016 Review Process. 2017. ⟨hal-01668377⟩

Share

Metrics

Record views

152