Skip to Main content Skip to Navigation
Journal articles

Tune and mix: learning to rank using ensembles of calibrated multi-class classifiers

Róbert Busa-Fekete 1, 2 Balázs Kégl 3, 1, 4 Tamás Éltetõ 5 György Szarvas 6
3 TAO - Machine Learning and Optimisation
CNRS - Centre National de la Recherche Scientifique : UMR8623, Inria Saclay - Ile de France, UP11 - Université Paris-Sud - Paris 11, LRI - Laboratoire de Recherche en Informatique
Abstract : In subset ranking, the goal is to learn a ranking function that approximates a gold standard partial ordering of a set of objects (in our case, a set of documents retrieved for the same query). The partial ordering is given by relevance labels representing the relevance of documents with respect to the query on an absolute scale. Our approach consists of three simple steps. First, we train standard multi-class classifiers (AdaBoost.MH and multi-class SVM) to discriminate between the relevance labels. Second, the posteriors of multi-class classifiers are calibrated using probabilistic and regression losses in order to estimate the Bayes-scoring function which optimizes the Normalized Discounted Cumulative Gain (NDCG). In the third step, instead of selecting the best multi-class hyperparameters and the best calibration, we mix all the learned models in a simple ensemble scheme. Our extensive experimental study is itself a substantial contribution. We compare most of the existing learning-to-rank techniques on all of the available large-scale benchmark data sets using a standardized implementation of the NDCG score. We show that our approach is competitive with conceptually more complex listwise and pairwise methods, and clearly outperforms them as the data size grows. As a technical contribution, we clarify some of the confusing results related to the ambiguities of the evaluation tools, and propose guidelines for future studies.
Complete list of metadatas

https://hal.inria.fr/in2p3-00869803
Contributor : Sabine Starita <>
Submitted on : Friday, October 4, 2013 - 10:51:05 AM
Last modification on : Tuesday, June 30, 2020 - 10:00:50 AM

Links full text

Identifiers

Collections

Citation

Róbert Busa-Fekete, Balázs Kégl, Tamás Éltetõ, György Szarvas. Tune and mix: learning to rank using ensembles of calibrated multi-class classifiers. Machine Learning, Springer Verlag, 2013, 93 (2-3), pp.261-292. ⟨10.1007/s10994-013-5360-9⟩. ⟨in2p3-00869803⟩

Share

Metrics

Record views

438