Tune and mix: learning to rank using ensembles of calibrated multi-class classifiers

Róbert Busa-Fekete 1, 2 Balázs Kégl 3, 1, 4 Tamás Éltetõ 5 György Szarvas 6
3 TAO - Machine Learning and Optimisation
LRI - Laboratoire de Recherche en Informatique, UP11 - Université Paris-Sud - Paris 11, Inria Saclay - Ile de France, CNRS - Centre National de la Recherche Scientifique : UMR8623
Abstract : In subset ranking, the goal is to learn a ranking function that approximates a gold standard partial ordering of a set of objects (in our case, a set of documents retrieved for the same query). The partial ordering is given by relevance labels representing the relevance of documents with respect to the query on an absolute scale. Our approach consists of three simple steps. First, we train standard multi-class classifiers (AdaBoost.MH and multi-class SVM) to discriminate between the relevance labels. Second, the posteriors of multi-class classifiers are calibrated using probabilistic and regression losses in order to estimate the Bayes-scoring function which optimizes the Normalized Discounted Cumulative Gain (NDCG). In the third step, instead of selecting the best multi-class hyperparameters and the best calibration, we mix all the learned models in a simple ensemble scheme. Our extensive experimental study is itself a substantial contribution. We compare most of the existing learning-to-rank techniques on all of the available large-scale benchmark data sets using a standardized implementation of the NDCG score. We show that our approach is competitive with conceptually more complex listwise and pairwise methods, and clearly outperforms them as the data size grows. As a technical contribution, we clarify some of the confusing results related to the ambiguities of the evaluation tools, and propose guidelines for future studies.
Type de document :
Article dans une revue
Machine Learning, Springer Verlag, 2013, 93 (2-3), pp.261-292. 〈10.1007/s10994-013-5360-9〉
Liste complète des métadonnées

Contributeur : Sabine Starita <>
Soumis le : vendredi 4 octobre 2013 - 10:51:05
Dernière modification le : samedi 7 avril 2018 - 14:30:02

Lien texte intégral




Róbert Busa-Fekete, Balázs Kégl, Tamás Éltetõ, György Szarvas. Tune and mix: learning to rank using ensembles of calibrated multi-class classifiers. Machine Learning, Springer Verlag, 2013, 93 (2-3), pp.261-292. 〈10.1007/s10994-013-5360-9〉. 〈in2p3-00869803〉



Consultations de la notice