User-Centric vs. System-Centric Evaluation of Recommender Systems

Abstract : Recommender Systems (RSs) aim at helping users search large amounts of contents and identify more effectively the items (products or services) that are likely to be more useful or attractive. The quality of a RS can be defined from two perspectives: system-centric, in which quality measures (e.g., precision, recall) are evaluated using vast datasets of preferences and opinions on items previously collected from users that are not interacting with the RS under study; user-centric, in which user measures are collected from users interacting with the RS under study. Prior research in e-commerce has provided some empirical evidence that system-centric and user-centric quality methods may lead to inconsistent results, e.g., RSs that were “best” according to system-centric measures were not the top ones according to user-centric measures. The paper investigates if a similar mismatch also exists in the domain of e-tourism. We discuss two studies that have adopted a system-centric approach using data from 210000 users, and a user-centric approach involving 240 users interacting with an online hotel booking service. In both studies, we considered four RSs that employ an implicit user preference elicitation technique and different baseline and state-of-the-art recommendation algorithms. In these four experimental conditions, we compared system-centric quality measures against user-centric evaluation results. System-centric quality measures were consistent with user-centric measures, in contrast with past studies in e-commerce. This pinpoints that the relationship between the two kinds of metrics may depend on the business sector, is more complex that we may expect, and is a challenging issues that deserves further research.
Type de document :
Communication dans un congrès
Paula Kotzé; Gary Marsden; Gitte Lindgaard; Janet Wesson; Marco Winckler. 14th International Conference on Human-Computer Interaction (INTERACT), Sep 2013, Cape Town, South Africa. Springer, Lecture Notes in Computer Science, LNCS-8119 (Part III), pp.334-351, 2013, Human-Computer Interaction – INTERACT 2013. 〈10.1007/978-3-642-40477-1_21〉
Liste complète des métadonnées

Littérature citée [34 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-01504894
Contributeur : Hal Ifip <>
Soumis le : lundi 10 avril 2017 - 16:30:04
Dernière modification le : mardi 11 avril 2017 - 08:55:00
Document(s) archivé(s) le : mardi 11 juillet 2017 - 14:07:37

Fichier

978-3-642-40477-1_21_Chapter.p...
Fichiers produits par l'(les) auteur(s)

Licence


Distributed under a Creative Commons Paternité 4.0 International License

Identifiants

Citation

Paolo Cremonesi, Franca Garzotto, Roberto Turrin. User-Centric vs. System-Centric Evaluation of Recommender Systems. Paula Kotzé; Gary Marsden; Gitte Lindgaard; Janet Wesson; Marco Winckler. 14th International Conference on Human-Computer Interaction (INTERACT), Sep 2013, Cape Town, South Africa. Springer, Lecture Notes in Computer Science, LNCS-8119 (Part III), pp.334-351, 2013, Human-Computer Interaction – INTERACT 2013. 〈10.1007/978-3-642-40477-1_21〉. 〈hal-01504894〉

Partager

Métriques

Consultations de la notice

133

Téléchargements de fichiers

446