A Systematic Review on the Practice of Evaluating Visualization

Abstract : We present an assessment of the state and historic development of evaluation practices as reported in papers published at the IEEE Visualization conference. Our goal is to reflect on a meta-level about evaluation in our community through a systematic understanding of the characteristics and goals of presented evaluations. For this purpose we conducted a systematic review of ten years of evaluations in the published papers using and extending a coding scheme previously established by Lam et al. [2012]. The results of our review include an overview of the most common evaluation goals in the community, how they evolved over time, and how they contrast or align to those of the IEEE Information Visualization conference. In particular, we found that evaluations specific to assessing resulting images and algorithm performance are the most prevalent (with consistently 80-90% of all papers since 1997). However, especially over the last six years there is a steady increase in evaluation methods that include participants, either by evaluating their performances and subjective feedback or by evaluating their work practices and their improved analysis and reasoning capabilities using visual tools. Up to 2010, this trend in the IEEE Visualization conference was much more pronounced than in the IEEE Information Visualization conference which only showed an increasing percentage of evaluation through user performance and experience testing. Since 2011, however, also papers in IEEE Information Visualization show such an increase of evaluations of work practices and analysis as well as reasoning using visual tools. Further, we found that generally the studies reporting requirements analyses and domain-specific work practices are too informally reported which hinders cross-comparison and lowers external validity.
Type de document :
Article dans une revue
IEEE Transactions on Visualization and Computer Graphics, Institute of Electrical and Electronics Engineers, 2013, 19 (12), pp.2818-2827. <10.1109/TVCG.2013.126>
Liste complète des métadonnées



https://hal.inria.fr/hal-00846775
Contributeur : Isenberg Tobias <>
Soumis le : dimanche 25 août 2013 - 09:59:04
Dernière modification le : jeudi 9 février 2017 - 15:47:23
Document(s) archivé(s) le : mardi 26 novembre 2013 - 02:20:11

Fichiers

Isenberg_2013_SRP.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

Collections

Citation

Tobias Isenberg, Petra Isenberg, Jian Chen, Michael Sedlmair, Torsten Möller. A Systematic Review on the Practice of Evaluating Visualization. IEEE Transactions on Visualization and Computer Graphics, Institute of Electrical and Electronics Engineers, 2013, 19 (12), pp.2818-2827. <10.1109/TVCG.2013.126>. <hal-00846775>

Partager

Métriques

Consultations de
la notice

262

Téléchargements du document

728