Ontology matching benchmarks: generation, stability, and discriminability

Abstract : The OAEI Benchmark test set has been used for many years as a main reference to evaluate and compare ontology matching systems. However, this test set has barely varied since 2004 and has become a relatively easy task for matchers. In this paper, we present the design of a flexible test generator based on an extensible set of alterators which may be used programmatically for generating different test sets from different seed ontologies and different alteration modalities. It has been used for reproducing Benchmark both with the original seed ontology and with other ontologies. This highlights the remarkable stability of results over different generations and the preservation of difficulty across seed ontologies, as well as a systematic bias towards the initial Benchmark test set and the inability of such tests to identify an overall winning matcher. These were exactly the properties for which Benchmark had been designed. Furthermore, the generator has been used for providing new test sets aiming at increasing the difficulty and discriminability of Benchmark. Although difficulty may be easily increased with the generator, attempts to increase discriminability proved unfruitful. However, efforts towards this goal raise questions about the very nature of discriminability.
Type de document :
Article dans une revue
Journal of Web Semantics, Elsevier, 2013, 21, pp.30-48. 〈10.1016/j.websem.2013.05.002〉
Liste complète des métadonnées

Littérature citée [26 références]  Voir  Masquer  Télécharger

Contributeur : Jérôme Euzenat <>
Soumis le : jeudi 12 décembre 2013 - 16:17:22
Dernière modification le : jeudi 11 octobre 2018 - 08:48:02
Document(s) archivé(s) le : vendredi 14 mars 2014 - 11:26:09


Fichiers produits par l'(les) auteur(s)




Jérôme Euzenat, Maria Rosoiu, Cássia Trojahn dos Santos. Ontology matching benchmarks: generation, stability, and discriminability. Journal of Web Semantics, Elsevier, 2013, 21, pp.30-48. 〈10.1016/j.websem.2013.05.002〉. 〈hal-00917906〉



Consultations de la notice


Téléchargements de fichiers