HAL will be down for maintenance from Friday, June 10 at 4pm through Monday, June 13 at 9am. More information
Skip to Main content Skip to Navigation
Journal articles

Ontology matching benchmarks: generation, stability, and discriminability

Abstract : The OAEI Benchmark test set has been used for many years as a main reference to evaluate and compare ontology matching systems. However, this test set has barely varied since 2004 and has become a relatively easy task for matchers. In this paper, we present the design of a flexible test generator based on an extensible set of alterators which may be used programmatically for generating different test sets from different seed ontologies and different alteration modalities. It has been used for reproducing Benchmark both with the original seed ontology and with other ontologies. This highlights the remarkable stability of results over different generations and the preservation of difficulty across seed ontologies, as well as a systematic bias towards the initial Benchmark test set and the inability of such tests to identify an overall winning matcher. These were exactly the properties for which Benchmark had been designed. Furthermore, the generator has been used for providing new test sets aiming at increasing the difficulty and discriminability of Benchmark. Although difficulty may be easily increased with the generator, attempts to increase discriminability proved unfruitful. However, efforts towards this goal raise questions about the very nature of discriminability.
Complete list of metadata

Cited literature [26 references]  Display  Hide  Download

Contributor : Jérôme Euzenat Connect in order to contact the contributor
Submitted on : Thursday, December 12, 2013 - 4:17:22 PM
Last modification on : Thursday, April 7, 2022 - 3:38:05 AM
Long-term archiving on: : Friday, March 14, 2014 - 11:26:09 AM


Files produced by the author(s)



Jérôme Euzenat, Maria Rosoiu, Cassia Trojahn dos Santos. Ontology matching benchmarks: generation, stability, and discriminability. Journal of Web Semantics, Elsevier, 2013, 21, pp.30-48. ⟨10.1016/j.websem.2013.05.002⟩. ⟨hal-01123792v2⟩



Record views


Files downloads