Learning Semantic and Visual Similarity for Endomicroscopy Video Retrieval

Abstract : Traditional Content-Based Image Retrieval (CBIR) systems only deliver visual outputs that are not directly interpretable by the physicians. Our objective is to provide a system for endomicroscopy video retrieval which delivers both visual and semantic outputs that are consistent with each other. In a previous study, we developed an adapted bag-of-visual-words method for endomicroscopy retrieval that computes a visual signature for each video. In this study, we first leverage semantic ground-truth data to transform these visual signatures into semantic signatures that reflect how much the presence of each semantic concept is expressed by the visual words describing the videos. Using cross-validation, we demonstrate that our visual-word-based semantic signatures enable a recall performance which is significantly higher than those of several state-of-the-art methods in CBIR. In a second step, we propose to improve retrieval relevance by learning, from a perceived similarity ground truth, an adjusted similarity distance. Our distance learning method allows to improve, with statistical significance, the correlation with the perceived similarity. Our resulting retrieval system is efficient in providing both visual and semantic information that are correlated with each other and clinically interpretable by the endoscopists.
Type de document :
[Research Report] RR-7722, INRIA. 2011
Liste complète des métadonnées

Littérature citée [24 références]  Voir  Masquer  Télécharger

Contributeur : Barbara André <>
Soumis le : mercredi 31 août 2011 - 15:47:15
Dernière modification le : jeudi 11 janvier 2018 - 16:39:44
Document(s) archivé(s) le : jeudi 1 décembre 2011 - 02:27:03


Fichiers produits par l'(les) auteur(s)


  • HAL Id : inria-00618057, version 1



Barbara André, Tom Vercauteren, Anna Buchner, Michael Wallace, Nicholas Ayache. Learning Semantic and Visual Similarity for Endomicroscopy Video Retrieval. [Research Report] RR-7722, INRIA. 2011. 〈inria-00618057〉



Consultations de la notice


Téléchargements de fichiers