Skip to Main content Skip to Navigation
New interface
Reports (Research report)

Learning Semantic and Visual Similarity for Endomicroscopy Video Retrieval

Abstract : Traditional Content-Based Image Retrieval (CBIR) systems only deliver visual outputs that are not directly interpretable by the physicians. Our objective is to provide a system for endomicroscopy video retrieval which delivers both visual and semantic outputs that are consistent with each other. In a previous study, we developed an adapted bag-of-visual-words method for endomicroscopy retrieval that computes a visual signature for each video. In this study, we first leverage semantic ground-truth data to transform these visual signatures into semantic signatures that reflect how much the presence of each semantic concept is expressed by the visual words describing the videos. Using cross-validation, we demonstrate that our visual-word-based semantic signatures enable a recall performance which is significantly higher than those of several state-of-the-art methods in CBIR. In a second step, we propose to improve retrieval relevance by learning, from a perceived similarity ground truth, an adjusted similarity distance. Our distance learning method allows to improve, with statistical significance, the correlation with the perceived similarity. Our resulting retrieval system is efficient in providing both visual and semantic information that are correlated with each other and clinically interpretable by the endoscopists.
Document type :
Reports (Research report)
Complete list of metadata

Cited literature [24 references]  Display  Hide  Download
Contributor : Barbara André Connect in order to contact the contributor
Submitted on : Wednesday, August 31, 2011 - 3:47:15 PM
Last modification on : Friday, November 18, 2022 - 9:23:57 AM
Long-term archiving on: : Thursday, December 1, 2011 - 2:27:03 AM


Files produced by the author(s)


  • HAL Id : inria-00618057, version 1


Barbara André, Tom Vercauteren, Anna M. Buchner, Michael B. Wallace, Nicholas Ayache. Learning Semantic and Visual Similarity for Endomicroscopy Video Retrieval. [Research Report] RR-7722, INRIA. 2011. ⟨inria-00618057⟩



Record views


Files downloads