Zero-Shot 3D Shape Correspondence - Inria - Institut national de recherche en sciences et technologies du numérique Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Zero-Shot 3D Shape Correspondence

Résumé

We propose a novel zero-shot approach to computing correspondences between 3D shapes. Existing approaches mainly focus on isometric and near-isometric shape pairs (\textit{e}.\textit{g}., human vs. human), but less attention has been given to strongly \emph{non-isometric} and \emph{inter-class} shape matching (\textit{e}.\textit{g}., human vs. cow). To this end, we introduce a fully automatic method that exploits the exceptional reasoning capabilities of recent foundation models in language and vision to tackle difficult shape correspondence problems. Our approach comprises multiple stages. First, we classify the 3D shapes in a zero-shot manner by feeding rendered shape views to a language-vision model (\textit{e}.\textit{g}., BLIP2) to generate a list of class proposals per shape. These proposals are unified into a single class per shape by employing the reasoning capabilities of ChatGPT. Second, we attempt to segment the two shapes in a zero-shot manner, but in contrast to the co-segmentation problem, we do not require a mutual set of semantic regions. Instead, we propose to exploit the in-context learning capabilities of ChatGPT to generate two different sets of \emph{semantic regions} for each shape and a \emph{semantic mapping} between them. This enables our approach to match strongly non-isometric shapes with significant differences in geometric structure. Finally, we employ the generated semantic mapping to produce coarse correspondences that can further be refined by the functional maps framework to produce dense point-to-point maps. Our approach\footnote{Project webpage: \projecthref}, despite its simplicity, produces highly plausible results in a zero-shot manner, especially between \emph{strongly non-isometric} shapes.
Fichier principal
Vignette du fichier
ZeroShotCorr_SIGGAsia2023.pdf (11.95 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04352378 , version 1 (21-12-2023)

Identifiants

Citer

Ahmed Abdelreheem, Abdelrahman Eldesokey, Maks Ovsjanikov, Peter Wonka. Zero-Shot 3D Shape Correspondence. SIGGRAPH ASIA 2023 - The 16th ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques, Dec 2023, Sydney, Australia. ⟨10.1145/3610548.3618228⟩. ⟨hal-04352378⟩
29 Consultations
11 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More