Painting-to-3D Model Alignment Via Discriminative Visual Elements

Mathieu Aubry 1, 2 Bryan C. Russell 3 Josef Sivic 2
2 WILLOW - Models of visual object recognition and scene understanding
DI-ENS - Département d'informatique de l'École normale supérieure, ENS Paris - École normale supérieure - Paris, Inria Paris-Rocquencourt, CNRS - Centre National de la Recherche Scientifique : UMR8548
Abstract : This paper describes a technique that can reliably align arbitrary 2D depictions of an architectural site, including drawings, paintings and historical photographs, with a 3D model of the site. This is a tremendously difficult task as the appearance and scene structure in the 2D depictions can be very different from the appearance and geometry of the 3D model, e.g., due to the specific rendering style, drawing error, age, lighting or change of seasons. In addition, we face a hard search problem: the number of possible alignments of the painting to a large 3D model, such as a partial reconstruction of a city, is huge. To address these issues, we develop a new compact representation of complex 3D scenes. The 3D model of the scene is represented by a small set of discriminative visual elements that are automatically learnt from rendered views. Similar to object detection, the set of visual elements, as well as the weights of individual features for each element, are learnt in a discriminative fashion. We show that the learnt visual elements are reliably matched in 2D depictions of the scene despite large variations in rendering style (e.g. watercolor, sketch, historical photograph) and structural changes (e.g. missing scene parts, large occluders) of the scene. We demonstrate an application of the proposed approach to automatic re-photography to find an approximate viewpoint of historical paintings and photographs with respect to a 3D model of the site. The proposed alignment procedure is validated via a human user study on a new database of paintings and sketches spanning several sites. The results demonstrate that our algorithm produces significantly better alignments than several baseline methods.
Type de document :
Article dans une revue
ACM Transactions on Graphics, Association for Computing Machinery, 2014, 33 (2), pp.14. 〈10.1145/2591009〉
Liste complète des métadonnées

Littérature citée [49 références]  Voir  Masquer  Télécharger

https://hal.inria.fr/hal-00863615
Contributeur : Mathieu Aubry <>
Soumis le : jeudi 21 août 2014 - 17:41:54
Dernière modification le : jeudi 11 janvier 2018 - 06:23:05
Document(s) archivé(s) le : jeudi 27 novembre 2014 - 13:16:32

Fichier

aubry14_painting_to_3d.pdf
Fichiers éditeurs autorisés sur une archive ouverte

Identifiants

Collections

Citation

Mathieu Aubry, Bryan C. Russell, Josef Sivic. Painting-to-3D Model Alignment Via Discriminative Visual Elements. ACM Transactions on Graphics, Association for Computing Machinery, 2014, 33 (2), pp.14. 〈10.1145/2591009〉. 〈hal-00863615v3〉

Partager

Métriques

Consultations de la notice

377

Téléchargements de fichiers

474