Painting-to-3D Model Alignment Via Discriminative Visual Elements

Mathieu Aubry 1, 2 Bryan C. Russell 3 Josef Sivic 2
2 WILLOW - Models of visual object recognition and scene understanding
DI-ENS - Département d'informatique de l'École normale supérieure, Inria Paris-Rocquencourt, CNRS - Centre National de la Recherche Scientifique : UMR8548
Abstract : This paper describes a technique that can reliably align arbitrary 2D depictions of an architectural site, including drawings, paintings and historical photographs, with a 3D model of the site. This is a tremendously difficult task as the appearance and scene structure in the 2D depictions can be very different from the appearance and geometry of the 3D model, e.g., due to the specific rendering style, drawing error, age, lighting or change of seasons. In addition, we face a hard search problem: the number of possible alignments of the painting to a large 3D model, such as a partial reconstruction of a city, is huge. To address these issues, we develop a new compact representation of complex 3D scenes. The 3D model of the scene is represented by a small set of discriminative visual elements that are automatically learnt from rendered views. Similar to object detection, the set of visual elements, as well as the weights of individual features for each element, are learnt in a discriminative fashion. We show that the learnt visual elements are reliably matched in 2D depictions of the scene despite large variations in rendering style (e.g. watercolor, sketch, historical photograph) and structural changes (e.g. missing scene parts, large occluders) of the scene. We demonstrate an application of the proposed approach to automatic re-photography to find an approximate viewpoint of historical paintings and photographs with respect to a 3D model of the site. The proposed alignment procedure is validated via a human user study on a new database of paintings and sketches spanning several sites. The results demonstrate that our algorithm produces significantly better alignments than several baseline methods.
Type de document :
Article dans une revue
ACM Transactions on Graphics, Association for Computing Machinery, 2014, 33 (2), pp.14. 〈10.1145/2591009〉
Liste complète des métadonnées

Littérature citée [49 références]  Voir  Masquer  Télécharger
Contributeur : Mathieu Aubry <>
Soumis le : jeudi 21 août 2014 - 17:41:54
Dernière modification le : vendredi 25 mai 2018 - 12:02:06
Document(s) archivé(s) le : jeudi 27 novembre 2014 - 13:16:32


Fichiers éditeurs autorisés sur une archive ouverte




Mathieu Aubry, Bryan C. Russell, Josef Sivic. Painting-to-3D Model Alignment Via Discriminative Visual Elements. ACM Transactions on Graphics, Association for Computing Machinery, 2014, 33 (2), pp.14. 〈10.1145/2591009〉. 〈hal-00863615v3〉



Consultations de la notice


Téléchargements de fichiers