Skip to Main content Skip to Navigation
Journal articles

Painting-to-3D Model Alignment Via Discriminative Visual Elements

Mathieu Aubry 1, 2 Bryan C. Russell 3 Josef Sivic 2 
2 WILLOW - Models of visual object recognition and scene understanding
DI-ENS - Département d'informatique - ENS Paris, Inria Paris-Rocquencourt, CNRS - Centre National de la Recherche Scientifique : UMR8548
Abstract : This paper describes a technique that can reliably align arbitrary 2D depictions of an architectural site, including drawings, paintings and historical photographs, with a 3D model of the site. This is a tremendously difficult task as the appearance and scene structure in the 2D depictions can be very different from the appearance and geometry of the 3D model, e.g., due to the specific rendering style, drawing error, age, lighting or change of seasons. In addition, we face a hard search problem: the number of possible alignments of the painting to a large 3D model, such as a partial reconstruction of a city, is huge. To address these issues, we develop a new compact representation of complex 3D scenes. The 3D model of the scene is represented by a small set of discriminative visual elements that are automatically learnt from rendered views. Similar to object detection, the set of visual elements, as well as the weights of individual features for each element, are learnt in a discriminative fashion. We show that the learnt visual elements are reliably matched in 2D depictions of the scene despite large variations in rendering style (e.g. watercolor, sketch, historical photograph) and structural changes (e.g. missing scene parts, large occluders) of the scene. We demonstrate an application of the proposed approach to automatic re-photography to find an approximate viewpoint of historical paintings and photographs with respect to a 3D model of the site. The proposed alignment procedure is validated via a human user study on a new database of paintings and sketches spanning several sites. The results demonstrate that our algorithm produces significantly better alignments than several baseline methods.
Document type :
Journal articles
Complete list of metadata

Cited literature [49 references]  Display  Hide  Download
Contributor : Mathieu Aubry Connect in order to contact the contributor
Submitted on : Thursday, August 21, 2014 - 5:41:54 PM
Last modification on : Thursday, April 28, 2022 - 4:09:05 PM
Long-term archiving on: : Thursday, November 27, 2014 - 1:16:32 PM


Publisher files allowed on an open archive




Mathieu Aubry, Bryan C. Russell, Josef Sivic. Painting-to-3D Model Alignment Via Discriminative Visual Elements. ACM Transactions on Graphics, Association for Computing Machinery, 2014, 33 (2), pp.14. ⟨10.1145/2591009⟩. ⟨hal-00863615v3⟩



Record views


Files downloads