Skip to Main content Skip to Navigation
New interface
Journal articles

Multi-view Relighting using a Geometry-Aware Network

Abstract : We propose the first learning-based algorithm that can relight images in a plausible and controllable manner given multiple views of an outdoor scene. In particular, we introduce a geometry-aware neural network that utilizes multiple geometry cues (normal maps, specular direction, etc.) and source and target shadow masks computed from a noisy proxy geometry obtained by multi-view stereo. Our model is a three-stage pipeline: two subnetworks refine the source and target shadow masks, and a third performs the final relighting. Furthermore, we introduce a novel representation for the shadow masks, which we call RGB shadow images. They reproject the colors from all views into the shadowed pixels and enable our network to cope with inacuraccies in the proxy and the non-locality of the shadow casting interactions. Acquiring large-scale multi-view relighting datasets for real scenes is challenging, so we train our network on photorealistic synthetic data. At train time, we also compute a noisy stereo-based geometric proxy, this time from the synthetic renderings. This allows us to bridge the gap between the real and synthetic domains. Our model generalizes well to real scenes. It can alter the illumination of drone footage, image-based renderings, textured mesh reconstructions, and even internet photo collections.
Complete list of metadata

Cited literature [56 references]  Display  Hide  Download

https://hal.inria.fr/hal-02125095
Contributor : Team Reves Connect in order to contact the contributor
Submitted on : Friday, May 10, 2019 - 10:18:13 AM
Last modification on : Friday, November 18, 2022 - 9:25:33 AM

File

Multi-view-Relighting.pdf
Files produced by the author(s)

Identifiers

Collections

Citation

Julien Philip, Michaël Gharbi, Tinghui Zhou, Alexei A Efros, George Drettakis. Multi-view Relighting using a Geometry-Aware Network. ACM Transactions on Graphics, In press, 38, ⟨10.1145/3306346.3323013⟩. ⟨hal-02125095⟩

Share

Metrics

Record views

384

Files downloads

219