We present GS-ID, a novel framework for illumination decomposition on Gaussian Splatting, achieving photorealistic novel view synthesis and intuitive light editing. Illumination decomposition is an ill-posed problem facing three main challenges: 1) priors for geometry and material are often lacking; 2) complex illumination conditions involve multiple unknown light sources; and 3) calculating surface shading with numerous light sources is computationally expensive. To address these challenges, we first introduce intrinsic diffusion priors to estimate the attributes for physically based rendering. Then we divide the illumination into environmental and direct components for joint optimization. Last, we employ deferred rendering to reduce the computational load. Our framework uses a learnable environment map and Spherical Gaussians (SGs) to represent light sources parametrically, therefore enabling controllable and photorealistic relighting on Gaussian Splatting. Extensive experiments and applications demonstrate that GS-ID produces state-of-the-art illumination decomposition results while achieving better geometry reconstruction and rendering performance.
We introduce a novel three-stage framework for illumination decomposition called GS-ID. Unlike previous methods focusing solely on environmental illumination, GS-ID decomposes the light field using environmental illumination (represented by a learnable panoramic map) and parametric direct light sources (modeled by Spherical Gaussians).
In this stage, we adopt 2DGS to reconstruct geometry. However, we observe that 2DGS mistakenly interprets glossy regions as holes and reduces the expressiveness of distant areas. To address these issues, we incorporate priors from a monocular geometric estimator to enhance the output geometric structures. Specifically, we use a pre-trained Omnidata model to provide normal supervision.
Compared with the normal produced by vanilla 2DGS, our method better handles highlights and even accurately depicts distance scenes. The second row shows that the enhanced geometry improves novel view synthesis.
To model environmental light transport more accurately, we bake probes that precompute and store occlusion. This enhances albedo reconstruction and illumination decomposition by calculating the exposure of each point near obstructing surfaces.
We use SG mixture to model direct light sources. After optimization, the initially regular light sources converge to the correct distribution. Our progressive pruning scheme eliminates insignificant light sources, preserving the primary ones. In the second row, we demonstrate ID results on the Synthetic NeRF dataset, where each scene is illuminated by a direct light source.
The results show that we accurately separate direct illumination and recover lighting effects. Notably, we show the results in linear space to emphasize the lighting effects, thus the illuminated areas are prominent.
@misc{du2024gsidilluminationdecompositiongaussian,
title={GS-ID: Illumination Decomposition on Gaussian Splatting via Diffusion Prior and Parametric Light Source Optimization},
author={Kang Du and Zhihao Liang and Zeyu Wang},
year={2024},
eprint={2408.08524},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2408.08524},
}