Back to Search
Start Over
Spatially and color consistent environment lighting estimation using deep neural networks for mixed reality
- Source :
- Computers & Graphics. 102:257-268
- Publication Year :
- 2022
- Publisher :
- Elsevier BV, 2022.
-
Abstract
- The representation of consistent mixed reality (XR) environments requires adequate real and virtual illumination composition in real-time. Estimating the lighting of a real scenario is still a challenge. Due to the ill-posed nature of the problem, classical inverse-rendering techniques tackle the problem for simple lighting setups. However, those assumptions do not satisfy the current state-of-art in computer graphics and XR applications. While many recent works solve the problem using machine learning techniques to estimate the environment light and scene’s materials, most of them are limited to geometry or previous knowledge. This paper presents a CNN-based model to estimate complex lighting for mixed reality environments with no previous information about the scene. We model the environment illumination using a set of spherical harmonics (SH) environment lighting, capable of efficiently represent area lighting. We propose a new CNN architecture that inputs an RGB image and recognizes, in real-time, the environment lighting. Unlike previous CNN-based lighting estimation methods, we propose using a highly optimized deep neural network architecture, with a reduced number of parameters, that can learn high complex lighting scenarios from real-world high-dynamic-range (HDR) environment images. We show in the experiments that the CNN architecture can predict the environment lighting with an average mean squared error (MSE) of 7.85 × 10−4 when comparing SH lighting coefficients. We validate our model in a variety of mixed reality scenarios. Furthermore, we present qualitative results comparing relights of real-world scenes.
- Subjects :
- Mean squared error
Computer science
business.industry
General Engineering
Spherical harmonics
Computer Graphics and Computer-Aided Design
Mixed reality
Human-Computer Interaction
Computer graphics
Set (abstract data type)
Deep neural networks
Computer vision
Artificial intelligence
Architecture
Representation (mathematics)
business
Subjects
Details
- ISSN :
- 00978493
- Volume :
- 102
- Database :
- OpenAIRE
- Journal :
- Computers & Graphics
- Accession number :
- edsair.doi...........163e0a1a3aa724a031e76ea268601004