101. Neural Light Spheres for Implicit Image Stitching and View Synthesis
- Author
-
Chugunov, Ilya, Joshi, Amogh, Murthy, Kiran, Bleibel, Francois, and Heide, Felix
- Subjects
Computer Science - Computer Vision and Pattern Recognition - Abstract
Challenging to capture, and challenging to display on a cellphone screen, the panorama paradoxically remains both a staple and underused feature of modern mobile camera applications. In this work we address both of these challenges with a spherical neural light field model for implicit panoramic image stitching and re-rendering; able to accommodate for depth parallax, view-dependent lighting, and local scene motion and color changes during capture. Fit during test-time to an arbitrary path panoramic video capture -- vertical, horizontal, random-walk -- these neural light spheres jointly estimate the camera path and a high-resolution scene reconstruction to produce novel wide field-of-view projections of the environment. Our single-layer model avoids expensive volumetric sampling, and decomposes the scene into compact view-dependent ray offset and color components, with a total model size of 80 MB per scene, and real-time (50 FPS) rendering at 1080p resolution. We demonstrate improved reconstruction quality over traditional image stitching and radiance field methods, with significantly higher tolerance to scene motion and non-ideal capture settings., Comment: Project site: https://light.princeton.edu/publication/neuls/
- Published
- 2024
- Full Text
- View/download PDF