Single-view neural illumination estimation and editing for dynamic light field display

Xuyang Hong; Jie Xie; Jie Sheng; Feng Xie; Jin Zhang; Kangwei Wang; Ming Cheng; Chun Chen; Jae-Hyeung Park; Cheng Wu

doi:10.1038/s41377-026-02234-4

Your Location：

Home >

Browse articles >

Single-view neural illumination estimation and editing for dynamic light field display

Original Articles

- Single-view neural illumination estimation and editing for dynamic light field display
- Single-view neural illumination estimation and editing for dynamic light field display
- Light: Science & Applications 2026年15卷第5期页码：1624-1638
- Affiliations：
  
  1.School of Rail Transportation, Soochow University, 8 Jixue Rd., Suzhou 215100, China
  2.State Key Laboratory of Integrated Services Networks, Xidian University, Xian 710126, China
  3.Department of Electrical and Computer Engineering, Seoul National University, Seoul 08826, South Korea
- Author bio：
  
  Cheng Wu (cwu@suda.edu.cn)
- Funds：
- DOI：10.1038/s41377-026-02234-4
  中图分类号：
- 收稿：2025-11-30，
  
  修回：2026-02-05，
  
  录用：2026-02-06，
  
  网络首发：2026-03-05，
  
  纸质出版：2026-05-31
- Accepted：
Scan QR Code
Xuyang Hong, Jie Xie, Jie Sheng, 等. Single-view neural illumination estimation and editing for dynamic light field display[J]. Light: Science & Applications, 2026,15(5):1624-1638.

Hong, X. Y. et al. Single-view neural illumination estimation and editing for dynamic light field display. Light: Science & Applications, 15, 1624-1638 (2026).
Xuyang Hong, Jie Xie, Jie Sheng, 等. Single-view neural illumination estimation and editing for dynamic light field display[J]. Light: Science & Applications, 2026,15(5):1624-1638. DOI： 10.1038/s41377-026-02234-4.

Hong, X. Y. et al. Single-view neural illumination estimation and editing for dynamic light field display. Light: Science & Applications, 15, 1624-1638 (2026). DOI： 10.1038/s41377-026-02234-4.

摘要

Abstract

Light

field rendering is widely applied to virtual reality (VR)

augmented reality (AR)

mixed reality (MR) and extended reality (XR). For photorealistic light field displays

it requires a dense view sampling of the scene. However

in dynamic immersive interactions

the available observations are often too sparse to synthesize the complete light field required for a high-fidelity display. Therefore

it poses a huge challenge for generating photometrically consistent views between the virtual and real world. Here

we introduce a neural illumination estimation and editing framework for adaptive light field synthesis. The proposed method can explicitly encode intrinsic parameters of illumination from one single sampling view

which is used for a hybrid-guided generative network to synthesize photometrically plausible dense views of the scene under the guidance of a complete rendering model. It deconstructs the baked-in lighting to enable consistent and high-fidelity relighting from any viewpoint. Our method estimates and edits illumination with only 0.2397 W m

−2

irradiance error and 7.02° angular deviation

yielding synthesized images with an average 17.0% improvement in PSNR and a 51.2% reduction in LPIPS. This work presents a practical pathway towards truly interactive and adaptive digital light fields

enabling photorealistic content generation for the next generation of near-eye displays and computational imaging systems.

关键词

Keywords

references

Liu, H. S. et al. Learning-based real-time imaging through dynamic scattering media. Light Sci. Appl. 13 , 194 (2024)..

Adelson, E. H. & Bergen, J. R. The plenoptic function and the elements of early vision. In Landy, M. & Movshon, J. A. (eds.) Computational Models of Visual Processing , 3–20 (The MIT Press, Cambridge, 1991).

Levoy, M. & Hanrahan, P. Light field rendering. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques , 31–42 (ACM, New Orleans, LA, USA, 1996).

Wang, B. W. et al. Single-shot super-resolved fringe projection profilometry (sssr-fpp): 100,000 frames-per-second 3d imaging with deep learning. Light Sci. Appl. 14 , 70 (2025)..

Shi, L., Li, B. C. & Matusik, W. End-to-end learning of 3d phase-only holograms for holographic display. Light Sci. Appl. 11 , 247 (2022)..

Wang, Z. Q. et al. Computational optical imaging: on the convergence of physical and digital layers. Optica 12 , 113–130 (2025)..

Mengu, D. et al. At the intersection of optics and deep learning: statistical inference, computing, and inverse design. Adv. Opt. Photonics 14 , 209–290 (2022)..

Shi, L. et al. Towards real-time photorealistic 3d holography with deep neural networks. Nature 591 , 234–239 (2021)..

Wu, D. X. et al. Imaging biological tissue with high-throughput single-pixel compressive holography. Nat. Commun. 12 , 4712 (2021)..

Barbastathis, G., Ozcan, A. & Situ, G. H. On the use of deep learning for computational imaging. Optica 6 , 921–943 (2019)..

Suo, J. L. et al. Computational imaging and artificial intelligence: The next revolution of mobile vision. Proc. IEEE 111 , 1607–1639 (2023)..

Xiong, J. H. et al. Augmented reality and virtual reality displays: emerging technologies and future perspectives. Light Sci. Appl . 10 , 216 (2021)..

Solomashenko, A. B. et al. Industrial applications of ar headsets: a review of the devices and experience. Light Adv. Manuf . 6 , 358–387 (2025)..

Karsch, K. et al. Rendering synthetic objects into legacy pho tographs. ACM Transac. Graph . 30 , 1–12 (2011)..

Mania, K. et al. The effect of visual and interaction fidelity on spatial cognition in immersive virtual environments. IEEE Transac. Visual. Comput. Graph . 12 , 396–404 (2006)..

Kajiya, J. T. The rendering equation. ACM SIGGRAPH Computer Graphics 20, 143–150 (1986).

Cook, R. L. & Torrance, K. E. A reflectance model for computer graphics. ACM Transac. Graph . 1 , 7–24 (1982)..

Zhang, J. Y. o. Single image relighting based on illumination field reconstruction. Opt. Express 31 , 29676–29694 (2023)..

Hu, X. M. et al. Robust and accurate transient light transport decomposition via convolutional sparse coding. Opt. Lett . 39 , 3177–3180 (2014)..

Kim, T. S. et al. Future trends of display technology: micro-leds toward transparent, free-form, and near-eye displays. Light Sci. Appl . 14 , 335 (2025)..

Ding, Y. Q. et al. Breaking the i n-coupling efficiency limit in waveguide-based ar displays with polarization volume gratings. Light Sci. Appl . 13 , 185 (2024)..

Tian, Z. T. et al. An achromatic metasurface waveguide for augmented reality displays. Light Sci. Appl. 14 , 94 (2025)..

Luo, Z. Y. et al. Achromatic diffractive liquid-crystal optics for virtual reality displays. Light Sci. Appl. 12 , 230 (2023)..

Yin, K. et al. Advanced liquid crystal devices for augmented reality and virtual reality displays: principles and applications. Light Sci. Appl. 11 , 161 (2022)..

Kress, B. C. & Pace, M. Holographic optics in planar optical systems for next generation small form factor mixed reality headsets. Light Adv. Manuf. 3 , 771–801 (2022)..

Park, J.-H. & Lee, B. Holographic techniques for augmented reality and virtual reality near-eye displays. Light. Adv. Manuf. 3 , 137–150 (2022)..

Zhang, X. M. et al. Nerfactor: neural factorization of shape and reflectance under an unknown illumination. ACM Trans. Graph. 40 , 237 (2021 )..

Srinivasan, P. P. et al. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 7491–7500 (IEEE, Nashville, TN, USA, 2021).

Verbin, D. et al. Ref-nerf: structured view-dependent appearance for neural radiance fields. IEEE Trans. Pattern Anal. Mach. Intell. 47 , 9426–9437 (2025)..

Mildenhall, B. et al. Nerf: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65 , 99–106 (2022)..

Kerbl, B. et al. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42 , 139 (2023)..

Bell, S., Bala, K. & Snavely, N. Intrinsic images in the wild. ACM Trans. Graph. 33 , 159 (2014)..

Wang, F. et al. Single-pixel imaging using physics enhanced deep learning. Photonics Res. 10 , 104–110 (2021)..

Lalonde, J.-F. et al. Photo clip art. ACM Trans. Graph. 26 , 3 (2007)..

Gardner, A. et al. Linear light source reflectometry. ACM Trans. Graph. 22 , 749–758 (2003)..

Song, S. R. & Funkhouser, T. Neural illumination: Lighting prediction for indoor environments. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 6911–6919 (IEEE, Long Beach, CA, USA, 2019).

Shen, S. Y. et al. Illumidiff: indoor illumination estimation from a single image with diffusion model. IEEE Trans. Vis. Computer Graph. 31 , 7752–7768 (2025)..

Ponglertnapakorn, P., Tritrong, N. & Suwajanakorn, S. Difareli: Diffusion face relighting. In Proc. of the IEEE/CVF International Conference on Computer Vision (ICCV) , 22589–22600 (IEEE, Paris, France, 2023).

Hu, Z. Y. et al. Pnrnet: Physically-inspired neural rendering for any-to-any relighting. IEEE Trans. Image Process. 31 , 3935–3948 (2022)..

Zhang, L. M., Rao, A. Y. & Agrawala, M. Scaling in-the-wild training for diffusion-based illumination harmonization and editing by imposing consistent light transport. In Proc. 13th International Conference on Learning Representations (OpenReview. net, Singapore, 2025).

Zeng, C. et al. Relighting neural radiance fields with shadow and highlight hints. In ACM SIGGRAPH 2023 Conference Proceedings , 73 (ACM, Los Angeles, CA, USA, 2023).

Sharma, A., Tan, R. T. & Cheong, L.-F. Single-image camera response function using prediction consistency and gradual refinement. In Proc. 15th Asian Conference on Computer Vision , 19–35 (Springer, Kyoto, Japan, 2021).

Grossberg, M. D. & Nayar, S. K. Determining the camera response from images: What is knowable?. IEEE Trans. Pattern Anal. Mach. Intell. 25 , 1455–1467 (2003)..

Banterle, F. et al. A framework for inverse tone mapping. Vis. Computer 23 , 467–478 (2007)..

Reinhard, E. et al. Photographic tone reproduction for digital images. ACM Trans. Graph. 21 , 267–276 (2002)..

Zhang, R. et al. The unreasonable effectiveness of deep features as a perceptual metric. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 586–595 (IEEE, Salt Lake City, UT, USA, 2018).

Barron, J. T. & Malik, J. Shape, illumination, and reflectance from shading. IEEE Trans. Pattern Anal. Mach. Intell. 37 , 1670–1687 (2015)..

Ramamoorthi, R. & Hanrahan, P. A signal-processing framework for inverse rendering. k In Proc. 28th Annual Conference on Computer Graphics and Interactive Techniques , 117–128 (ACM,Los Angeles, CA, USA, 2001).

Debevec, P. The light stages and their applications to photoreal digital actors. In SIGGRAPH Asia (ACM, Singapore, 2012).

Basri, R. & Jacobs, D. W. Lambertian reflectance and linear subspaces. IEEE Trans. Pattern Anal. Mach. Intell. 25 , 218–233 (2003)..

Zhang, J. S. et al. All-weather deep outdoor lighting estimation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 10150–10158 (IEEE, Long Beach, CA, USA, 2019).

Sanh, V. et al. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter (2019).

Karras, T. et al. Analyzing and improving the image quality of stylegan. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 8107–8116 (IEEE, Seattle, WA, USA, 2020).

Rivenson, Y. et al. Deep learning microscopy. Optica 4 , 1437–1443 (2017)..

Schönberger, J. L. & Frahm, J.-M. Structure-from-motion revisited. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 4104–4113 (IEEE, Las Vegas, NV, USA, 2016).

Chen, W. W. et al. Deep-learning-enabled temporally super-resolved multiplexed fringe projection profilometry: high-speed khz 3d imaging with low-speed camera. PhotoniX 5 , 25 (2024)..

Müller, T. et al. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. 41 , 102 (2022)..

浏览量

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

Dual-comb absolute ranging reaches orbit

Intrinsically stretchable organic light-emitting-diode with high brightness and stretchability via elastic-microphase-engineered emitter and dual-embedded electrode

Multifunctional ligand engineering for pure-blue halide perovskite nanocrystal LEDs

Editors at the frontier: exploring advanced optical engineering in the UK

Integrated optomechanical ultrasonic sensors with nano-Pascal-level sensitivity