Abstract Light field rendering is widely applied to virtual reality (VR), augmented reality (AR), mixed reality (MR) and extended reality (XR). For photorealistic light field displays, it requires a dense view sampling of the scene. However, in dynamic immersive interactions, the available observations are often too sparse to synthesize the complete light field required for a high-fidelity display. Therefore, it poses a huge challenge for generating photometrically consistent views between the virtual and real world. Here, we introduce a neural illumination estimation and editing framework for adaptive light field synthesis. The proposed method can explicitly encode intrinsic parameters of illumination from one single sampling view, which is used for a hybrid-guided generative network to synthesize photometrically plausible dense views of the scene under the guidance of a complete rendering model. It deconstructs the baked-in lighting to enable consistent and high-fidelity relighting from any viewpoint. Our method estimates and edits illumination with only 0.2397 W m −2 irradiance error and 7.02 ∘ angular deviation, yielding synthesized images with an average 17.0% improvement in PSNR and a 51.2% reduction in LPIPS. This work presents a practical pathway towards truly interactive and adaptive digital light fields, enabling photorealistic content generation for the next generation of near-eye displays and computational imaging systems.
Hong et al. (2026) studied this question.