Temporally stable generative illumination with a one-step diffusion model

Large diffusion models can generate images with convincing illumination directly from strong learned priors. At the The 19th European Conference on Computer Vision — ECCV 2026 we are presenting a generative method for global illumination (GI) that produces geometrically aligned, temporally stable indirect lighting using a single-step latent diffusion model. Unlike iterative or video diffusion methods that are costly and hard to control, our model conditions a one-step image generator on scene-intrinsic signals and sparse lighting hints, and adds a Temporal VAE (TVAE) decoder for long-range temporal coherence — suitable for real-time rendering use cases with frame-by-frame generation.

Figure 1. (Left) Direct lighting, (Middle) Our result, and (Right) Reference GI.

Our method

We begin by reframing the problem itself. We formulate the screen-space approximation of GI as a conditional image generation task (Figure 2). Building on a distilled single-step model, we fine-tune the diffusion UNet while keeping the VAE frozen. Direct lighting serves as the initial lighting hint, encoded through the VAE encoder, while geometry (normals), materials (albedo, roughness, metallic), and lighting cues (noisy GI, radiance cues) are injected at multiple UNet levels through T2I-Adapters. Routing conditions through adapters rather than concatenating them onto the noisy input preserves quality and enables efficient multi-scale conditioning. We also demodulate direct lighting by albedo before encoding, so the model focuses on illumination, then re-modulate on output to restore texture and color.

To keep long sequences stable, the TVAE decoder takes the current latent along with the previous output, reprojected using motion vectors. A small history encoder injects these signals via zero convolutions, so the decoder reuses pixels from the previous frame where the current latent is close to it — suppressing flicker while preserving detail. We train only the decoder and simply swap it in at inference time.

Figure 2. Overview of our method. Multiple conditioning inputs are encoded via adapter modules and injected into a one-step diffusion model. The Temporal VAE decoder ensures temporal consistency over long sequences and replaces the frozen VAE decoder at inference time.

Results

We now turn to training and evaluation. We train on a synthetic dataset of interior scenes rendered with Blender® Cycles and evaluate on unseen scenes. Against a few prior methods, our results are visually closer to the reference and preserve geometric structure from the inputs, while the diffusion baselines show hallucinations and overly bright output (Figure 3). Quantitatively, we lead on PSNR, SSIM, and LPIPS by a large margin and achieve better temporal stability (FovVideoVDP) across all scenes.

Figure 3. Quality comparison on the test dataset. Direct lighting (a) is the initial hint and the reference (g) is path-traced GI. We compare against SSGI (b), BCNN ¹ (c), RGB↔X ² (d), and DiffusionRenderer ³ (e); arrows on our results (f) mark regions closer to the reference.

Although trained only on synthetic data, the model also adapts to real-world images, where the same conditioning images are not available. A compact variant (Ours-C), conditioned only on signals recoverable by inverse rendering of real-world images, aligns well with the features and illumination of unseen real photos and stays robust even with imperfect intrinsics (Figure 4).

Figure 4. Real-world results compared with RGB↔X. Ours-C uses conditioning inputs obtainable via inverse rendering. Original images courtesy of Pexels.

Figure 5. Our results compared to prior methods and reference ground truth.

• Further details can be found in our paper presented at ECCV 2026.
• Watch the video of our ECCV 2026 presentation.
• See our related research, Cardioid Caustics Generation with Conditional Diffusion Models.
• Discuss this research on the AMD Developer Community.
• Sign up to our AMD Developer Newsletter for the latest updates.

Disclaimers and attributions

Links to third party sites are provided for convenience and unless explicitly stated, AMD is not responsible for the contents of such linked sites and no endorsement is implied. GD-97.

Blender is a registered trademark (®) of the Blender Foundation in EU and USA.

References and footnotes