Scene Relighting & 3D Editing

"Lighting is the secret subject of every photograph. With neural fields, it's finally something you can also edit."

Pixel, Relighting-Obsessed AI Agent

23.5.1 Inverse Rendering Fundamentals

The forward rendering equation (Kajiya, 1986) states that the outgoing radiance at a point $x$ in direction $\omega_o$ is:

$$ L_o(x, \omega_o) = \int_\Omega f_r(x, \omega_i, \omega_o)\, L_i(x, \omega_i)\, (\omega_i \cdot n)\, d\omega_i $$

where $f_r$ is the BRDF (material), $L_i$ is the incoming radiance (illumination), and $n$ is the surface normal. Inverse rendering solves for $f_r$ and $L_i$ given $L_o$ from multiple views, along with the geometry implied by $n$.

For neural fields, the standard recipe extends a NeRF or 3DGS with explicit material and illumination heads:

• The geometry head outputs a density (NeRF) or Gaussian distribution (3DGS).
• The material head outputs a BRDF, typically a Disney principled BRDF or an analytical GGX model.
• The illumination head outputs an environment map, often parameterized as spherical Gaussians or low-order spherical harmonics for efficiency.

The full rendering integral is approximated through Monte Carlo sampling, importance sampling on either the BRDF lobe or the environment map. Methods like NeRFactor (Zhang et al., 2021), TensoIR (Jin et al., 2023), and Relightable 3D Gaussian (Gao et al., 2024) follow this template with different geometric backbones.

The Inverse-Rendering Optimization

Listing NeRFactor, TensoIR, and Relightable 3D Gaussian names the systems but not the optimization they all run. Write the image-formation model compactly as a renderer $f$ that maps the three intrinsic components to an image,

$$ I = f(\,\mathcal{G},\ \mathcal{B},\ \mathcal{L}\,), $$

where $\mathcal{G}$ is geometry (densities or Gaussian positions plus normals $n$), $\mathcal{B}$ is the material (the BRDF $f_r$, often summarized by an albedo, a roughness, and a metallic value), and $\mathcal{L}$ is the illumination (the incoming radiance $L_i$, parameterized as an environment map). The renderer $f$ is exactly the Kajiya integral above evaluated per pixel by Monte Carlo sampling. Relighting requires inverting this map: recover the components $(\mathcal{G}, \mathcal{B}, \mathcal{L})$ that produced the captured photos, then substitute a new illumination $\mathcal{L}'$ and re-render $I' = f(\mathcal{G}, \mathcal{B}, \mathcal{L}')$. Geometry and material stay fixed; only the lighting changes, which is what makes the result a relighting rather than a repaint.

The inversion is posed as a per-scene optimization. Given a set of captured views $\{I_k\}$ with known camera poses, jointly fit the three components by minimizing a re-rendering loss that compares the renderer's output to the observations,

$$ \mathcal{L}_{render} = \sum_k \big\lVert f(\mathcal{G}, \mathcal{B}, \mathcal{L}; \pi_k) - I_k \big\rVert^2, $$

where $\pi_k$ is the $k$-th camera. This term alone is hopeless to optimize, because (as the Key Insight noted) the decomposition is ill-posed: the renderer multiplies albedo by shading, so a pixel value of, say, $0.4$ is explained equally well by albedo $0.8$ under shading $0.5$ or albedo $0.4$ under shading $1.0$. This is the classic albedo-shading ambiguity, and $\mathcal{L}_{render}$ cannot break it because every such split reproduces the observed pixels exactly. The optimizer therefore needs a regularizer $\mathcal{R}$ that injects outside knowledge about which decompositions are plausible:

$$ \min_{\mathcal{G},\,\mathcal{B},\,\mathcal{L}}\ \mathcal{L}_{render} + \lambda\, \mathcal{R}(\mathcal{G}, \mathcal{B}, \mathcal{L}). $$

Classical inverse-rendering systems build $\mathcal{R}$ from hand-designed priors (a low-order spherical-harmonic illumination is smooth, albedos are piecewise constant, normals vary slowly), exactly the terms Relightable 3D Gaussian adds as its monochromatic and normal-smoothness losses in subsection 23.5.3. The learned-prior alternative replaces those hand-designed terms with a generative model of plausible lit appearance. A diffusion relighting prior in the IC-Light style (subsection 23.5.2) has seen millions of correctly lit images, so it scores a candidate relit render by how probable it looks under that distribution; used as $\mathcal{R}$, it pushes the joint fit toward decompositions whose re-renders look like real photographs rather than physically valid but visually wrong albedo-shading splits. The regularization is what converts an unsolvable inverse problem into a well-behaved optimization, which is why every practical relighting pipeline is a re-rendering loss plus a strong prior, never the loss alone.

23.5.2 IC-Light: The Pretrained Relighting Prior

IC-Light (Zhang et al., 2024), released by the ControlNet author, took a different route. Instead of solving the full inverse rendering problem, IC-Light frames relighting as a 2D image-to-image task. The model is a Stable Diffusion fine-tune that takes (foreground image with mask, desired lighting condition) and outputs a relit foreground.

Lighting conditions come in two flavors:

• Text-conditioned: a prompt like "sunset, soft golden light from the left."
• Image-conditioned: an environment map or a background image whose lighting the model should infer and apply.

The breakthrough was IC-Light's training scheme: a self-supervised consistency loss enforces that the relit foreground, when composited with the implied background, matches the original lighting cues. This sidesteps explicit BRDF estimation while still producing physically reasonable results.

# IC-Light inference: relight a foreground image given a target
# background that defines the desired illumination.
from ic_light import ICLight
from PIL import Image

pipe = ICLight.from_pretrained("lllyasviel/ic-light").cuda()

fg = Image.open("product_photo.png").convert("RGBA")
bg = Image.open("warm_studio_bg.jpg")

relit = pipe(
    foreground=fg,
    background=bg,
    prompt="warm studio key light from upper left, soft fill",
    strength=0.85,
    guidance_scale=2.5,
).images[0]
relit.save("product_relit.png")

# For 3D relighting, render each view from the splat, run IC-Light per view,
# then either (a) use the relit views as supervision for a new splat fine-tune,
# or (b) bake the per-pixel lighting deltas back into the original Gaussians.

23.5.3 Relightable 3D Gaussians (2024)

Relightable 3D Gaussian (Gao et al., 2024) extends each Gaussian with explicit material parameters: an albedo color, a roughness scalar, a metallic scalar, and a per-Gaussian normal. The illumination is a low-resolution environment map represented by 4th-order spherical harmonics. Each rendered pixel is evaluated through the Disney principled BRDF.

The training loop is similar to vanilla 3DGS but with three extra losses:

1. Standard photometric loss against the captured views.
2. A monochromatic regularizer that nudges shadowed regions toward grayscale (encouraging the illumination, not the albedo, to vary with view).
3. A smoothness regularizer on the normals.

Output: a 3DGS scene whose albedo, geometry, and lighting can each be independently changed. Swap the environment map and re-render; the scene relights with correct shadows and specular highlights.

23.5.4 Language-Grounded 3D Editing

Editing a captured 3D scene with text prompts is the natural next step after generating one. Two families of tools dominate:

• Instruct-NeRF2NeRF (Haque et al., 2023) and its successors: edit a NeRF or 3DGS by iteratively re-rendering views, modifying them with InstructPix2Pix or Magic-Brush, and fine-tuning the 3D representation toward the edited views.
• SAM-grounded splat selection: tools like GaussianEditor and SuperSplat attach a CLIP or SAM feature to each Gaussian. Text or click selects a subset of Gaussians; the user can then translate, rotate, delete, or restyle them.

Hands-On

Segment Anything Model (SAM) promptable segmentation

Steps

The Segment Anything Model is a promptable segmenter: given an image plus a prompt (a click point, a box, or a rough mask) it returns the object mask under that prompt. It has three parts. A heavy ViT image encoder runs once per image to produce a dense embedding. A lightweight prompt encoder turns clicks and boxes into prompt tokens. A fast transformer mask decoder then cross-attends the prompt tokens against the image embedding and emits a few candidate masks with confidence scores, resolving the ambiguity of an underspecified click. Because the expensive encoding is amortized, each new prompt yields a mask in milliseconds, which is what makes interactive click-to-select editing practical.

import torch
from PIL import Image
from transformers import SamModel, SamProcessor

processor = SamProcessor.from_pretrained("facebook/sam-vit-base")
model = SamModel.from_pretrained("facebook/sam-vit-base").eval()

image = Image.open("scene.jpg").convert("RGB")
input_points = [[[450, 600]]]   # one click at pixel (450, 600)

inputs = processor(image, input_points=input_points, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

# Three candidate masks per prompt; pick the highest IoU score.
masks = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(), inputs["original_sizes"].cpu(),
    inputs["reshaped_input_sizes"].cpu(),
)
scores = outputs.iou_scores[0, 0].tolist()
best = scores.index(max(scores))
print(f"Picked mask {best} with IoU score {scores[best]:.3f}")

Output: Picked mask 2 with IoU score 0.943

Code Fragment 23.5.2a: Point-prompted segmentation with SAM through HuggingFace. The image encoder runs inside the first model(**inputs) call (the bulk of the latency); subsequent calls on the same image with fresh prompts can reuse the cached image embedding by passing image_embeddings=outputs.image_embeddings, which is what makes interactive editing tools (GaussianEditor, SuperSplat) feel responsive.

The mask decoder's behaviour is captured by two equations. Writing $\mathbf{F} \in \mathbb{R}^{H \times W \times 256}$ for the image embedding ($H = W = 64$ for SAM's $1024 \times 1024$ input), $\mathbf{P} \in \mathbb{R}^{N \times 256}$ for the prompt tokens, and three learnable output tokens $\{\mathbf{m}_1, \mathbf{m}_2, \mathbf{m}_3\}$ plus one IoU token $\mathbf{q}$, the decoder runs two-way cross-attention to produce updated tokens $\{\tilde{\mathbf{m}}_k\}$ and $\tilde{\mathbf{q}}$ and updated image features $\tilde{\mathbf{F}}$. Each mask is then a per-pixel dot product, and each IoU prediction is a small MLP head on the IoU token:

$$\hat{\mathbf{M}}_k = \sigma\!\left(\tilde{\mathbf{F}}\, \tilde{\mathbf{m}}_k^{\top}\right) \in [0, 1]^{H \times W}, \qquad \hat{\mathrm{IoU}}_k = \mathrm{MLP}_k(\tilde{\mathbf{q}}) \in [0, 1].$$

The selected output mask is the one with the largest predicted IoU, $k^\star = \arg\max_k \hat{\mathrm{IoU}}_k$, and the IoU head is trained with an MSE regression loss against the true IoU between $\hat{\mathbf{M}}_k$ (thresholded at $0.5$) and the ground-truth mask. The three output tokens specialize during training to "whole / part / subpart" candidates, which is why a single ambiguous click on a person can simultaneously propose the whole body, the upper torso, and just the shirt.

# Instruct-NeRF2NeRF-style edit loop in pseudocode.
# Iteratively replaces training views with edited ones,
# then fine-tunes the 3D scene to match.
from diffusers import StableDiffusionInstructPix2PixPipeline

editor = StableDiffusionInstructPix2PixPipeline.from_pretrained(
    "timbrooks/instruct-pix2pix"
).to("cuda")
prompt = "Turn the chair into a wicker armchair"

for step in range(edit_steps):
    view_idx = random.randint(0, len(cameras) - 1)
    current = render_view(gaussians, cameras[view_idx])
    # Edit the rendered view with InstructPix2Pix
    edited = editor(
        prompt, image=current,
        image_guidance_scale=1.5,
        guidance_scale=7.5,
    ).images[0]
    # Replace the training view with the edited one
    training_views[view_idx] = edited
    # Continue 3DGS fine-tuning on the updated training set
    fine_tune_step(gaussians, training_views, cameras)

23.5.5 Failure Modes and the Consistency Tax

The most common failure mode in 3D editing is view-inconsistent edits. InstructPix2Pix is stochastic; edited views may show the wicker pattern in incompatible ways. The iterative fine-tune of Instruct-NeRF2NeRF averages over these inconsistencies but can produce blurry results. Three remedies are common:

• Joint multi-view editing: use MVDream-style editing models that produce consistent multi-view edits in a single pass.
• Masked editing: restrict the edit to a SAM-segmented region so unrelated parts of the scene do not drift.
• Edit-Anything-with-3DGS: directly modify Gaussian parameters (color, opacity, position) for the selected Gaussians instead of going through a 2D editor.

23.5.6 Composition and Export

The last step in a production workflow is composition: placing a relit, edited 3D asset into a target scene. This requires aligning lighting, ground plane, and scale. The 2025 toolchain typically:

1. Estimates the target scene's environment map (via a model like StyleLight or via HDR capture).
2. Re-relights the inserted asset under that environment map (IC-Light, Relightable 3D Gaussian).
3. Estimates a shadow plane and adds a synthetic shadow.
4. Exports the composed result to a target renderer (Unreal, Blender, Three.js).

For a fully end-to-end example connecting capture, relighting, edit, and export, see the LumaAI Splat Tools and the Polycam SDK docs.

pip install gsplat nerfstudio numpy torch torchvision
# Install COLMAP separately: apt-get install colmap (Linux) or download .exe (Windows)