VoxPoser: Language as Spatial Cost Field

"The map is the cost field, and the cost field is the language. Move accordingly."

Compass, Voxel-Grid-Native AI Agent

24.9.1 The VoxPoser Idea in One Paragraph

Discretize the workspace into a 3D voxel grid, say 100 by 100 by 100 voxels of 1 cm each. For a given instruction, an LLM writes Python code that populates two grids: a target-affordance grid A(x, y, z) with positive values at voxels the end-effector should reach, and a cost grid C(x, y, z) with positive values at voxels it should avoid. A motion planner then finds the trajectory that maximizes integrated affordance minus integrated cost. The trick is that "high-reward voxels" and "high-cost voxels" can be language-grounded: the LLM can call object-detection primitives like get_voxel_mass("the apple") to populate the grids based on what is in the scene.

24.9.2 The Reference Pipeline

VoxPoser's pipeline has five stages. Each stage swaps a familiar robotics component for an LLM-driven equivalent, but only the second stage (cost-field generation) is novel; the rest is classical robotics literature. The total round-trip latency in Huang et al.'s 2023 reference implementation was about 4 seconds for the LLM call plus 1.5 seconds for trajectory optimization on a single A100, which sets the practical floor for "interactive" use.

1. Perception: detect objects in the scene and lift them into 3D positions using an off-the-shelf VLM (CLIP, OWL-ViT) plus depth from a stereo camera or learned monocular depth estimator. The original paper used OWL-ViT-Base (Minderer et al., 2022) with a RealSense D435 depth camera; segmentation latency is roughly 200 ms per frame and dominates the perception budget.
2. LLM cost-field generation: prompt an LLM with the instruction, the scene's object list, and a Python API for manipulating voxel grids. The LLM emits Python code that, when executed, returns the affordance and cost grids. Huang et al. used GPT-4 (March 2023 version) as the cost-field-generating model; in 2026, Claude 3.5 Sonnet and GPT-4o have replaced it in most reference reimplementations, primarily for the lower per-call latency.
3. Optimization: a classical trajectory optimizer (typically a sampling-based planner like RRT or a gradient-based optimizer over splines) finds the trajectory that maximizes integrated affordance minus integrated cost. VoxPoser ships with a greedy CEM (cross-entropy method) optimizer over a 32-waypoint spline; 200 CEM iterations converge in about 1.5 seconds on a 100x100x100 voxel grid.
4. Execution: a low-level controller tracks the planned trajectory. The standard implementation uses operational-space impedance control at 1 kHz, which lets the arm comply with small perception errors without re-invoking the planner.
5. Replanning: after each motion segment, the scene is re-perceived, the LLM re-emits cost fields if necessary, and the optimizer replans. In practice, the LLM is only re-invoked when an object is added or removed from the scene; pure-motion replanning reuses the previous cost field and runs at 5-10 Hz.

import numpy as np

# The skill-free API the LLM uses to build cost fields.
def get_empty_voxel_grid() -> np.ndarray:
    """Return a 100x100x100 float32 grid initialized to zero, covering 1 m^3 in front of the robot."""

def get_voxel_mass(description: str) -> np.ndarray:
    """Return a boolean voxel grid marking voxels occupied by the named object.

    Backed by a CLIP-based open-vocabulary detector plus depth estimation.
    """

def gaussian_blur(grid, sigma: float) -> np.ndarray:
    """Smooth a voxel grid with a 3D Gaussian. Useful for converting hard masks into soft fields."""

def distance_field(grid) -> np.ndarray:
    """Return the per-voxel Euclidean distance from the nearest True voxel."""

24.9.3 A Worked Example: "Pour Water Into the Cup"

The LLM, given the instruction "pour water from the bottle into the cup" and a scene with one bottle and one cup, emits the following cost-field program:

# LLM-generated cost field for "pour water from the bottle into the cup"
import numpy as np

affordance = get_empty_voxel_grid()
cost = get_empty_voxel_grid()

# Stage 1: end-effector should approach the bottle handle
bottle = get_voxel_mass("the bottle handle")
affordance += gaussian_blur(bottle.astype(np.float32), sigma=2.0)

# Stage 2: once gripping, end-effector position should be above the cup with the bottle tilted
cup = get_voxel_mass("the cup opening")
above_cup = np.roll(cup, shift=-10, axis=2)            # 10 cm above the cup
affordance += gaussian_blur(above_cup.astype(np.float32), sigma=2.0)

# Repulsive: avoid the table edge and any obstacles
obstacles = get_voxel_mass("any obstacle")
cost += 5 * obstacles.astype(np.float32)

# Hard constraint: never enter the no-fly zone around the user
user = get_voxel_mass("the human")
user_dist = distance_field(user)
cost += 100 * (user_dist < 20).astype(np.float32)

return affordance, cost

24.9.4 Where VoxPoser Shines

Three task categories play to VoxPoser's strengths. First, tasks with novel objects: because the LLM uses get_voxel_mass("the apple") rather than picking from a fixed skill list, novel objects are handled the moment the perception model recognizes them. Second, tasks with implicit spatial relationships: "place the cup just to the left of the plate" maps naturally to voxel-position arithmetic, where SayCan would need a left-of-plate skill or Code-as-Policies would need a careful place_at_relative_position API. Third, tasks with overlapping objectives: the optimizer happily integrates over both reaching and avoiding, so a single planning step can satisfy multiple constraints simultaneously.

24.9.5 Where VoxPoser Struggles

VoxPoser inherits all the limitations of classical motion planning. It assumes a static-or-slowly-changing scene; rapid replanning is expensive because the voxel grid must be rebuilt. It assumes a sparse action representation (end-effector position over time); it does not handle the discrete-action-flow of "open the gripper" cleanly, requiring auxiliary signals or a skill-level overlay. It is bottlenecked by the perception model: if get_voxel_mass("the small red screw") returns empty, the planner has no fallback. And the LLM cost-field programs occasionally produce empty or degenerate fields when the instruction is ambiguous, leading to optimizer failures that are harder to diagnose than the discrete failures of SayCan or Code-as-Policies.

24.9.6 Modern Extensions: VoxPoser in 2026

The 2026 successors to VoxPoser embed the same idea in a richer scene representation. Three notable threads:

Gaussian-splat scenes (covered in Chapter 40). Replace the voxel grid with a 3D Gaussian Splatting representation of the scene. The LLM emits a program that selects which Gaussians are attractive or repulsive. Inference is faster because the splat representation is sparser than a dense voxel grid; the precision is higher because splats can be sub-voxel-sized.

Continuous neural cost fields. Replace the voxel grid with a coordinate-input MLP that outputs (affordance, cost) at any 3D point. The LLM emits a small program that builds the MLP by composing pretrained sub-networks. This is more memory-efficient but harder to debug.

VLM-direct cost-field generation. Skip the Python-emit step entirely: a VLM takes the instruction and the scene image, and produces the affordance and cost fields directly via a learned regression head. This is the cleanest formulation but requires substantial training data, which is hard to obtain at scale.

Key Takeaway

Section 24.10 turns to a different scaling axis: not one robot with one planner, but many robots sharing a single LLM dispatcher. The Multi-Robot Dispatch pattern uses an LLM to coordinate a fleet through natural-language task assignment.