OpenVLA-7B Reference Implementation

"Open the weights, read the tokenizer, run the policy. That is the whole pedagogy in one sentence."

Pip, Reference-Implementation-Reader AI Agent

24.2.1 The Architecture from Pixels to Action Tokens

OpenVLA stitches together four components, all of them pretrained and frozen or finetuned at different rates. Visual input is a single third-person RGB frame at 224 by 224 pixels (no wrist camera in the released checkpoint, although the data recipe supports one). The fused vision encoder concatenates the patch tokens from a SigLIP-SO400M tower and a DINOv2-Large tower, on the theory that SigLIP's semantic understanding and DINOv2's spatial-feature sharpness are complementary. A learned projection compresses the 1408-D fused vision feature into the 4096-D hidden size of the Llama-2 7B backbone. The instruction string is BPE-tokenized with the LLaMA tokenizer extended by the 1,792 action tokens described in Section 24.1. The decoder-only transformer trunk autoregresses over the unified sequence, emitting one action token per dimension.

24.2.2 The Training Data: Open X-Embodiment

OpenVLA trains on the Open X-Embodiment mixture (Open X-Embodiment Collaboration, 2024, arXiv:2310.08864), a curated union of 60 robot datasets contributed by 21 institutions covering 22 distinct robot embodiments. After filtering for usable language annotations and trajectory quality, the OpenVLA training mixture contains roughly 970,000 trajectories, with the largest contributors being BridgeData-V2 (60k WidowX trajectories), the original RT-1 dataset (130k Everyday Robot trajectories), and Berkeley Cable Routing (1.5k Franka trajectories). Each trajectory is a sequence of (image, instruction, action) tuples at 5-30 Hz. The actions are normalized to a unified 7-D end-effector delta convention, which is the mapping that makes cross-embodiment training tractable.

# Loading the Open X-Embodiment mixture through the rlds_dataset_builder.
import tensorflow_datasets as tfds
from openvla.data.oxe import OXE_NAMED_MIXES, make_interleaved_dataset

mixture_spec = OXE_NAMED_MIXES["oxe_magic_soup"]   # the OpenVLA training mix
dataset = make_interleaved_dataset(
    mixture_spec,
    data_dir="gs://gresearch/robotics",
    train=True,
    window_size=1,            # single-step action prediction
    future_action_window_size=0,
    action_proprio_normalization_type="normal",
    batch_size=32,
    shuffle_buffer_size=100_000,
)

for batch in dataset.take(1):
    print(batch["observation"]["image_primary"].shape)   # (32, 224, 224, 3)
    print(batch["task"]["language_instruction"].numpy()[0])
    print(batch["action"].shape)                       # (32, 1, 7)

24.2.3 The Action Tokenizer in the Released Checkpoint

OpenVLA's released action tokenizer follows the structure from Section 24.1, with two implementation details worth knowing. First, the bin boundaries are computed from the empirical 1st and 99th percentiles of each action dimension on the training data, not from a hand-chosen [lo, hi] range. This puts more bin resolution where the data actually lives. Second, the tokenizer reuses the 256 least-frequently-used tokens from the LLaMA vocabulary instead of growing the embedding table. The 256 chosen indices were a list of obscure Unicode characters and rare BPE pieces that empirical analysis showed almost never appear in instruction strings, so colliding them with action tokens caused no measurable text degradation.

# OpenVLA's actual action tokenizer (paraphrased from the released source).
import numpy as np
from transformers import AutoTokenizer

class OpenVLAActionTokenizer:
    def __init__(self, tokenizer, bins, n_action_dims=7):
        self.tokenizer = tokenizer
        self.bins = bins                          # np.ndarray of bin edges, shape (n_dims, n_bins+1)
        self.n_action_dims = n_action_dims
        # Use the last 256 tokens of LLaMA's vocab for actions.
        self.action_token_begin_idx = len(tokenizer) - 256

    def __call__(self, action: np.ndarray) -> list[str]:
        # Quantize each dim independently into [0, 255] bin indices.
        discretized = np.digitize(
            action, bins=self.bins[0, 1:-1]
        ).clip(0, 255)
        # Map each bin index to a special vocabulary token id.
        token_ids = self.tokenizer.vocab_size - 256 + discretized
        return self.tokenizer.batch_decode(token_ids)

    def decode(self, token_ids: np.ndarray) -> np.ndarray:
        bin_idx = token_ids - self.action_token_begin_idx
        centers = 0.5 * (self.bins[:, :-1] + self.bins[:, 1:])
        return centers[np.arange(self.n_action_dims), bin_idx]

24.2.4 Running the Model: The Six-Line Inference Loop

The OpenVLA Hugging Face checkpoint runs out of the box with the stock transformers library, which is the strongest possible signal that "VLA equals LLM with motor tokens" is a true description and not a marketing slogan. The full inference loop fits in a screen of code:

import torch
from PIL import Image
from transformers import AutoModelForVision2Seq, AutoProcessor

processor = AutoProcessor.from_pretrained(
    "openvla/openvla-7b", trust_remote_code=True
)
model = AutoModelForVision2Seq.from_pretrained(
    "openvla/openvla-7b",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    trust_remote_code=True,
)

image = Image.open("frame.png")
prompt = "In: What action should the robot take to place the red block on the plate?\nOut:"
inputs = processor(prompt, image).to("cuda", dtype=torch.bfloat16)
action = model.predict_action(**inputs, unnorm_key="bridge_orig", do_sample=False)
print(action)   # np.ndarray of shape (7,) ready to ship to the robot controller

24.2.5 Inference Performance and the Quantization Frontier

OpenVLA-7B in bfloat16 runs at about 6 Hz on a single A100 80GB, which is acceptable for slow-manipulation tasks but too slow for any task involving moving objects or human collaboration. The community has spent the last year optimizing this. Three results matter. First, INT4 quantization with bitsandbytes (the same recipe from Chapter 16 on QLoRA) brings inference to 11 Hz with under 1 percent success-rate degradation on standard benchmarks. Second, speculative decoding with a small draft model (a 350M-param policy distilled from OpenVLA) bumps wall-clock throughput to 18 Hz. Third, ahead-of-time TensorRT compilation of the full pipeline gives another 1.4x, pushing throughput past 25 Hz, which is fast enough for most contact-rich manipulation.

24.2.6 Finetuning OpenVLA on Your Own Robot

The released checkpoint understands the union of training distributions but is not specialized to any one robot you might own. Finetuning is straightforward thanks to LoRA. The recipe (LoRA rank 32 on all attention projections, learning rate 5e-4, AdamW, batch size 16, two epochs of demonstration data) consistently lifts success rate by 15 to 30 percentage points on a new robot after a few thousand demonstrations. The OpenVLA team has open-sourced both the LoRA finetuning script and a separate fully-finetune recipe.

# LoRA finetuning of OpenVLA on a custom dataset, ~30 lines total.
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForVision2Seq, AutoProcessor, Trainer

processor = AutoProcessor.from_pretrained("openvla/openvla-7b", trust_remote_code=True)
model = AutoModelForVision2Seq.from_pretrained(
    "openvla/openvla-7b",
    torch_dtype="bfloat16",
    trust_remote_code=True,
)

lora_cfg = LoraConfig(
    r=32, lora_alpha=64, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_cfg)
model.print_trainable_parameters()   # typically ~50M of 7.6B

trainer = Trainer(
    model=model,
    train_dataset=my_robot_dataset,            # your demonstrations, formatted with the OXE schema
    args=TrainingArguments(
        per_device_train_batch_size=16,
        learning_rate=5e-4,
        num_train_epochs=2,
        bf16=True,
        gradient_checkpointing=True,
    ),
)
trainer.train()

24.2.7 Where OpenVLA Falls Short

OpenVLA is the right starting point but is not state of the art. It uses a single third-person camera (no wrist cam), predicts a single action per forward pass (no chunking), and trains entirely on data from arm-and-gripper manipulators (no humanoids, no quadrupeds, no soft hands). Its absolute success rates on the harder Open X-Embodiment benchmarks are still in the 30-50 percent range for unseen objects, and its dexterity ceiling, set by the 7-D end-effector action space, excludes any task that requires multi-finger coordination. The next two sections cover the models that have pushed past those limits: pi-0 in Section 24.3 via flow-matching and continuous action heads, and the RT-2-X data-scaling story in Section 24.4.

Key Takeaway

You now know what a generalist VLA looks like at the source-code level. Section 24.3 jumps to Physical Intelligence's pi-0 and pi-0.5, which replace OpenVLA's discrete-token action head with a flow-matching network that emits continuous trajectories. The architectural delta is small but the trade-off space is interestingly different, and pi-0.5 has become the de facto reference for dexterous manipulation in 2026.