RT-2-X & the Data-Scaling Story

"We mixed 22 robot datasets in one tokenizer, and somewhere a control theorist quietly closed their laptop."

Scale, Data-Mixed-And-Proud AI Agent

24.4.1 What "Cross-Embodiment" Means in Practice

Cross-embodiment training means a single model parameterized by a single action vocabulary learns to control many different robot bodies. The technical obstacle is that a 7-DOF Franka, a 6-DOF UR5, a wheeled mobile manipulator with a 7-DOF arm, and a bimanual 14-DOF ALOHA have action spaces that are not directly comparable. The Open X-Embodiment solution is a unified action space: all actions are expressed as 7-D end-effector deltas in a canonical world frame, plus a binary gripper command. Each contributing dataset ships a small adapter that maps its native action representation into this canonical 7-D space. The adapter is one of three pieces of robotics-specific code in the entire training stack; the other two are the action tokenizer and the image preprocessing.

24.4.2 The Open X-Embodiment Data Pyramid

The Open X-Embodiment v1 mixture (frozen in 2024) contains:

• 1.4 million trajectories total across 60 contributing datasets,
• 22 distinct robot embodiments (16 single-arm, 4 bimanual, 2 mobile manipulators),
• 970,000 trajectories after the OpenVLA "magic soup" filtering for usable language annotations and clean camera viewpoints,
• ~1.2 billion (image, language, action) tuples after timestep expansion, comparable in raw count to the GPT-2 training corpus.

The pyramid is sharply unbalanced: BridgeData-V2 alone is 60k trajectories, RT-1 is 130k, but most contributing datasets ship 1k to 5k trajectories. The "long tail" of small datasets matters more than the count suggests because each small dataset is the only source of a particular robot embodiment, camera arrangement, or task type. Removing the smallest 30 datasets (totaling under 5 percent of trajectories) drops out-of-distribution success rate by 15 percentage points, a finding that has shaped the design of v2 mixtures now under preparation.

# Inspecting the OXE mixture composition.
from openpi.training.config import OXE_MAGIC_SOUP

total_trajs = 0
for name, spec in sorted(OXE_MAGIC_SOUP.items(), key=lambda kv: -kv[1].weight):
    print(f"{name:30s}  weight={spec.weight:.4f}  n_trajs={spec.n_trajectories:6d}")
    total_trajs += spec.n_trajectories
print(f"Total: {total_trajs:,} trajectories")
# Top 3 lines (approximate):
#   bridge_orig                weight=0.2400  n_trajs= 60064
#   fractal20220817_data       weight=0.2300  n_trajs=130000
#   kuka                       weight=0.1100  n_trajs=200000
#   ...
#   Total: 968,123 trajectories

24.4.3 The Scaling Curves That Came Out of RT-2-X

The RT-2-X paper reports a power-law relationship between training-data size and out-of-distribution success rate that closely mirrors text-LLM scaling. Let N be the number of training trajectories and let SR_OOD(N) be the success rate on a held-out task that was not in the training mixture. The empirical fit is approximately

SR_OOD(N) ~ 1 - C * N^(-alpha)

with alpha roughly 0.21 and C roughly 6.3, fit on the public benchmark suite. The interpretation: each doubling of cross-embodiment data buys you about 13 percentage points of out-of-distribution success rate, until you saturate near the high-90s. This is shockingly close to the text-LLM exponent of around 0.34 (Hoffmann et al., 2022, arXiv:2203.15556), the difference being that robot data costs roughly $5 per trajectory in human teleoperation time versus essentially free for web-scraped text.

24.4.4 The RT-2-X Architecture Relative to RT-2

Architecturally, RT-2-X is essentially RT-2 with no meaningful changes; the contribution is the data. RT-2 (Brohan et al., 2023, arXiv:2307.15818) is a 55B-parameter PaLM-X-derived backbone (or a 5B-parameter PaLI derivative, in the smaller variant), with the same vocabulary-extension trick OpenVLA uses. RT-2-X swaps the training mixture from the RT-1 dataset alone to the full OXE mixture and otherwise keeps the model code unchanged. The result is a model that retains RT-2's web-scale visual reasoning, gains the cross-embodiment generalization of OXE, and demonstrates emergent capabilities not present in either ingredient alone (the most-cited example: RT-2-X can manipulate objects whose names were never in any robot dataset, but appear in the web-scale pretraining corpus of the PaLM backbone).

# Conceptual sketch of the RT-2-X training pipeline.
# The data side does the heavy lifting; the model side reuses RT-2 unchanged.

def build_rt2x_training_step(model, oxe_mixture, action_tokenizer):
    def step(batch):
        # 1. Mix from 22 robots according to the magic-soup weights.
        obs, action, lang = oxe_mixture.sample(batch)

        # 2. Map each robot's native action into the canonical 7-D space.
        canonical_action = oxe_mixture.to_canonical(action, batch.embodiment)

        # 3. Tokenize image, text, and action into one sequence.
        action_tokens = action_tokenizer(canonical_action)
        sequence = model.build_io_sequence(obs, lang, action_tokens)

        # 4. Standard next-token cross-entropy loss.
        logits = model(sequence)
        return cross_entropy(logits[:, :-1], sequence[:, 1:])
    return step

24.4.5 What the Curves Imply for Practitioners

If you are a robotics team in 2026 planning a VLA project, the scaling curves give three actionable rules. First, do not train a VLA on a single robot if you can avoid it; even a small admixture of OXE data (10 to 20 percent) measurably lifts out-of-distribution generalization. Second, the marginal value of collecting your 1,001st demonstration on your specific robot is much higher than the marginal value of the 100,001st OXE trajectory; small high-quality data scoped to your task complements rather than competes with the OXE base. Third, language quality matters disproportionately: trajectories with carefully worded instructions are worth roughly 3x what trajectories with auto-generated or noisy instructions are worth, in terms of downstream success rate. The implication is that hiring a contractor to relabel your demonstrations is often the highest-ROI data work you can do.

24.4.6 The Data Frontier and What 2026 Looks Like

The OXE v1 mixture froze in 2024. Two efforts are pushing the data frontier in 2026. The DROID dataset (Khazatsky et al., 2024) adds 76,000 trajectories of contact-rich manipulation across 18 institutions, and is approximately the same scale as BridgeData-V2 but with substantially higher dexterity content. The AutoRT effort at Google DeepMind (Ahn et al., 2024) uses LLMs to autonomously direct robots in real-world settings, generating roughly 77,000 trajectories per day across a fleet of mobile manipulators in office environments. The combination of these two streams, plus the unreleased Physical Intelligence teleop corpus, is plausibly pushing the effective training-data scale past 5 million trajectories by mid-2026, with the predicted (extrapolated) out-of-distribution success rates above 80 percent.

24.4.7 When RT-2-X Is the Wrong Answer

The RT-2-X family is not open-weights. Google has released the training methodology and partial reproductions exist (OpenVLA is the closest open-weights cousin), but the production RT-2-X checkpoints remain proprietary. For most practical purposes you should treat RT-2-X as the research artifact that established the scaling laws and use OpenVLA or pi-0 as the working implementation. The exceptions are (a) you have access to a frontier-lab partnership, (b) you are explicitly studying the scaling laws and need to reproduce them, or (c) you are at one of the contributing institutions and have inherited a working RT-2-X-class stack. In all three cases the architectural details in this section transfer directly.

Key Takeaway

Sections 39.2-39.4 covered three concrete VLA models (OpenVLA, pi-0, RT-2-X). Section 24.5 lines them up side by side with a capability matrix that lets you pick the right model for a given application. Then Section 24.6 turns to the limitations that all three share.