Robotics, Embodied AI & Scientific Discovery

"I can reason about the world with words. Now give me a body, and I will reason about it with actions."

Deploy, Freshly Embodied AI Agent

1. LLMs as Robot Planners

The key insight behind using LLMs for robotics is that language models possess extensive world knowledge about objects, their properties, and how they relate to each other.

A human saying "make me a sandwich" implies a sequence of actions (get bread, get ingredients, assemble, plate) that an LLM can decompose into steps. The challenge is grounding these steps in the robot's actual physical capabilities and environment.

SayCan: Grounding Language in Robot Actions

Google's SayCan combines an LLM's knowledge of what makes sense to do with a robot's learned affordances (what it can physically do). The LLM proposes candidate next actions, and a value function scores each action based on whether the robot can actually execute it in the current state. This product of "what should I do" (LLM) and "what can I do" (affordance model) produces grounded action plans that are both semantically correct and physically feasible. Figure 28.7.1 shows the SayCan architecture.

RT-2: Vision-Language-Action Models

Google's RT-2 (Robotics Transformer 2) takes grounding further by training a single vision-language model that directly outputs robot actions. The model processes camera images and language instructions and outputs discretized action tokens (arm positions, gripper states). By co-training on both internet-scale vision-language data and robot demonstration data, RT-2 acquires emergent reasoning capabilities: it can follow instructions involving concepts never seen during robot training (like "move the object to the picture of a country" by recognizing flags). Code Fragment 28.7.2 below puts this into practice.

# Conceptual: LLM as robot task planner
from openai import OpenAI
import json

client = OpenAI()

def plan_robot_actions(
 instruction: str,
 available_actions: list,
 scene_description: str,
) -> list:
 response = client.chat.completions.create(
 model="gpt-4o",
 messages=[
 {"role": "system", "content": f"""You are a robot task planner.
Available primitive actions: {json.dumps(available_actions)}
Current scene: {scene_description}
Decompose the instruction into a sequence of available actions.
Return a JSON array of action steps with 'action' and 'target'."""},
 {"role": "user", "content": instruction},
 ],
 response_format={"type": "json_object"},
 )
 return json.loads(response.choices[0].message.content)

plan = plan_robot_actions(
 instruction="Put the dirty dishes in the dishwasher",
 available_actions=["pick", "place", "open", "close", "navigate"],
 scene_description="Kitchen counter with 3 plates and 2 cups. Dishwasher closed.",
)
print(json.dumps(plan, indent=2))

2. Vision-Language-Action Models: The VLA Revolution

The progression from SayCan to modern VLA models traces a clear arc toward tighter integration between language understanding and physical control. SayCan kept the LLM and the robot's affordance model as separate modules, with the LLM proposing actions and a learned value function filtering them. RT-2 collapsed this into a single vision-language model that outputs discretized action tokens directly. But the field has moved considerably further since then.

OpenVLA: Open-Source VLA at Scale

OpenVLA (2024) demonstrated that a 7-billion-parameter open-source VLA can outperform the 55-billion-parameter RT-2-X on generalized robot manipulation benchmarks. Built on a Llama 2 backbone with a SigLIP vision encoder, OpenVLA was trained on the Open X-Embodiment dataset (described below) and released with fully open weights. This is significant because it shows that careful data curation and architecture design can compensate for raw parameter count, a pattern familiar from the efficient model discussions in Section 16.1.

pi0 and pi0.5: Flow Matching for Continuous Control

Physical Intelligence's pi0 (2024) and its successor pi0.5 (2025) take a fundamentally different approach to action generation. Instead of discretizing robot actions into tokens (as RT-2 and OpenVLA do), pi0 uses flow matching to produce continuous action trajectories. Flow matching learns a velocity field that transports a noise distribution to the target action distribution along straight paths, similar to how flow-matching image generators work (see Section 27.1). This enables smoother, more precise motor control, particularly for dexterous manipulation tasks like folding laundry or assembling objects. pi0.5 extends this with a hierarchical architecture: a high-level VLM plans task steps in language, while a low-level flow-matching policy generates the actual motor commands.

GR00T N1: Open Humanoid Foundation Model

NVIDIA's GR00T N1 (2025) is an open foundation model specifically designed for humanoid robots. It combines a vision-language backbone for scene understanding with a diffusion-based action head for whole-body control, including locomotion, manipulation, and coordination of dozens of joints simultaneously. GR00T N1 was trained in NVIDIA's Isaac Sim environment and supports zero-shot transfer to multiple humanoid platforms. Its release as an open model follows the pattern established by OpenVLA: making robot foundation models accessible to the broader research community rather than keeping them behind proprietary walls.

VLA Model Comparison

3. Foundation Models and Cross-Embodiment Transfer

A central challenge in robot learning is that data is scarce and expensive to collect. Every robot lab has its own hardware, environment, and task set. The Open X-Embodiment project (a collaboration of 33 research labs) addresses this by pooling over one million robot episodes from 22 different robot types into a single dataset. The motivation is directly analogous to LLM pretraining: just as GPT-3 benefits from training on diverse web text, a robot foundation model benefits from training on diverse robot experiences. The same principles of scale and diversity that drive language model performance (see Section 06.1) apply to robot data.

Generalist Policies: Octo and CrossFormer

Octo (2024) is a generalist robot policy trained on the Open X-Embodiment dataset. It uses a transformer architecture that processes both visual observations and language instructions, outputting actions for any robot represented in the training set. Crucially, Octo is designed for fine-tuning: you can take the pretrained policy and adapt it to a new robot with as few as 50 demonstrations. This mirrors the pretrain-then-fine-tune paradigm that dominates NLP (see Section 14.1 for fine-tuning fundamentals and Section 15.1 for parameter-efficient approaches).

CrossFormer extends this idea by using a modular architecture where robot-specific tokenizers handle the differences in observation and action spaces across embodiments, while a shared transformer backbone captures the common structure of manipulation tasks. This separation of embodiment-specific and embodiment-general components is conceptually similar to adapter-based fine-tuning in NLP, where a shared backbone is combined with small task-specific modules (as described in Section 15.2).

4. Sim-to-Real: LLMs in the Training Loop

Collecting real-world robot data is slow, expensive, and potentially dangerous. Simulation offers a way to generate virtually unlimited training data, but bridging the gap between simulated and real-world performance (the "sim-to-real" problem) has historically been a major bottleneck. LLMs are now being used to address multiple aspects of this challenge.

Eureka and DrEureka: LLM-Generated Reward Functions

Reward design is one of the hardest parts of robot reinforcement learning. Human experts spend weeks crafting reward functions for each new task, and small errors in the reward can lead to unexpected or unsafe behaviors. NVIDIA's Eureka (2023) uses an LLM to automatically generate reward functions from task descriptions. Given a text description like "make the robot hand rotate a pen," Eureka prompts GPT-4 to write Python reward functions, evaluates them in simulation, and iteratively refines the best candidates using the LLM's own analysis of the training curves. Eureka-generated rewards outperformed expert human-designed rewards on 83% of tasks tested, including achieving a new state-of-the-art for dexterous pen spinning. DrEureka extends this approach to handle domain randomization parameters, automatically tuning the simulation's physics to improve real-world transfer.

Synthetic Training Data at Scale

NVIDIA's combination of Isaac Sim (a GPU-accelerated physics simulator) with Cosmos (a world foundation model) enables LLM-guided generation of massive synthetic training datasets. By using language prompts to specify scenarios ("a robot arm picking up a red cup from a cluttered desk"), the system can generate 780,000 training trajectories in just 11 hours. This approach directly applies the synthetic data generation principles from Chapter 13 to the robotics domain, where the stakes of data scarcity are even higher because real-world data collection involves physical hardware, safety constraints, and wall-clock time.

World Models: Text to Interactive 3D Environments

Genie 3 and similar world models take simulation generation one step further by creating interactive 3D environments from text or image prompts. Rather than rendering predefined simulation scenes, these models generate entirely new environments that a robot policy can interact with, providing a training distribution that is broader and more diverse than any hand-designed simulator. This is analogous to how LLMs generate diverse text for training data (see Section 13.2), but extended to 3D physical scenes with interactive dynamics.

5. Safety and Grounding in Physical Systems

When an LLM hallucinates in a text conversation, the consequence is a wrong answer. When an LLM hallucinates while controlling a robot, the consequence can be a broken object, a damaged machine, or an injured person. This fundamental difference makes safety not just important but essential in embodied AI systems. The safety principles from Chapter 32 apply with amplified urgency in physical settings.

The Symbol Grounding Problem

To understand why embodied AI is fundamentally harder than text-based AI, consider what happens when you tell a robot to "gently place the egg on the counter." The word "gently" carries enormous physical specificity that is completely absent from its token representation. For a human, "gently" activates a lifetime of sensorimotor experience: the feeling of fragile objects, the muscle memory of controlled deceleration, the visual feedback of watching something land softly. For an LLM, "gently" is a statistical relationship to other tokens. Bridging this gap, translating linguistic concepts into precise physical parameters, is the central challenge of embodied AI and the reason why robot foundation models need physical demonstration data in addition to text pretraining.

LLMs operate on symbols (words, tokens), but the physical world operates on continuous quantities (forces, positions, velocities). The symbol grounding problem asks: how do we ensure that the LLM's internal representations of concepts like "fragile," "heavy," or "sharp" correspond to the actual physical properties that matter for safe robot operation? An LLM might know that eggs are fragile (from text), but translating "fragile" into the correct grip force requires a grounding mechanism that connects linguistic knowledge to physical parameters. Current VLA models learn this grounding implicitly from demonstration data, but failures still occur when encountering objects or situations outside the training distribution.

RoboGuard: Formal Safety Constraints

RoboGuard (2024) addresses robot safety by combining LLM planning with formal temporal logic constraints. Instead of relying solely on the LLM's judgment about what is safe, RoboGuard translates safety requirements ("never move the arm above the table while carrying a glass of water," "always verify grip force before lifting") into signal temporal logic (STL) formulas. The robot's planned trajectory is then checked against these formal constraints before execution. If the plan violates any safety specification, it is rejected and the LLM is asked to re-plan. This approach provides mathematically verifiable safety guarantees rather than relying on the LLM's probabilistic understanding of safety, which can fail in subtle ways.

6. Web Automation and Browser Agents

Web automation agents use LLMs to navigate websites, fill forms, click buttons, and complete tasks that normally require human interaction. These agents observe the page (through screenshots, accessibility trees, or DOM parsing), decide what action to take, execute it, and observe the result. This is the same agentic loop from Chapter 22, applied to browser environments. Code Fragment 28.7.5 below puts this into practice.

# Conceptual: web automation agent using browser tools
def web_agent_step(task: str, page_state: dict) -> dict:
 response = client.chat.completions.create(
 model="gpt-4o",
 messages=[
 {"role": "system", "content": """You are a web automation agent.
Given the current page state, decide the next action to complete
the task. Actions: click(selector), type(selector, text),
navigate(url), scroll(direction), wait(), done(result).
Return JSON with 'thought', 'action', and 'params'."""},
 {"role": "user", "content": f"""Task: {task}
Page title: {page_state['title']}
Interactive elements: {json.dumps(page_state['elements'])}"},
 ],
 response_format={"type": "json_object"},
 )
 return json.loads(response.choices[0].message.content)

OS-Level Agents and Computer Use

OS-level agents extend web automation to the entire desktop. Anthropic's Computer Use API, for example, lets Claude interact with a computer through screenshots and mouse/keyboard actions. The agent observes the screen, reasons about what it sees, and executes actions like clicking buttons, typing text, or switching between applications. This capability enables automation of tasks that span multiple applications (like copying data from a spreadsheet to an email) without requiring application-specific APIs.

7. AI for Mathematics and Theorem Proving

LLMs are making significant inroads in mathematical reasoning and formal theorem proving. Google's AlphaProof and AlphaGeometry achieved silver-medal-equivalent performance at the 2024 International Mathematical Olympiad, solving 4 of 6 problems by combining LLMs for informal reasoning with formal verification systems for proof checking. These systems represent a new paradigm where AI augments mathematical discovery rather than just calculation. Code Fragment 28.7.3 below puts this into practice.

# Using an LLM for mathematical reasoning
response = client.chat.completions.create(
 model="gpt-4o",
 messages=[
 {"role": "system", "content": """You are a mathematical reasoning assistant.
Work through problems step by step. Show all reasoning.
When uncertain, explore multiple approaches. Verify your
answer by checking boundary cases and special values."""},
 {"role": "user", "content": """Prove that for any positive integer n,
the sum 1 + 2 + ... + n = n(n+1)/2.
Use mathematical induction."""},
 ],
)
print(response.choices[0].message.content)

8. Scientific Literature Mining and Hypothesis Generation

The scientific literature grows by millions of papers per year, making it impossible for any researcher to stay current even in a narrow field. LLMs can mine this literature to identify connections between findings, generate novel hypotheses, and suggest experimental designs. Systems like Semantic Scholar's AI-powered features and specialized scientific LLMs (Galactica, SciBERT) demonstrate how language models can accelerate the scientific discovery process. Figure 28.7.2 illustrates the AI-assisted scientific discovery pipeline.

Exercises

pip install openai-whisper transformers torch torchaudio

import torch
import torchaudio
import urllib.request
import os

# Download a sample audio clip (LibriSpeech test sample)
audio_url = "https://www.voiptroubleshooter.com/open_speech/american/OSR_us_000_0010_8k.wav"
audio_path = "sample_speech.wav"
if not os.path.exists(audio_path):
 urllib.request.urlretrieve(audio_url, audio_path)

waveform, sample_rate = torchaudio.load(audio_path)
duration = waveform.shape[1] / sample_rate
print(f"Audio loaded: {duration:.1f}s at {sample_rate} Hz")
print(f"Channels: {waveform.shape[0]}, Samples: {waveform.shape[1]}")

import whisper

# Load a small model (choose "base", "small", "medium", or "large")
model = whisper.load_model("base")

# Transcribe the audio file
result = model.transcribe(audio_path, fp16=torch.cuda.is_available())

transcript = result["text"].strip()
language = result.get("language", "unknown")

print(f"Detected language: {language}")
print(f"Transcript ({len(transcript)} chars):")
print(f" {transcript[:500]}")

# Inspect segment-level timestamps
for seg in result["segments"][:5]:
 start, end = seg["start"], seg["end"]
 print(f" [{start:.1f}s - {end:.1f}s] {seg['text'].strip()}")

from transformers import pipeline

# Load a summarization model
summarizer = pipeline(
 "summarization",
 model="facebook/bart-large-cnn",
 device=0 if torch.cuda.is_available() else -1,
)

# Summarize the transcript
# BART-CNN has a 1024-token limit; truncate if needed
max_chars = 3000
input_text = transcript[:max_chars]

summary = summarizer(
 input_text,
 max_length=150,
 min_length=30,
 do_sample=False,
)

print("Original transcript length:", len(transcript), "chars")
print("Summary:")
print(f" {summary[0]['summary_text']}")

from transformers import pipeline

# One-liner ASR pipeline using Whisper
asr_pipeline = pipeline(
 "automatic-speech-recognition",
 model="openai/whisper-base",
 device=0 if torch.cuda.is_available() else -1,
)

# Transcribe with the pipeline API
asr_result = asr_pipeline(audio_path)
pipeline_transcript = asr_result["text"].strip()

# Compare outputs
print("Manual Whisper transcript:")
print(f" {transcript[:200]}...")
print()
print("Pipeline transcript:")
print(f" {pipeline_transcript[:200]}...")
print()

# Check similarity
from difflib import SequenceMatcher
similarity = SequenceMatcher(
 None, transcript.lower(), pipeline_transcript.lower()
).ratio()
print(f"Transcript similarity: {similarity:.2%}")