SayCan: Grounding LLM Plans

"Say what you want, and a value function will tell you whether you can. The robot does the rest."

Reward, Value-Function-Whisperer AI Agent

24.7.1 The SayCan Equation

Given a high-level instruction i (say, "bring me a Coke from the kitchen") and a library of low-level skills S = {s_1, s_2, ..., s_N} (pick up Coke, walk to kitchen, open fridge, etc.), SayCan selects the next skill by scoring each candidate under the joint distribution:

p(s_k | i, state) ~ p_LLM(s_k | i, history) * p_aff(s_k | state)

The left factor p_LLM is the language-model likelihood that skill s_k's natural-language description is the right next step in completing i, given the history of previously chosen skills. The right factor p_aff is the robot's learned value function evaluated at the current state, scoring whether s_k is executable here. Multiplying the two filters out plans the LLM would prefer but the robot cannot perform (suggest "open the fridge" when the robot is in the bedroom), and plans the robot can perform but the LLM did not pick (greedy executable actions that do not advance the goal).

24.7.2 The Skill Library: The Hidden Engineering Cost

SayCan presupposes a library of low-level skills, each with two artifacts: a natural-language description that the LLM can score, and a learned value function that can be evaluated on the current observation. In the original paper, the skill library had 551 entries spanning navigation, pick-and-place, and door manipulation on a mobile Everyday Robot. Each skill was trained separately with imitation learning and behavioral cloning on roughly 700 to 7,000 teleoperated demonstrations. The total demonstration cost for the SayCan skill library was on the order of 100,000 trajectories.

24.7.3 The Reference Implementation

The original SayCan implementation runs a beam search over plan continuations, with each beam step scored by the SayCan product. The LLM (originally PaLM-540B; in 2026 reimplementations typically GPT-4o or Claude Sonnet) is prompted with the instruction, a list of available skills, and the history of already-chosen skills. The value function for each candidate skill is evaluated against the current observation. The top-K continuations under the joint score advance to the next iteration.

import numpy as np

class SayCanPlanner:
    def __init__(self, llm, skills):
        self.llm = llm                # anything with a logprob(text) method
        self.skills = skills          # list of (description, value_fn, policy) tuples

    def score(self, skill, instruction, history, observation):
        desc, value_fn, _ = skill
        history_str = "\n".join(f"{i+1}. {h}" for i, h in enumerate(history))
        prompt = f"Goal: {instruction}\nPlan so far:\n{history_str}\nNext step:"
        say = self.llm.logprob(desc, prompt=prompt)     # LLM "say" score
        can = np.log(value_fn(observation) + 1e-9)     # affordance "can" score
        return say + can                              # log-sum is product of probs

    def plan(self, instruction, observation_fn, max_steps=10):
        history = []
        for _ in range(max_steps):
            obs = observation_fn()
            scores = [(self.score(s, instruction, history, obs), s)
                      for s in self.skills]
            best = max(scores, key=lambda x: x[0])[1]
            if best[0] == "DONE":
                return history
            desc, _, policy = best
            policy.execute()
            history.append(desc)
        return history

24.7.4 Prompt Engineering for the LLM Side

The original SayCan paper used few-shot examples to teach PaLM what "scoring a skill description" looked like. With instruction-tuned models in 2026 (GPT-4o, Claude Sonnet 4, Gemini Ultra 2), the prompt can be much shorter, but the structure remains the same: provide the goal, the history of executed steps, and a clear ask for the LLM to rank the candidate skills. The prompt that works best in practice is:

SAYCAN_PROMPT = """You are a robot planning assistant. The robot has a fixed library of skills.
Given a goal and the steps already taken, score each candidate skill's likelihood of being the right next step.

Goal: {instruction}

Steps so far:
{history}

Candidate next skills:
{candidate_descriptions}

For each candidate, output a number from 0.0 to 1.0 representing how likely it is to be the right next step.
Output strictly as a JSON object mapping skill name to score, no other text.
"""

24.7.5 What SayCan Got Right and Wrong

What SayCan got right was the separation of concerns. The LLM does world knowledge and goal decomposition, which it is good at; the robot's value functions do executability, which only the robot can know. The product is a clean two-factor model that exposes both factors for inspection: when the planner fails, you can ask "did the LLM rank the right skill highly?" and separately "did the affordance function correctly score the state?" Each is debuggable in isolation. This separation is the conceptual contribution that survived into modern stacks.

What SayCan got wrong was the skill-library assumption. The system needs a closed library of skills with hand-trained value functions, which is exactly the bottleneck VLAs are designed to remove. A 2026 SayCan-style planner uses a much smaller library (10-20 skills, where each is a "VLA conditioned on a sub-instruction"), and the LLM is responsible for decomposing the goal into sub-instructions rather than picking from a discrete list. This is the "SayCan with a VLA backbone" pattern that dominates production multi-step robot deployments today.

24.7.6 Where SayCan-Style Planners Still Win

SayCan-style hierarchical planners remain competitive against pure end-to-end VLAs in three regimes. First, long-horizon tasks (more than ~10 sub-steps) where a flat VLA's context window cannot hold the full plan. Second, multi-robot coordination (covered in Section 24.10) where the planner must reason about which robot does which step. Third, safety-critical settings where each sub-step needs to be inspectable and pre-approved before execution; this is the operating regime for warehouse, manufacturing, and surgical robotics in 2026.

Key Takeaway

Section 24.8 moves to the second major LLM-planning paradigm: Code-as-Policies. Instead of ranking a fixed skill list, the LLM writes the plan as Python code that uses the skills as function calls. The generalization is significant; the failure modes are different.