Test-Time Compute & Reasoning Models

"A wise person does not always answer faster. Sometimes wisdom means pausing to think."

Quant, Thoughtfully Pausing AI Agent

1. The Test-Time Compute Paradigm

For years, the dominant recipe for improving LLM performance was straightforward: train a bigger model on more data. The scaling laws formalized by Kaplan et al. (2020) and Hoffmann et al. (2022) showed that loss decreases predictably as you increase model parameters and training tokens. But this approach has diminishing returns: doubling model size requires roughly doubling both training compute and serving costs, while the performance gain follows a power law with a small exponent.

Test-time compute scaling offers a different tradeoff. Instead of making the model permanently larger (which increases cost for every query, whether easy or hard), you let the model "think longer" only when it needs to. The key insight is that many problems benefit from extended reasoning: breaking a problem into steps, checking intermediate results, exploring alternative approaches, and self-correcting errors. By generating additional "thinking tokens" at inference time, a smaller model can match or exceed the performance of a much larger model on challenging tasks, while remaining fast and cheap on easy ones.

This paradigm shift was first demonstrated at scale by OpenAI's o1 model in late 2024, which showed that a model trained with reinforcement learning to produce extended chain-of-thought reasoning could achieve dramatic improvements on mathematics, coding, and scientific reasoning benchmarks. The results were striking: on the American Invitational Mathematics Examination (AIME), o1 scored comparably to top human competitors, while standard GPT-4 level models struggled with even basic problems from the same exam.

2. Reasoning Models: Architecture and Training

2.1 The "Thinking Tokens" Paradigm

Reasoning models generate two distinct types of output: thinking tokens (internal reasoning that the user may or may not see) and answer tokens (the final response). During inference, the model first generates a potentially lengthy chain of thought, exploring the problem space, checking its own work, and refining its approach. Only after this thinking phase does it produce the visible answer. The thinking tokens are typically enclosed in special delimiters (such as <think>...</think>) and may be hidden from the end user or shown as a collapsible "reasoning trace."

This is not simply prompt engineering with "think step by step" instructions. Reasoning models are specifically trained (typically through reinforcement learning) to produce high-quality internal reasoning. The RL training reward signal comes from verifiable outcomes: did the model get the math problem right? Did the generated code pass all test cases? This outcome-based training teaches the model to develop effective reasoning strategies organically, rather than mimicking the surface form of chain-of-thought demonstrations.

2.2 Key Reasoning Models

OpenAI o1, o3, and o4-mini. The o-series models pioneered commercial reasoning at scale. Released starting in late 2024, o1 demonstrated that reinforcement learning could train models to produce extended internal reasoning chains. The model generates a hidden chain of thought before every response, with the thinking process sometimes consuming thousands of tokens. The o3 and o4-mini models refined this approach with improved training recipes and more efficient architectures. o4-mini, in particular, showed that smaller models trained for reasoning can outperform much larger standard models on hard benchmarks, while keeping costs manageable for production workloads.

DeepSeek R1. Released in January 2025, DeepSeek R1 is an open-weight reasoning model that demonstrated a remarkably simple training recipe. The team applied group relative policy optimization (GRPO) directly to a base language model, rewarding correct answers on math and coding tasks, without any supervised fine-tuning on human-written chain-of-thought examples. The model spontaneously developed reasoning behaviors (self-verification, backtracking, exploring alternatives) purely through RL training. DeepSeek R1 matched or exceeded o1-preview on several benchmarks while being fully open-weight, enabling the research community to study and build upon its approach. The training recipe is described in detail in DeepSeek-AI (2025).

Google Gemini 2.5 Thinking Mode. Google's Gemini 2.5 models include an optional "thinking" mode that enables extended reasoning. When activated, the model generates a visible thinking trace before its answer, similar to the o-series approach. The Gemini implementation is notable for its integration with multimodal inputs (discussed in Section 27.1): the model can reason over images, charts, and documents, not just text.

Anthropic Claude with extended thinking. Claude models also support an extended thinking mode where the model produces a structured reasoning trace before answering. The thinking budget can be configured by the API caller, allowing fine-grained control over the compute/quality tradeoff.

3. Process Reward Models (PRMs)

A standard reward model (often called an Outcome Reward Model, or ORM) evaluates only the final answer. Given a question and a complete response, the ORM assigns a single scalar score. This is the approach used in standard RLHF, as covered in Section 17.1. However, for multi-step reasoning, outcome-level reward has a critical limitation: it provides no signal about where the reasoning went wrong. A model that makes a correct guess for the wrong reasons receives the same reward as one that reasons flawlessly.

Process Reward Models (PRMs) address this by scoring each step of the reasoning chain independently. Given a partial solution up to step k, the PRM predicts the probability that continuing from this step will eventually reach a correct answer. This provides dense, step-level feedback that can guide both training and inference-time search.

3.1 Training PRMs

The seminal work on PRMs is Lightman et al. (2023), which introduced the PRM800K dataset: 800,000 step-level human annotations on mathematical reasoning chains. Each step in a solution is labeled as "correct," "incorrect," or "neutral." Training a reward model on these annotations produces a PRM that can evaluate reasoning quality at each intermediate step.

A key finding from this work is that PRMs substantially outperform ORMs when used to select among multiple candidate solutions. Given 100 candidate solutions to a math problem, ranking them by the PRM's minimum step-level score (the "worst step" heuristic) identifies correct solutions far more reliably than ranking by a single outcome score. The Math-Shepherd project (Wang et al., 2024) extended this approach by automating the collection of step-level labels, using Monte Carlo rollouts to estimate whether each step leads to a correct answer. This automation makes PRM training scalable without requiring expensive human annotations for every step.

3.2 How PRMs Enable Better Reasoning

PRMs unlock several powerful inference-time strategies:

• Best-of-N with step-level scoring: Generate N candidate reasoning chains and select the one with the highest minimum PRM score across all steps. This catches chains that arrive at the right answer through flawed reasoning.
• Step-level beam search: At each reasoning step, generate multiple continuations, score them with the PRM, and keep only the top-scoring branches. This prunes bad reasoning paths early, before they waste compute.
• Training signal for RL: Use the PRM as a dense reward signal during reinforcement learning, providing feedback at every step rather than only at the end of a complete solution. This dramatically improves sample efficiency during RL training.

4. Search at Inference Time

The combination of a reasoning model and a verifier (PRM or ORM) naturally leads to search at inference time. Rather than generating a single reasoning chain and hoping it is correct, the system generates multiple candidates and uses the verifier to select or guide the best one. This is conceptually similar to how AlphaGo combined a neural network with Monte Carlo Tree Search; the model provides intuition and the search procedure provides reliability.

4.1 Best-of-N Sampling

The simplest search strategy is best-of-N: generate N independent solutions to a problem, score each one with a verifier, and return the highest-scoring solution. Despite its simplicity, best-of-N is remarkably effective. On the MATH benchmark, generating 256 solutions and selecting the best one (as scored by a PRM) can boost accuracy from roughly 50% (single sample) to over 85%. The cost scales linearly with N, but the returns diminish logarithmically, so there is a practical sweet spot (typically N = 16 to 64 for most applications).

4.2 Monte Carlo Tree Search (MCTS) for Reasoning

MCTS extends best-of-N by introducing structured exploration. Instead of generating complete solutions independently, MCTS builds a tree where each node represents a partial reasoning state and each edge represents a reasoning step. The algorithm iterates through four phases: (1) selection, using an upper confidence bound to balance exploration and exploitation; (2) expansion, generating new reasoning steps from promising nodes; (3) simulation, rolling out the reasoning to completion; and (4) backpropagation, updating node values based on the outcome. A PRM provides the value estimates at each node, replacing the random rollouts used in traditional MCTS with informed step-level evaluations.

4.3 The Compute-Optimal Frontier

A central question in test-time compute scaling is: when should you think more versus use a bigger model? Snell et al. (2024) systematically studied this question and found that the answer depends on problem difficulty. For easy problems, a standard model answers correctly on the first attempt, so additional thinking wastes compute. For moderately hard problems, best-of-N sampling with a small model can match a much larger model at lower total cost. For extremely hard problems, even extensive search with a small model cannot compensate for fundamental capability gaps, and a larger model is necessary.

The practical implication is a routing strategy: classify incoming queries by difficulty and route them to the appropriate compute tier. Easy queries go to a fast, cheap model with no extra thinking. Medium queries go to a reasoning model with moderate thinking budget. Hard queries go to a large reasoning model with extensive search. This adaptive allocation can reduce overall inference costs by 3x to 10x compared to sending every query to the most capable model. We revisit routing architectures in the context of Section 29.4.

Why test-time compute changes the optimization calculus. Every technique in this chapter so far (quantization, KV cache optimization, speculative decoding, pruning) optimizes for the same objective: minimize the cost of producing each token. Reasoning models invert this objective. They deliberately spend more tokens (and more compute) per query because the marginal value of a correct answer on hard problems far exceeds the marginal cost of extra tokens. This means the serving infrastructure you build must support both modes: high-throughput, low-latency serving for easy queries (using all the techniques from Sections 9.1 through 9.5) and high-compute, variable-latency serving for reasoning queries. The frameworks discussed in Section 9.4 (vLLM, SGLang) are evolving to support this dual mode, but the optimal architecture for mixed reasoning/standard workloads remains an active area of engineering.

5. Using Reasoning Models in Practice

5.1 API Usage

Reasoning model APIs follow the same chat completion interface as standard models, but with additional parameters to control thinking behavior. Code Fragment 9.6.1 demonstrates how to call a reasoning model and inspect both the thinking trace and the final answer.

# Call a reasoning model (o1/o3) with extended thinking enabled.
# The response includes the internal reasoning trace alongside the answer.
from openai import OpenAI

client = OpenAI()

# Call a reasoning model with extended thinking
response = client.chat.completions.create(
 model="o4-mini", # reasoning model
 messages=[
 {
 "role": "user",
 "content": (
 "Find all positive integers n such that "
 "n^2 + 2n + 3 is divisible by n + 1."
 )
 }
 ],
 # Some APIs allow controlling thinking budget:
 # max_completion_tokens includes both thinking + answer tokens
 max_completion_tokens=16000
)

# The response includes both reasoning and answer
message = response.choices[0].message
print("=== Final Answer ===")
print(message.content)

# Access thinking tokens (if exposed by the API)
# Note: OpenAI o-series models hide thinking tokens by default;
# other providers (DeepSeek, Anthropic) may expose them.
if hasattr(message, "reasoning_content") and message.reasoning_content:
 print("\n=== Thinking Trace (first 500 chars) ===")
 print(message.reasoning_content[:500])

# Cost awareness: reasoning models consume many more tokens
usage = response.usage
print(f"\nTokens used: {usage.total_tokens}")
print(f" Prompt tokens: {usage.prompt_tokens}")
print(f" Completion tokens: {usage.completion_tokens}")
# For o-series, completion_tokens includes hidden thinking tokens
# A simple math question might use 2,000+ thinking tokens
# A hard problem might use 20,000+ thinking tokens

5.2 DeepSeek R1: Open-Weight Reasoning

For teams that need on-premise deployment or want to customize reasoning behavior, DeepSeek R1 provides a fully open-weight alternative. The model can be served using the same infrastructure covered in Section 9.4 (vLLM, SGLang, TGI) and exposes its thinking trace directly in the output. Code Fragment 9.6.2 shows how to use R1 with visible reasoning.

# Run DeepSeek R1 locally: generate with a thinking prompt template
# and parse the <think>...</think> tags to extract the reasoning chain.
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load DeepSeek R1 (or a distilled variant for testing)
model_name = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
 model_name,
 torch_dtype=torch.bfloat16,
 device_map="auto"
)

prompt = "How many prime numbers are there between 100 and 130?"
messages = [{"role": "user", "content": prompt}]
input_text = tokenizer.apply_chat_template(
 messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

# Generate with enough tokens for thinking + answer
outputs = model.generate(
 **inputs,
 max_new_tokens=4096,
 temperature=0.6,
 do_sample=True
)

full_response = tokenizer.decode(
 outputs[0][inputs["input_ids"].shape[1]:],
 skip_special_tokens=True
)

# R1 outputs thinking inside <think>...</think> tags
if "<think>" in full_response and "</think>" in full_response:
 think_start = full_response.index("<think>") + len("<think>")
 think_end = full_response.index("</think>")
 thinking = full_response[think_start:think_end].strip()
 answer = full_response[think_end + len("</think>"):].strip()
 print(f"Thinking ({len(thinking.split())} words):")
 print(thinking[:300] + "...")
 print(f"\nFinal answer: {answer}")
else:
 print(full_response)

5.3 Cost and Latency Considerations

Thinking tokens are not free. Each thinking token incurs the same per-token cost as an output token (and sometimes more, depending on the provider). A reasoning model that generates 10,000 thinking tokens before producing a 200-token answer costs roughly 50x more than a standard model that produces the same 200-token answer directly. The latency impact is similarly significant: those 10,000 thinking tokens must be generated sequentially before the answer begins streaming, creating a noticeable delay.

For production systems, the key practices are:

• Set thinking budgets: Most reasoning model APIs allow you to cap the maximum number of thinking tokens. For well-defined tasks, experiment to find the minimum budget that maintains accuracy.
• Use streaming for user experience: Even when the thinking phase takes 30 seconds, streaming the thinking trace (where available) gives users feedback that the model is working, reducing perceived latency.
• Batch hard problems: If latency is not critical (offline analysis, batch processing), reasoning models provide the best accuracy per dollar on hard tasks. The cost premium is justified when correctness matters more than speed.
• Monitor token usage: Track thinking token consumption per query type. Unexpected spikes in thinking tokens often indicate that the model is struggling with ambiguous inputs that could be clarified through better prompting (see Section 11.1).

6. The Compute-Optimal Frontier: Think More or Use a Bigger Model?

Snell et al. (2024) formalized the tradeoff between test-time compute and model size. Their key finding is that the optimal strategy depends on a "difficulty curve": for each problem, there exists a crossover point where additional test-time compute on a smaller model becomes more expensive than simply querying a larger model once. Below this crossover, test-time compute wins. Above it, model scale wins.

The practical framework for deciding between test-time compute and model scale involves three factors:

1. Problem difficulty distribution: If most of your queries are hard (e.g., competitive programming, advanced mathematics), investing in a larger model may be more cost-effective than scaling test-time compute on a small one. If difficulty varies widely, adaptive routing yields the best results.
2. Latency requirements: Test-time compute introduces variable latency proportional to problem difficulty. For real-time applications (chatbots, autocomplete), this variability can be unacceptable. For batch or offline applications, it is irrelevant.
3. Verifier quality: The effectiveness of test-time search depends critically on having a good verifier. For domains where correctness can be checked automatically (math, code, formal logic), test-time compute is highly effective. For open-ended tasks (creative writing, summarization), verifiers are weaker, and the returns from additional thinking diminish rapidly.

Exercises