Prompting and Using Reasoning Models

The best prompt for a reasoning model is often no prompt at all. Just ask the question and get out of the way.

Chinchilla, Minimally Prompted AI Agent

1. When to Use a Reasoning Model

The first decision is whether to use a reasoning model at all. Reasoning models are not universally better; they are better on specific task types and worse (or wastefully expensive) on others.

1.1 Decision Framework

Figure 8.4.1: Decision flowchart for choosing between standard and reasoning models. The key factors are task complexity, verifiability, latency tolerance, and cost sensitivity.

2. Prompting Differences

The most important practical insight for working with reasoning models is that many standard prompting techniques are counterproductive. Reasoning models have been trained via RL to develop their own internal reasoning strategies. External prompting instructions that attempt to control the reasoning process can interfere with these learned strategies.

2.1 What NOT to Do

2.2 What TO Do

• State the problem clearly and completely. Provide all necessary information up front. Reasoning models perform best when the problem is well-defined and unambiguous.
• Specify the desired output format separately. If you need structured output (JSON, specific format), state the format requirement after the problem, not woven into reasoning instructions.
• Use the reasoning budget parameter. Control thinking depth via the API parameter (reasoning_effort, thinkingBudget) rather than through prompt engineering.
• Provide constraints explicitly. If the answer must satisfy certain constraints (e.g., "answer must be a positive integer"), state them clearly. Reasoning models are good at checking constraints during their thinking phase.
• For complex tasks, break them into subtasks. If a task has multiple independent components, consider making separate API calls for each rather than asking the model to handle everything in one shot. This gives you finer control over reasoning budget per subtask.

3. Budget Control APIs

Each major reasoning model provider exposes parameters that control how much "thinking" the model performs. Effective use of these parameters is essential for managing both cost and latency.

3.1 OpenAI: reasoning_effort

This snippet configures the reasoning_effort parameter to control how much compute an OpenAI reasoning model uses.

from openai import OpenAI
client = OpenAI()

# Example: Using o4-mini with controlled reasoning effort
def solve_with_reasoning(problem, effort="medium"):
 """
 Solve a problem using OpenAI's reasoning model with
 explicit effort control.

 Args:
 problem: The problem statement
 effort: "low", "medium", or "high"
 low = ~100-500 thinking tokens, fast, cheap
 medium = ~500-3000 tokens, balanced (default)
 high = ~3000-50000+ tokens, maximum accuracy
 """
 response = client.chat.completions.create(
 model="o4-mini",
 reasoning_effort=effort,
 messages=[
 {
 "role": "user",
 "content": problem
 }
 ]
 )

 # The response includes usage information
 usage = response.usage
 print(f"Input tokens: {usage.prompt_tokens}")
 print(f"Reasoning tokens: {usage.completion_tokens_details.reasoning_tokens}")
 print(f"Output tokens: {usage.completion_tokens}")
 print(f"Reasoning effort: {effort}")

 return response.choices[0].message.content

# Example: Route based on estimated difficulty
def solve_with_routing(problem, difficulty_score):
 """
 Route to appropriate effort level based on difficulty.
 difficulty_score: 0.0 (trivial) to 1.0 (extremely hard)
 """
 if difficulty_score < 0.3:
 # Simple problem: use low effort to save cost
 return solve_with_reasoning(problem, effort="low")
 elif difficulty_score < 0.7:
 # Moderate problem: default effort
 return solve_with_reasoning(problem, effort="medium")
 else:
 # Hard problem: maximum thinking
 return solve_with_reasoning(problem, effort="high")

# Usage
result = solve_with_reasoning(
 "Find all integer solutions to x^3 + y^3 = z^3 where "
 "x, y, z are positive integers less than 100.",
 effort="high"
)
print(result)
# Expected: The model will reason through Fermat's Last Theorem
# and conclude there are no solutions.

3.2 Anthropic: Extended Thinking

This snippet enables extended thinking in the Anthropic API, giving the model a dedicated token budget for internal reasoning.

# Anthropic extended thinking example (pseudocode)
import anthropic
client = anthropic.Anthropic()

response = client.messages.create(
 model="claude-sonnet-4-20250514",
 max_tokens=16000,
 thinking={
 "type": "enabled",
 "budget_tokens": 10000 # Max thinking tokens (1024 to 128000)
 },
 messages=[
 {
 "role": "user",
 "content": "Prove that the sum of the first n odd numbers equals n^2."
 }
 ]
)

# The response contains separate thinking and text blocks
for block in response.content:
 if block.type == "thinking":
 print(f"Thinking ({len(block.thinking)} chars):")
 print(block.thinking[:200] + "...")
 elif block.type == "text":
 print(f"\nFinal answer:")
 print(block.text)

3.3 Google Gemini: thinkingBudget

Gemini 2.5 exposes a thinkingBudget parameter (0 to 24,576) in the generation configuration. Setting it to 0 disables thinking entirely. The model adapts its thinking length within the budget, using fewer tokens for easier problems and more for harder ones.

Controlling reasoning depth is one challenge; getting the output into a usable format is another. Reasoning models generate free-form thinking traces that do not naturally conform to structured schemas, which creates a practical obstacle for production systems that need machine-readable responses.

4. Structured Output from Reasoning Models

A common challenge with reasoning models is extracting structured output (JSON, specific formats) from the response. The thinking phase produces free-form reasoning text, and the answer phase may not always conform to strict formatting requirements.

4.1 Strategies for Structured Output

1. Post-processing: Let the model reason freely, then parse the answer from the response using regex or a secondary model call. This is the simplest approach and works well for many tasks.
2. Explicit format instruction (end of prompt): Add a format requirement at the end of your prompt: "After your analysis, provide the answer as a JSON object with keys: 'result', 'confidence', 'explanation'." This is less intrusive than embedding format instructions in the reasoning prompt.
3. Two-stage pipeline: Use the reasoning model for the hard thinking, then pass its output to a standard model (e.g., GPT-4o-mini) for formatting. The standard model is better at following format instructions and costs far less for the formatting step.
4. Structured output API: OpenAI's o-series supports the response_format parameter for JSON mode. Use response_format={"type": "json_object"} to enforce JSON output. Note that this constrains only the final answer, not the thinking tokens.

5. Best-of-N Sampling in Practice

Even without a dedicated reasoning model, you can apply test-time compute scaling to any model using best-of-N sampling with a reward model. This approach is useful when you need reasoning improvements but cannot use a dedicated reasoning model (e.g., due to cost, latency, or deployment constraints).

5.1 Implementation Considerations

• Temperature: Use a temperature of 0.6 to 0.8 for generation. Too low (0.0) produces identical samples. Too high (>1.0) produces low-quality, incoherent reasoning.
• Reward model choice: For math, use a trained PRM or ORM. For general tasks, you can use the same model as a self-evaluator (ask it to score its own output) or use a stronger model as the judge.
• Parallelism: Generate all N samples in a single batched API call if the provider supports it, or use async requests. Best-of-N is embarrassingly parallel.
• N selection: Start with N=8 and increase if accuracy is insufficient. Diminishing returns typically set in around N=32 to 64. Profile the cost/accuracy curve on your evaluation set before choosing N for production.

6. Common Pitfalls

6.1 Over-Thinking on Simple Tasks

Reasoning models can spend thousands of tokens deliberating over questions that have obvious answers. "What is the capital of France?" might generate 100+ thinking tokens as the model considers, verifies, and double-checks before answering "Paris." This is harmless for a single query but wasteful at scale.

Mitigation: Always implement a difficulty-based routing layer. Use a lightweight classifier (or even simple heuristics like query length and presence of mathematical operators) to route simple queries to standard models.

6.2 Thinking Token Cost Surprises

Many developers are surprised by reasoning model costs because they estimate based on output token counts without considering thinking tokens. A response that appears to be 200 tokens might have consumed 5,000 thinking tokens that are charged but not visible in the response.

Mitigation: Always monitor the reasoning_tokens field in the usage response. Set up alerts for queries that exceed expected thinking token budgets. Use the reasoning_effort parameter to cap thinking when appropriate.

6.3 Hallucination in Reasoning Chains

Reasoning models can produce extended, coherent, step-by-step arguments that reach incorrect conclusions. The very fluency and detail of the reasoning trace can make these errors more convincing than a simple wrong answer from a standard model. A user who sees 2,000 tokens of careful reasoning is more likely to trust the (wrong) conclusion than a user who sees a terse, unsupported answer.

Mitigation: For critical applications, combine reasoning models with independent verification. Use the reasoning trace to identify specific factual claims, then verify those claims against trusted sources. For math and code, always verify the final answer computationally when possible.

6.4 Latency for Interactive Applications

Reasoning models are fundamentally slower than standard models because they generate many more tokens per query. A typical reasoning response takes 5 to 30 seconds, compared to 0.5 to 3 seconds for a standard model. This makes them unsuitable for real-time interactive applications where users expect sub-second responses.

Mitigation: For interactive use cases, stream the response (if the API supports streaming of thinking tokens) so the user sees activity while the model thinks. Alternatively, show a "thinking..." indicator with an estimated completion time. For truly latency-sensitive applications, use standard models with chain-of-thought prompting as a faster alternative.

Exercises