Synthetic Reasoning Data

"The surprising discovery was not that large models can reason, but that small models can learn to reason when trained on the right data."

Synth, Surprisingly Optimistic AI Agent

1. Why Reasoning Data Is Different

Standard synthetic data generation produces input/output pairs: a question and its answer, a prompt and its completion. Reasoning data adds an intermediate layer: the thinking process that connects the question to the answer. This distinction matters because models trained on reasoning traces develop qualitatively different capabilities. They learn to decompose problems, verify intermediate steps, and recover from mistakes within a single generation.

The shift toward reasoning-focused training data was catalyzed by three developments. First, OpenAI's o1 model (2024) demonstrated that extended chain-of-thought reasoning during inference dramatically improves performance on math, science, and coding tasks. Second, DeepSeek-R1 (2025) showed that open models can develop strong reasoning through reinforcement learning on verifiable tasks, then distill that capability into smaller models via synthetic reasoning traces. Third, the community discovered that training on reasoning traces transfers across domains: a model trained to reason through math problems also reasons better about code, logic puzzles, and multi-step planning tasks.

What Makes a Good Reasoning Trace

Not all chain-of-thought data is equally useful. High-quality reasoning traces share several properties:

• Step decomposition: The trace breaks the problem into clear, sequential sub-problems rather than jumping to the answer.
• Verification steps: The trace includes self-checks ("Let me verify this by...") that catch and correct errors mid-reasoning.
• Exploration and backtracking: Strong traces occasionally consider wrong approaches, recognize the error, and redirect. This teaches the model that reasoning is not always linear.
• Correct final answers: The trace must arrive at the right answer. Traces with elegant reasoning but wrong conclusions are actively harmful for training.

2. The Reasoning Data Pipeline

Generating synthetic reasoning data follows a pipeline with distinct stages. Each stage introduces opportunities for quality control that do not exist in standard data generation. Figure 13.7.1 illustrates this pipeline.

Stage 1: Curating Seed Problems

The pipeline begins with a bank of problems that have verifiable answers. This is the strongest constraint: you need problems where you can programmatically check whether the final answer is correct. The most common sources include:

• Math competition problems with numerical answers (MATH, GSM8K, AMC/AIME datasets)
• Coding challenges with unit tests (LeetCode, CodeForces, Project Euler)
• Logic puzzles with deterministic solutions
• Science questions with known quantitative answers

For domains without easy verification (creative writing, open-ended analysis), you can use LLM-as-judge evaluation, but the signal is noisier. The strongest reasoning data pipelines stick to verifiable domains where correctness is unambiguous.

Stage 2: Generating Multiple Traces via Rejection Sampling

The core technique for producing high-quality reasoning data is rejection sampling: generate many candidate reasoning traces for each problem, then keep only the ones that arrive at the correct answer. This approach leverages a fundamental property of language models. Even when a model's pass@1 accuracy on a problem is only 30%, its pass@64 (getting it right in at least one of 64 attempts) might be 95%. By sampling many completions and filtering for correctness, you extract the model's best reasoning even on problems it usually gets wrong. Code Fragment 13.7.2 shows this approach in practice.

# Generate reasoning traces with rejection sampling
# Only traces that produce the correct final answer are kept for training
import openai
import re
from typing import List, Dict, Optional

client = openai.OpenAI()

def extract_final_answer(trace: str) -> Optional[str]:
 """Extract the boxed answer from a reasoning trace."""
 # Look for common answer formats
 patterns = [
 r"\\boxed\{(.+?)\}",
 r"[Ff]inal [Aa]nswer:\s*(.+?)(?:\n|$)",
 r"[Tt]he answer is\s*(.+?)(?:\.|$)",
 ]
 for pattern in patterns:
 match = re.search(pattern, trace)
 if match:
 return match.group(1).strip()
 return None

def rejection_sample(
 problem: str,
 correct_answer: str,
 n_samples: int = 64,
 temperature: float = 0.7,
 model: str = "o3-mini",
) -> List[Dict]:
 """
 Generate multiple reasoning traces and keep only correct ones.

 Returns list of dicts with 'trace' and 'answer' keys.
 """
 correct_traces = []

 # Generate in batches to manage API costs
 batch_size = min(n_samples, 16)
 for batch_start in range(0, n_samples, batch_size):
 responses = client.chat.completions.create(
 model=model,
 messages=[
 {"role": "system", "content": (
 "Solve this problem step by step. Show your reasoning "
 "clearly. End with 'Final Answer: [your answer]'."
 )},
 {"role": "user", "content": problem},
 ],
 n=batch_size,
 temperature=temperature,
 )

 for choice in responses.choices:
 trace = choice.message.content
 extracted = extract_final_answer(trace)

 if extracted and normalize(extracted) == normalize(correct_answer):
 correct_traces.append({
 "trace": trace,
 "answer": extracted,
 "problem": problem,
 })

 return correct_traces

def normalize(answer: str) -> str:
 """Normalize answer for comparison (strip whitespace, lowercase)."""
 return answer.strip().lower().replace(" ", "")

# Example usage
problem = "If 3x + 7 = 22, what is the value of x?"
correct = "5"

traces = rejection_sample(problem, correct, n_samples=32)
print(f"Generated {len(traces)} correct traces out of 32 samples")
print(f"Pass rate: {len(traces)/32:.1%}")
print(f"\nSample trace (first 200 chars):\n{traces[0]['trace'][:200]}...")

Stage 3: Verification Strategies

Verification is what separates useful reasoning data from hallucinated reasoning data. The verification strategy depends on the domain:

3. The R1-Style Pipeline: From RL to Distillation

DeepSeek-R1 introduced a two-phase approach to reasoning capability that has become a template for the field. Understanding this pipeline clarifies why synthetic reasoning data is so powerful.

Phase 1: Reinforcement Learning on the Teacher

The teacher model (DeepSeek-R1) was trained using Group Relative Policy Optimization (GRPO) on problems with verifiable answers. The model received a reward of +1 for correct answers and 0 for incorrect ones, with no supervision of the reasoning process itself. The key insight: given only outcome-based rewards, the model spontaneously developed chain-of-thought reasoning, self-verification, and backtracking behaviors. Nobody told the model to "think step by step"; it discovered that explicit reasoning improved its reward.

Phase 2: Distillation into Smaller Models

Once the teacher develops strong reasoning, its traces become training data for smaller student models via supervised fine-tuning (SFT). The DeepSeek team distilled R1's reasoning into models as small as 1.5B parameters, and remarkably, the 14B distilled model outperformed much larger models that had not been trained on reasoning data. This demonstrates that the data matters more than the model size for reasoning capabilities.

4. Formatting Reasoning Data for Training

Reasoning traces need specific formatting to work well as training data. The format must clearly separate the thinking process from the final answer, so the model learns when to reason and when to produce output. Code Fragment 13.7.2 shows this approach in practice.

# Structure reasoning traces as JSON with step-level annotations
# Each trace includes the problem, reasoning chain, and verified answer
import json
from typing import List, Dict

def format_reasoning_sft(
 traces: List[Dict],
 system_prompt: str = "You are a helpful assistant that thinks step by step.",
 thinking_tags: bool = True,
) -> List[Dict]:
 """
 Format verified reasoning traces as chat-format SFT examples.

 Args:
 traces: List of dicts with 'problem', 'trace', 'answer' keys
 system_prompt: System message for the chat template
 thinking_tags: Whether to wrap reasoning in <think> tags

 Returns:
 List of conversation dicts ready for SFT training
 """
 formatted = []

 for item in traces:
 if thinking_tags:
 # Format with explicit thinking delimiters
 # This teaches the model to separate reasoning from output
 assistant_content = (
 f"<think>\n{item['trace']}\n</think>\n\n"
 f"The answer is {item['answer']}."
 )
 else:
 # Plain format: reasoning flows directly into answer
 assistant_content = (
 f"{item['trace']}\n\n"
 f"Final Answer: {item['answer']}"
 )

 conversation = {
 "messages": [
 {"role": "system", "content": system_prompt},
 {"role": "user", "content": item["problem"]},
 {"role": "assistant", "content": assistant_content},
 ]
 }
 formatted.append(conversation)

 return formatted

# Example: format traces and save as JSONL
traces = [
 {
 "problem": "A train travels 120 km in 1.5 hours. What is its speed in km/h?",
 "trace": (
 "I need to find speed given distance and time.\n"
 "Speed = Distance / Time\n"
 "Speed = 120 km / 1.5 hours\n"
 "Speed = 80 km/h\n"
 "Let me verify: 80 km/h * 1.5 h = 120 km. Correct."
 ),
 "answer": "80 km/h",
 },
 {
 "problem": "What is the sum of the first 10 positive integers?",
 "trace": (
 "I can use the formula: n(n+1)/2\n"
 "With n = 10: 10 * 11 / 2 = 110 / 2 = 55\n"
 "Let me verify by adding: 1+2+3+4+5+6+7+8+9+10\n"
 "= (1+10) + (2+9) + (3+8) + (4+7) + (5+6)\n"
 "= 11 + 11 + 11 + 11 + 11 = 55. Confirmed."
 ),
 "answer": "55",
 },
]

sft_data = format_reasoning_sft(traces, thinking_tags=True)

# Write to JSONL for training
output_path = "reasoning_sft_data.jsonl"
with open(output_path, "w") as f:
 for example in sft_data:
 f.write(json.dumps(example) + "\n")

print(f"Wrote {len(sft_data)} examples to {output_path}")
print(f"\nSample formatted message:")
print(json.dumps(sft_data[0]["messages"][2], indent=2))

Choosing Between SFT and RL for Training

Once you have reasoning data, there are two main approaches to training:

• Supervised Fine-Tuning (SFT) on traces: The simplest approach. Train the student model to predict the reasoning trace token by token. Works well for distillation where you have high-quality traces from a strong teacher. The student learns to imitate the teacher's reasoning patterns.
• GRPO with outcome-based rewards: The student generates its own traces and receives rewards based on answer correctness. This is harder to set up but allows the student to develop its own reasoning style rather than mimicking the teacher. GRPO requires only a reward function, not curated traces.

In practice, the most effective approach combines both: start with SFT on teacher traces to initialize the student's reasoning ability, then refine with a short RL phase using GRPO. The SFT phase provides a warm start so the RL phase converges faster. DeepSeek found that SFT-then-GRPO outperformed either approach alone.

5. Quality vs. Quantity Tradeoffs

A persistent question in reasoning data generation is whether to invest in generating more traces or in improving trace quality. Empirical evidence from multiple teams suggests the following guidelines:

The DeepSeek-R1 distillation used approximately 800K reasoning examples. However, ablation studies showed that 80% of the benefit came from the first 100K examples. Beyond that, the returns diminished steeply. For most practitioners working with smaller budgets, 50K to 100K verified reasoning traces across diverse problem types represents the practical optimum.

6. Common Pitfalls and Mitigations

Reward Hacking in Verification

Models can learn to game simple verification. For math problems, a model might generate a trace with incorrect reasoning but guess the correct numerical answer. Mitigation: verify not just the final answer but also key intermediate steps. For coding tasks, use diverse test cases rather than a single test.

Reasoning Collapse

When training on traces from a single teacher model, the student may learn a narrow set of reasoning patterns and fail on problems requiring different approaches. Mitigation: use multiple teacher models (for example, both o3-mini and DeepSeek-R1) and include traces with different reasoning styles for the same problem.

Length Bias

Rejection sampling naturally favors longer traces because they explore more possibilities and are more likely to stumble on the right answer. If uncontrolled, this produces a student that generates unnecessarily verbose reasoning. Mitigation: among correct traces for a given problem, prefer shorter ones. Alternatively, apply a length penalty during selection.

7. Practical Considerations for Practitioners

Cost Estimation

Generating reasoning data is expensive because each problem requires many samples. A rough cost model: generating 64 traces per problem with o3-mini at approximately 4,000 output tokens per trace costs about $0.50 per problem (at early 2025 pricing). For a 10,000-problem dataset, that is roughly $5,000 in API costs. Using open models like DeepSeek-R1 on your own hardware reduces this to compute costs only, but requires significant GPU resources.

When to Generate Reasoning Data

Synthetic reasoning data is most valuable when:

• You need a small model (1B to 14B) that reasons well on a specific domain
• Your task involves multi-step problem solving (math, code, planning, analysis)
• You have a way to verify correctness automatically
• You have access to a strong reasoning model as a teacher

It is less valuable when your task is primarily about style, tone, or creative generation, where "correctness" is subjective and verification is unreliable.