Training Reasoning Models: RLVR, GRPO, PRM

You do not teach a child to think by giving them a textbook of thoughts. You give them problems and let them struggle.

Chinchilla, Tough Love AI Agent

8.3.1 Reinforcement Learning with Verifiable Rewards (RLVR)

The standard RLHF pipeline (covered in Section 18.1) trains models using human preference data: annotators compare two model outputs and the model learns to produce outputs that humans prefer. This works well for alignment (making models helpful and harmless) but has a fundamental limitation for reasoning: human annotators cannot easily verify whether a 10,000-token mathematical proof is correct.

Reinforcement Learning with Verifiable Rewards (RLVR) sidesteps this problem entirely. Instead of relying on human judgments, RLVR uses tasks where correctness can be verified automatically:

• Mathematics: Compare the model's final numerical answer against the ground truth. The answer is either correct or incorrect, with no ambiguity.
• Code generation: Execute the model's code against a test suite. It either passes all tests or it does not.
• Formal proofs: Check the proof in a formal verification system (Lean, Coq). The proof either compiles or it does not.
• Constraint satisfaction: Verify that the output meets explicit constraints (e.g., "use exactly 100 words" or "include all specified keywords").

The key insight is that verifiable tasks provide a free, scalable, perfectly accurate reward signal. There is no need for human annotators, no inter-annotator disagreement, and no label noise. This allows RL training at a scale that would be impractical with human feedback.

Formally, RLVR optimizes the same expected-reward objective as standard RL, but with a deterministic verifier $V$ in place of a learned reward model. A KL anchor to the reference policy $\pi_{\text{ref}}$ (the supervised checkpoint) prevents the policy from drifting into reward-hacking strategies:

Here $q$ is a problem, $s$ is a sampled solution (reasoning trace plus final answer), $V(s, q) \in \{0, 1\}$ is the verifier's verdict, and $\beta$ controls how far the policy may stray from the reference (typical values 0.01 to 0.05). The gradient is taken on this objective via PPO or GRPO. The contrast with RLHF is sharp: RLHF replaces $V$ with a learned reward model $r_{\phi}$ that must itself be trained on human preference labels and can be exploited by the policy.

The end-to-end loop is a tight cycle of sampling, verifying, and updating, with the verifier providing the only learning signal, as sketched in Figure 8.3.1a.

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import GRPOConfig, GRPOTrainer
import torch, re

# Minimal RLVR loop with TRL's GRPOTrainer.
# Verifier is a Python function: no human labels, no learned reward model.

MODEL = "Qwen/Qwen2.5-Math-7B"
tok = AutoTokenizer.from_pretrained(MODEL)
policy = AutoModelForCausalLM.from_pretrained(
    MODEL, torch_dtype=torch.bfloat16, device_map="auto"
)

def math_verifier(solution: str, gold: str) -> float:
    """Extract the final answer after 'ANSWER:' and compare to the gold value."""
    m = re.search(r"ANSWER:\s*([\-\d\./]+)", solution)
    if not m:
        return 0.0
    try:
        return 1.0 if abs(float(m.group(1)) - float(gold)) < 1e-6 else 0.0
    except ValueError:
        return 0.0

# Each example in gsm8k_train is {"prompt": ..., "gold": ...}.
# GRPOTrainer samples G completions per problem, scores them with math_verifier,
# normalizes rewards within the group, and runs a PPO-style clipped update.
# No critic model is required, which halves the GPU memory vs. PPO.
trainer = GRPOTrainer(
    model=policy,
    args=GRPOConfig(
        num_generations=8,            # G samples per problem
        learning_rate=5e-6,
        beta=0.04,                    # KL weight against pi_ref
        max_prompt_length=512,
        max_completion_length=1024,
    ),
    train_dataset=gsm8k_train,
    reward_funcs=[lambda c, gold: math_verifier(c, gold)],
)
trainer.train()

8.3.1.1 The RLVR Training Loop

Algorithm 1 formalizes the RLVR training loop. The critical difference from standard RLHF is step 3: rewards come from automatic verification rather than a learned reward model.

Input: policy model pi, problem dataset D, verifier V, num_iterations T
Output: trained policy pi*

1. for iteration = 1 to T:
  a. Sample a batch of problems {p_1, ..., p_B} from D
  b. for each problem p_i:
    Generate solution s_i (reasoning trace + final answer) using pi
  c. for each solution s_i:
    r_i = V(s_i, ground_truth_i) // automatic verification
    // e.g., r_i = 1 if answer matches, 0 otherwise
  d. Optionally add format rewards (e.g., +0.1 for using <think> tags)
  e. Update pi using policy optimizer (PPO or GRPO) to maximize
    expected reward while staying close to reference policy
2. return pi* (the final policy)

Input: problem p, policy pi, reference policy pi_ref, group size G, KL weight beta
Output: policy gradient update for pi
1. Generate G solutions {s_1, ..., s_G} from pi for problem p
2. Compute rewards: r_i = verify(s_i) for i = 1 to G
3. Normalize rewards within the group:
A_i = (r_i - mean(r_1..r_G)) / std(r_1..r_G)
4. for each solution s_i with advantage A_i:
a. Compute ratio: rho_i = pi(s_i|p) / pi_old(s_i|p)
b. Clip ratio: rho_clipped = clip(rho_i, 1-eps, 1+eps)
c. L_i = min(rho_i * A_i, rho_clipped * A_i)
5. Add KL penalty: L_total = mean(L_i) - beta * KL(pi || pi_ref)
6. Update pi by gradient ascent on L_total

The power of RLVR lies in the verifier: because correctness checking is automatic, the system can generate millions of training signals without human annotators. This enables RL training at a scale that would be impractical with human feedback.

With a scalable reward signal in hand, the next question is: which RL algorithm should drive the optimization? Standard PPO requires a separate critic model that doubles GPU memory. DeepSeek developed a more efficient alternative specifically for training reasoning models.

8.3.2 Group Relative Policy Optimization (GRPO)

GRPO is the RL algorithm developed by DeepSeek for training R1. Its primary advantage over PPO (the algorithm used in standard RLHF) is that it does not require a separate critic (value) model, which halves the GPU memory requirement during training.

8.3.2.1 How GRPO Works

In standard PPO, a critic model estimates the expected reward for each state (partial generation), and the advantage function is computed as the difference between the actual reward and the critic's estimate. Training the critic model requires additional GPU memory and compute roughly equal to training the policy model itself.

GRPO eliminates the critic by computing advantages relative to a group of samples. Algorithm 2 formalizes the procedure for a single problem within the training loop.

The key insight is in step 3: by normalizing rewards within each group, GRPO converts absolute rewards into relative comparisons. A solution that scores 1.0 in a group where all others also score 1.0 receives zero advantage (nothing to learn from), while the same score in a group of mostly failures receives high positive advantage. This eliminates the need for a separate critic model to estimate expected reward, halving the GPU memory requirement.

Figure 8.3.2 shows this loop end to end: one problem fans out into a group of sampled solutions, the verifier scores each automatically, and subtracting the group mean (then dividing by the standard deviation) turns those raw scores into the advantages that push the policy toward above-average attempts and away from below-average ones.

8.3.2.2 GRPO vs. PPO

The key trade-off visible in this table is memory versus samples: PPO needs a large critic model but few samples per problem, while GRPO eliminates the critic at the cost of generating many more samples. For models at the scale of DeepSeek V3 (671B parameters), the memory savings from eliminating the critic model are decisive.

Why group normalization works intuitively. Consider grading on a curve: if a professor does not know how hard an exam was, they can still assign fair grades by comparing students to each other. GRPO does the same thing. It does not need to predict how hard a problem is (which the critic model attempts); it simply observes how the current policy performs on each problem across multiple attempts and rewards the above-average attempts. This is why GRPO needs more samples: with only two "students," the curve is meaningless. With 16 or more, the relative rankings become informative. The same principle applies when you are building reward-based fine-tuning pipelines: relative comparisons are often more robust than absolute scores.

8.3.3 Process Reward Models vs. Outcome Reward Models

Reward models play a central role in both training and deploying reasoning models. They serve two purposes: (1) providing reward signals during RL training, and (2) scoring candidate solutions during inference (best-of-N selection, tree search). There are two fundamental types:

8.3.3.1 Outcome Reward Models (ORM)

An ORM evaluates the complete solution and produces a single score. It answers the question: "How likely is this solution to be correct?" ORMs are trained on (problem, solution, correctness_label) triples, where the correctness label is a binary indicator of whether the solution produced the right final answer.

Advantages of ORMs:

• Easy to train (binary classification on complete solutions)
• Easy to label (just check if the final answer is correct)
• No need for step-level annotations

Limitations of ORMs:

• Cannot distinguish "right answer, wrong reasoning" from "right answer, right reasoning." A solution that arrives at the correct answer through a lucky error is scored the same as a logically sound derivation.
• Cannot provide feedback on where a wrong solution went astray. The ORM only says "wrong" without indicating which step caused the failure.
• Less effective for guiding tree search, since it can only score complete solutions, not partial reasoning paths.

8.3.3.2 Process Reward Models (PRM)

A PRM evaluates each reasoning step individually, producing a score for every step in the chain. It answers the question: "Is this step logically valid given the previous steps?" PRMs are trained on step-level annotations where human annotators (or automated systems) label each reasoning step as correct or incorrect.

Formally, given a reasoning trace decomposed into $T$ steps $y = (y_1, \dots, y_T)$ with step embeddings $h_t$, a PRM produces per-step probabilities $p_t = \sigma(w^\top h_t + b) \in [0, 1]$. Two common aggregation choices for selecting one of $N$ best-of-$N$ candidates are the minimum-step score (penalises any weak step) and the product score (rewards uniformly strong chains):

The $\min$ aggregation matches the empirical observation that one bad step usually ruins the final answer, while the $\prod$ aggregation is the maximum-likelihood choice when the per-step labels are conditionally independent. Lightman et al. (2023) report that $\min$ outperforms $\prod$ by 1 to 2 points on MATH best-of-N selection.

The landmark dataset for PRM training is PRM800K (Lightman et al., 2023), which contains 800,000 step-level annotations of mathematical reasoning solutions. Each step is labeled as "positive" (correct), "negative" (incorrect), or "neutral" (valid but not helpful). Math-Shepherd (Wang et al., 2024) later provided an automated alternative to human annotation, using outcome verification to automatically label steps.

To make the PRM concept concrete, the following code fragment shows a simplified implementation that scores each reasoning step using a linear classifier on top of a pretrained language model backbone.

# Simplified PRM scoring of reasoning steps
import torch
import torch.nn as nn
class ProcessRewardModel(nn.Module):
    """
    A simplified Process Reward Model that scores each
    reasoning step in a chain-of-thought solution.
    In practice, PRMs are typically built by adding a
    classification head on top of a pretrained language model.
    """
def __init__(self, base_model, hidden_size=4096):
    super().__init__()
    self.base_model = base_model # Pretrained LM backbone
    self.step_scorer = nn.Linear(hidden_size, 1) # Step-level score
def forward(self, input_ids, step_boundaries):
    """
    Args:
    input_ids: Tokenized [problem + solution] sequence
    step_boundaries: List of token positions where each
    reasoning step ends (e.g., at newlines)
    Returns:
    step_scores: Score for each reasoning step (0 to 1)
    """
    # Get hidden states from base model
    hidden_states = self.base_model(
        input_ids, output_hidden_states=True
        ).last_hidden_state # [1, seq_len, hidden_size]
    # Extract hidden state at each step boundary
    step_scores = []
    for boundary_pos in step_boundaries:
        step_hidden = hidden_states[0, boundary_pos, :]
        score = torch.sigmoid(self.step_scorer(step_hidden))
        step_scores.append(score.item())
        return step_scores
        # Example usage (pseudocode)
def score_solution_with_prm(prm, tokenizer, problem, solution_text):
    """Score each step in a reasoning solution."""
    full_text = f"Problem: {problem}\nSolution:\n{solution_text}"
    tokens = tokenizer(full_text, return_tensors="pt")
    # Identify step boundaries (e.g., at newline characters)
    step_boundaries = [
        i for i, tok in enumerate(tokens.input_ids[0])
        if tokenizer.decode([tok]) == "\n"
        ]
    step_scores = prm(tokens.input_ids, step_boundaries)
    # Print step-level scores
    steps = solution_text.split("\n")
    for i, (step, score) in enumerate(zip(steps, step_scores)):
        status = "PASS" if score > 0.5 else "FAIL"
        print(f" Step {i+1} [{status}] (score={score:.3f}): {step[:60]}...")
        # Overall solution score: product of step scores
        # (all steps must be correct for the solution to be correct)
        overall_score = 1.0
        for s in step_scores:
            overall_score *= s
            return overall_score, step_scores
            # Example output:
            # Step 1 [PASS] (score=0.97): Let x be the number of apples. We know...
            # Step 2 [PASS] (score=0.94): Setting up the equation: 3x + 7 = 22...
            # Step 3 [FAIL] (score=0.12): Dividing both sides by 3: x = 22/3 = 7...
            # Step 4 [PASS] (score=0.89): Therefore, there are 7 apples.
            # Overall score: 0.097
            # Note: Step 3 is flagged as incorrect (should be (22-7)/3 = 5)

8.3.3.3 PRM Training Methods

There are three main approaches to training PRMs:

1. Human annotation (PRM800K): Human annotators read each reasoning step and label it as correct, incorrect, or neutral. This produces high-quality labels but is expensive and slow. Lightman et al. (2023) employed mathematics graduate students for annotation, at a cost of roughly $12 per fully-annotated solution.
2. Automated step verification (Math-Shepherd): Wang et al. (2024) proposed automatically labeling steps by checking whether completing the solution from each step leads to the correct final answer. If completing from step $k$ yields the correct answer with high probability (across many completions), step $k$ is labeled as correct. If completions from step $k$ almost never reach the correct answer, step $k$ is labeled as incorrect.
3. Monte Carlo estimation: Similar to Math-Shepherd but using rollout statistics from MCTS. Each step's value is estimated by the fraction of rollouts from that step that eventually reach a correct solution.

8.3.4 STaR and Bootstrapping Approaches

Self-Taught Reasoner (STaR), proposed by Zelikman et al. (2022), is an iterative bootstrapping method that was an important precursor to modern RL-based reasoning training. The core idea is simple: use the model's own correct reasoning traces as training data for the next iteration.

8.3.4.1 The STaR Algorithm

1. Generate: For each problem in the training set, generate a reasoning trace and answer using the current model.
2. Filter: Keep only the traces that produced the correct final answer.
3. Rationalize: For problems where the model failed, provide the correct answer as a hint and ask the model to generate a reasoning trace that arrives at that answer. This "rationalization" step generates correct traces for problems the model could not solve unaided.
4. Train: Fine-tune the model on the combined set of correct traces (from step 2) and rationalized traces (from step 3).
5. Repeat: Use the improved model as the starting point for the next iteration.

Each iteration produces a model that can solve more problems correctly, which in turn generates more training data for the next iteration. This creates a positive feedback loop where the model gradually bootstraps its way to stronger reasoning.

Why bootstrapping converges instead of amplifying errors. The key safeguard is the filter in step 2: only traces that produce verifiably correct answers are kept. This means errors cannot propagate across iterations because they are removed before training. The rationalization step (step 3) is what prevents the model from plateauing: by giving the model the answer as a hint, you let it practice reasoning about problems just beyond its current ability, similar to how a student learns more from working a problem with the answer key than from skipping it entirely. This same principle underlies the rejection sampling pipelines used in practical synthetic data generation.

8.3.4.2 Quiet-STaR

Quiet-STaR (Zelikman et al., 2024) extends the STaR idea by training the model to generate internal "thoughts" at every token position, not just in response to explicit reasoning prompts. The model learns to insert hidden reasoning before generating each token, essentially making chain-of-thought a continuous, always-on process rather than something triggered by a specific prompt. This is conceptually closer to how reasoning models like o1 operate, where thinking is built into the generation process rather than being prompt-elicited.

8.3.5 Synthetic Reasoning Data Generation

Both the RL-based approaches (GRPO, PPO) and the bootstrapping methods (STaR) share a common dependency: they need large quantities of problems with verifiable answers. Training reasoning models requires large quantities of problems with verifiable answers. While math competition archives and code challenge repositories provide some data, the demand far exceeds the supply of naturally occurring verifiable problems. This creates a need for synthetic data generation (building on the techniques in Chapter 15).

8.3.5.1 Problem Synthesis Strategies

• Template-based generation: Create parameterized problem templates and fill in different values. For example, "If a train travels at {speed} km/h for {time} hours, how far does it go?" This is simple but produces limited diversity.
• LLM-generated problems: Use a strong model to generate novel math/code problems, then verify the ground truth answers automatically. This produces more diverse problems but requires careful filtering for solvability and difficulty.
• Evol-Instruct for reasoning: Adapt the Evol-Instruct methodology (see Section 15.2) to progressively increase problem difficulty: take an existing problem and ask a model to make it harder by adding constraints, increasing the number of steps, or combining multiple concepts.
• Backward generation: Start with an answer and generate a problem that has that answer. This guarantees a verifiable ground truth and can produce creative, non-standard problems.

8.3.5.2 Quality Filtering

Not all synthetically generated problems are useful for training. Common quality filters include:

• Solvability check: Verify that the problem has a unique, correct answer by solving it with multiple models and checking for agreement.
• Difficulty calibration: Estimate problem difficulty by measuring what fraction of a baseline model's attempts produce the correct answer. Problems that are too easy (>95% accuracy) or too hard (<1% accuracy) provide limited training signal.
• Deduplication: Remove problems that are semantically similar to existing training examples, to maintain diversity.

Exercises