GRPO, Reward Hacking, and Choosing an Alignment Method

Language models know a lot, but they do not know what we want. That is the alignment problem in one sentence.

Reward, Existentially Aware AI Agent

Building on the alignment framing and the PPO pipeline from Section 18.1, this part moves from the canonical RLHF recipe to the methods now competing with it. We start with GRPO, the group-relative variant that powers many reasoning-trained models, then study reward hacking and its mitigations, compare RLHF, DPO, and GRPO side by side, and close with practical tips and the infrastructure realities at scale.

18.2.1 GRPO: Group Relative Policy Optimization

Group Relative Policy Optimization (Shao et al., 2024), introduced as part of DeepSeek's training pipeline, offers an elegant simplification of PPO for LLM alignment. The core idea: instead of training a separate value network to estimate baselines, GRPO generates a group of G responses for each prompt and uses the group's reward statistics as the baseline. This eliminates one of the four models entirely, cutting memory requirements by roughly 25 to 40 percent.

18.2.1.1 How Group Normalization Replaces the Value Network

In standard PPO, the value network V(s) estimates "how good is this prompt on average?" so that the advantage A(t) = R(t) − V(s) measures whether a specific response is better or worse than expected. Training this value network requires its own forward and backward passes, its own optimizer states, and careful coordination with the policy.

GRPO sidesteps this entirely. For a given prompt x, it generates G responses {$y_{1}$, ..., $y_{G}$}, computes their rewards {$r_{1}$, ..., $r_{G}$}, and normalizes within the group:

This group-level z-score normalization serves the same purpose as a learned value baseline: it tells the algorithm which responses are above or below average for that specific prompt. The intuition is straightforward. If you generate eight responses and score them, the best ones get positive advantages and the worst ones get negative advantages, regardless of the absolute reward scale. This relative comparison is robust and requires no learned parameters.

A quick numeric example shows why this works. Suppose G = 4 responses receive rewards [0.2, 0.8, 0.5, 0.3]:

# GRPO group normalization: numeric walkthrough
import numpy as np
rewards = np.array([0.2, 0.8, 0.5, 0.3])
mean, std = rewards.mean(), rewards.std() # mean=0.45, std=0.2217
advantages = (rewards - mean) / (std + 1e-8)
print(f"mean={mean:.2f}, std={std:.4f}")
print(f"advantages={np.round(advantages, 2)}")
# advantages=[-1.13 1.58 0.23 -0.68]
# Response 2 (reward 0.8) gets the strongest positive signal.

18.2.1.2 When to Prefer GRPO Over PPO

GRPO is particularly well suited for tasks with verifiable rewards (math, coding, factual QA) where a reward function can be computed programmatically rather than learned from human preferences. DeepSeek-R1 used GRPO with outcome-based rewards (correct/incorrect) to train reasoning capabilities. For subjective tasks where the reward model itself is noisy, PPO's learned value function may provide more stable training because it smooths out reward model noise over time.

The mechanical difference is concentrated in how the advantage $\hat{A}$ is computed. PPO trains a value network $V_\phi$ and uses Generalized Advantage Estimation:

GRPO discards $V_\phi$ entirely and uses the empirical group mean and standard deviation across $G$ responses to the same prompt as a baseline:

where $r_i$ is the total reward of response $i$ in the group. Both advantages then plug into the same PPO clipped surrogate loss, so the only structural change is removing the value network and the GAE machinery.

Code Fragment 18.2.4 provides a simplified GRPO implementation showing the group normalization mechanism and clipped policy gradient.

# GRPO: Group Relative Policy Optimization (simplified)
import torch
import torch.nn.functional as F
def grpo_loss(
    policy_model,
    ref_model,
    prompts,
    tokenizer,
    reward_fn,
    group_size=8,
    beta=0.1,
    clip_range=0.2,
    ):
    """
    Simplified GRPO training step.
    Key difference from PPO: no value network.
    Advantages are computed by normalizing rewards
    within each group of responses per prompt.
    """
    all_losses = []
    for prompt in prompts:
        # Generate a group of responses
        input_ids = tokenizer.encode(prompt, return_tensors="pt")
        responses = []
        for _ in range(group_size):
            output = policy_model.generate(
                input_ids, max_new_tokens=512, do_sample=True,
                temperature=0.8, top_p=0.95,
                )
            responses.append(output[0])
            # Compute rewards for each response
            rewards = torch.tensor([
                reward_fn(prompt, tokenizer.decode(r)) for r in responses
                ])
            # Group-level normalization (replaces value network)
            normalized_rewards = (rewards - rewards.mean()) / (rewards.std() + 1e-8)
            # Compute policy gradient with clipped objective
            for response, advantage in zip(responses, normalized_rewards):
                # Log probabilities under current and reference policy
                with torch.no_grad():
                    ref_logprobs = compute_logprobs(ref_model, input_ids, response)
                    policy_logprobs = compute_logprobs(policy_model, input_ids, response)
                    # Importance ratio
                    ratio = torch.exp(policy_logprobs - ref_logprobs)
                    # Clipped surrogate objective (PPO-style)
                    surr1 = ratio * advantage
                    surr2 = torch.clamp(ratio, 1 - clip_range, 1 + clip_range) * advantage
                    policy_loss = -torch.min(surr1, surr2).mean()
                    # KL penalty
                    kl = (ref_logprobs - policy_logprobs).mean()
                    total_loss = policy_loss + beta * kl
                    all_losses.append(total_loss)
                    return torch.stack(all_losses).mean()

18.2.2 Reward Hacking and Mitigation

Reward hacking (also called reward gaming or Goodhart's Law applied to RL) is the phenomenon where the policy learns to exploit weaknesses in the reward model rather than genuinely improving response quality. It is the most common failure mode in RLHF and one of the most difficult to detect early.

18.2.2.1 Mitigation Strategies

Several complementary strategies help prevent or detect reward hacking:

• KL divergence as a leash: The KL penalty between the policy and reference model is the first line of defense. If the policy drifts too far from the SFT model, it has likely found an exploit. Adaptive KL controllers (as in TRL) automatically increase the penalty when divergence spikes.
• Reward model ensembles: Training multiple reward models on different data splits and using their consensus score reduces the surface area for exploitation. Disagreement between ensemble members signals unreliable regions of the reward landscape.
• Iterative RLHF with reward model retraining: After each round of PPO training, collect fresh human preferences on the current policy's outputs and retrain the reward model. This closes the distribution gap between the reward model's training data and the policy's actual behavior. Anthropic's training process uses multiple iterations of this loop.
• Length normalization: Normalizing the reward by response length prevents the model from learning that "longer is better." This is especially important when the preference dataset has a length bias.
• Held-out human evaluation: Automated metrics are themselves subject to Goodhart's Law. Periodic human evaluation on randomly sampled outputs catches reward hacking that the reward model misses. See Section 42.1 on evaluation methodology for frameworks.

18.2.3 RLHF vs DPO vs GRPO: Choosing an Alignment Method

With three major alignment optimization methods available, practitioners need clear guidance on which to choose. The following comparison covers the key dimensions that affect both training feasibility and outcome quality.

The three methods optimize objectives that look very different but reduce to the same KL-regularized reward-maximization principle:

PPO realizes this with an explicit reward model plus a value baseline trained with the clipped-PPO surrogate. DPO algebraically derives an implicit reward from the policy itself, eliminating the reward and value models. GRPO replaces PPO's learned value baseline with the empirical mean reward over a sampled group of $G$ responses to the same prompt, removing the value model only. The "models in memory" column of Table 18.2.1 (4 vs 2 vs 3) is the operational consequence of these algorithmic differences.

pip install trl peft datasets
from trl import DPOTrainer, DPOConfig
from datasets import load_dataset
ds = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
cfg = DPOConfig(output_dir="dpo-out", beta=0.1, learning_rate=5e-7,
                num_train_epochs=1, per_device_train_batch_size=2)
trainer = DPOTrainer(model=model, ref_model=None, args=cfg,
                     train_dataset=ds, processing_class=tokenizer)
trainer.train()

18.2.4 Practical Tips for RL-Based Alignment

The TRL recipe will run as written, but whether it converges to a useful policy or a degraded one depends on a handful of tuning decisions. The practitioners who have actually shipped RLHF or DPO models converge on the same short checklist: pick a small enough learning rate, use a cosine schedule with warmup, and watch the KL divergence as carefully as the reward. We walk through each item with concrete numbers.

Learning Rate and Schedule

Use a learning rate 10 to 100 times smaller than your SFT learning rate. Typical ranges: 1e-6 to 5e-6 for PPO, 1e-7 to 5e-7 for DPO. A cosine schedule with a warmup period of 5 to 10 percent of training steps helps stabilize early training. For PPO specifically, the learning rate for the value head can be 2 to 5 times larger than the policy learning rate.

Batch Size

Larger batches produce more stable reward and advantage estimates. For PPO, use at least 64 responses per batch (across all prompts). For GRPO, the effective batch size is the number of prompts times the group size G; a minimum of 128 total responses per update is recommended. For DPO, batch sizes of 32 to 128 preference pairs work well, with larger batches improving gradient stability.

When to Stop Training

Monitor these signals to determine when to stop:

• KL divergence exceeds threshold: If KL between policy and reference exceeds 10 to 15 nats, the policy has drifted too far. In TRL, set target_kl to trigger adaptive adjustment.
• Reward plateau with rising KL: If the reward score stops improving but KL keeps growing, the policy is likely reward hacking rather than genuinely improving.
• Validation win rate stalls: If the model's win rate against the SFT baseline (measured on held-out prompts by an LLM judge) stops increasing, further training is unlikely to help.
• Generation diversity collapse: If distinct prompts produce nearly identical responses, the policy has collapsed. Measure this with type-token ratio or self-BLEU on a held-out prompt set.

Common Failure Modes and Diagnostics

18.2.5 RLHF Infrastructure at Scale

Running RLHF at production scale is an infrastructure challenge that goes far beyond the algorithm itself. A full RLHF training run requires simultaneously managing four models: the policy model being trained, the reference model (a frozen copy), the reward model, and (in standard PPO) the value model. This quadruples the GPU memory requirements compared to standard SFT.