Why GRPO training fails.

First confirm the task is learnable

Before debugging training, establish that the setup could work in principle. Sample the base policy on training tasks and measure success. If success is essentially zero everywhere, GRPO has nothing to amplify: the group-relative advantage needs some successes to point at. Fix that with an easier curriculum tier or a supervised warm start, not with more reinforcement-learning steps.

Also confirm a competent human or scripted agent can actually solve the tasks through the same interface the model uses. Environments frequently ship with tasks that are impossible through their own tool set.

Validate reward extraction end to end

The single most common failure is that the reward the trainer receives is not the reward you think you defined. Parsers silently return zero on unexpected formats, verifiers time out and default, answer extraction misses a changed output style, and every trajectory scores 0.0 while the loop runs green.

• Hand-score 20 trajectories and compare with the pipeline's scores, one by one
• Log the raw verifier output and the parsed reward separately, so silent defaults are visible
• Alert on degenerate reward distributions, such as long streaks of identical values
• Treat verifier errors as errors, never as zero reward

Inspect group reward variance

GRPO's learning signal is the spread of rewards within each group. If every response in a group gets the same reward, the advantage is zero and the batch contributes nothing. Plot the fraction of zero-variance groups per step. A healthy run keeps it well below half. A run at 90% zero-variance groups is training on a tenth of its data.

Zero variance has three usual causes: tasks too hard (all fail), tasks too easy (all succeed), or sampling temperature so low the group is near-identical. The fixes are curriculum placement, mixture rebalancing, and more sampling diversity, in that order.

Check for sparse and saturated rewards

A binary end-of-episode reward on a 40-step task gives one bit of signal for thousands of generated tokens, and learning will be slow at best. Add partial credit from verifiable milestones where they exist. At the other end, a reward already near its maximum leaves no headroom, and continued training only encourages exploitation of the remaining gap between the reward and true quality.

Check rollout diversity and sampling configuration

Read a sample of actual rollouts regularly. Degenerate diversity, groups of near-identical completions, repetitive loops, or an early collapse into one strategy all cap what the method can learn. Verify the sampling parameters the rollout fleet really uses (temperature, top-p, max tokens): configuration drift between the intended and deployed sampling setup is common and invisible in aggregate metrics.

Check environment termination and reset behavior

Episodes that terminate for the wrong reason poison credit assignment: crashes recorded as failures, hangs recorded as truncations, state leaking across resets so task difficulty silently drifts. Audit the distribution of termination reasons per task family, and replay outlier episodes from their seeds. If replay does not reproduce them, you have a nondeterminism bug feeding noise into the reward.

Check format and parsing failures

When rewards depend on output format (an answer tag, a JSON schema, a tool-call syntax), track the parse-failure rate as its own metric. Two failure patterns matter: a high steady rate means the reward is mostly measuring formatting rather than the capability, and a rate that collapses to zero alongside rising reward may mean the policy found a formatting exploit rather than better answers.

Inspect KL divergence and policy drift

The Kullback-Leibler (KL) term anchors the policy to its reference model. Watch its trajectory rather than its value at one step. Exploding KL with rising reward and falling held-out quality is the signature of reward hacking. KL pinned near zero with a flat reward means the constraint (or an over-large penalty coefficient) is preventing any movement. Ratcheting KL alongside repetitive rollouts signals collapse onto a narrow strategy.

Change the KL coefficient only after rewards, variance, and rollouts have been cleared. KL tuning is where healthy runs get broken chasing a symptom whose cause was upstream.

Inspect sequence length and truncation

If completions regularly hit the token budget, truncated responses collect whatever reward the truncated state earns, usually failure, and the policy learns length aversion instead of the task. Track the truncation rate and the reward-versus-length curve. Length is also a classic hacking channel in judged settings: if longer answers score better at equal quality, the judge is measuring length.

Check stale rollouts and weight synchronization

In a disaggregated setup, samplers serve the policy while the trainer updates it, and the two must resynchronize on schedule. Broken or lagging weight sync means training on trajectories from an old policy: the update math degrades quietly and reward curves go flat or unstable without an obvious cause. Log the policy version on every trajectory and alert when the training batch's version lag exceeds the tolerance you chose on purpose.

A related mismatch hides in the numerics: inference engines and training code can disagree slightly on log-probabilities for the same tokens. Verify that the importance-correction path between your sampler and trainer is the one your framework documents, especially after upgrading either side.

Check leakage and evaluate on held-out tasks

Rising training reward with flat held-out capability usually means the policy is learning the training distribution's quirks: repeated task instances, verifier gaps, or curriculum artifacts. Confirm the held-out families are truly excluded from every training path, then select checkpoints exclusively on held-out results. Training reward is a diagnostic, never a selection criterion.

Triage flowchart

flat or broken GRPO run
│
├─ base policy success ≈ 0 on training tasks?
│    └─ yes → easier tier or SFT warm start. Stop here first.
├─ hand-scored rewards ≠ pipeline rewards?
│    └─ yes → fix extraction/verifier defaults. Rerun.
├─ zero-variance groups > ~50%?
│    └─ yes → rebalance difficulty, raise sampling diversity.
├─ rollouts degenerate, wrong sampling config, bad terminations?
│    └─ yes → fix environment/sampler. Replay from seeds.
├─ parse failures high, or truncation rate high?
│    └─ yes → fix format handling and budgets before tuning.
├─ KL exploding with rising reward?
│    └─ yes → reward hacking: harden the verifier, then resume.
├─ trajectory policy-version lag beyond tolerance?
│    └─ yes → fix weight sync / logprob path.
└─ training reward up, held-out flat?
     └─ leakage or overfit distribution → fix splits, reselect
        checkpoints on held-out tasks.

What to log so diagnosis is possible

Every check above assumes the run records enough to answer it. Log these per step or per trajectory from the first experiment:

1. Reward distribution per task family, plus raw verifier output next to parsed reward.
2. Fraction of zero-variance groups.
3. Success rate per curriculum tier for the current policy.
4. Parse-failure rate and truncation rate.
5. Completion-length distribution and reward-versus-length curve.
6. KL divergence to the reference, and entropy of the policy.
7. Termination-reason counts per task family.
8. Policy version per trajectory and version lag per training batch.
9. Held-out evaluation on a fixed cadence with fixed seeds.
10. Environment version, reward version, and sampler config hash on every run record.