SFT vs DPO vs GRPO: choosing the right post-training stage.

The three methods solve different supervision problems

SFT, DPO, and GRPO are often presented as competing techniques. They are not. Each converts a different kind of supervision into weight updates, and the practical question is which kind of supervision your capability can produce at sufficient quality and volume.

• SFT consumes demonstrations: complete examples of the target behavior.
• DPO consumes comparisons: pairs where one response is judged better than another.
• GRPO consumes rewards: scores computed over groups of responses the model samples itself.

That ordering also tracks difficulty. Demonstrations are the easiest signal to collect and the easiest training to run. Rewards are the hardest to build honestly, and online reinforcement learning is the hardest training to operate. Skipping ahead to the harder method without the supervision it needs is the most common method-selection mistake.

SFT: demonstrate the target behavior

Supervised fine-tuning trains the model to imitate demonstrations with a next-token loss. If experts can write or curate correct outputs for representative inputs, SFT is the most direct and most stable way to move the model, and it establishes the warm start every later stage builds on.

SFT is limited by its own signal. It teaches what good looks like but not what bad looks like, so it cannot push probability away from plausible failures. It also inherits every flaw in the demonstrations: style quirks, distribution gaps, and mistakes all get imitated with equal enthusiasm.

If the demonstrations do not match the deployment distribution, SFT teaches the wrong lesson very efficiently.

DPO: teach relative preference

Direct Preference Optimization trains on chosen-versus-rejected pairs, raising the likelihood margin of the preferred response relative to a frozen reference model. It captures judgments that are easy for experts to make and hard to write down as a single correct answer, such as which of two analyses is more rigorous.

DPO is offline: the pairs are fixed before training, so the model never explores. Its quality ceiling is the quality and coverage of the preference data. Noisy labels, position bias in judging, and pairs sampled from a different model all degrade it, and aggressive training can collapse style or drift far from the reference.

DPO needs an SFT-quality starting point. Preference pairs teach direction, not competence, and a model that cannot produce a reasonable response yet has nothing for the preference gradient to sharpen.

GRPO: learn from sampled behavior and reward

Group Relative Policy Optimization is an online reinforcement-learning method. For each prompt or task, the current policy samples a group of responses, a reward scores each one, and the group's mean reward becomes the baseline: responses above it are reinforced, responses below it are suppressed. Unlike Proximal Policy Optimization (PPO), no separate value model is trained, which removes a large share of the systems complexity.

GRPO is the method of choice when success is checkable: a test suite passes, an answer matches, a verified end state is reached. That setting is called reinforcement learning from verifiable rewards (RLVR), and it is the core stage for tool use and long-horizon agent capabilities, because the model learns from the consequences of its own actions rather than from a static dataset.

The cost is that everything moves online. You need an environment or task suite the policy can act in, a reward that survives being optimized against, rollout infrastructure that keeps sampling and training in sync, and diagnostics for a training loop with far more failure modes than SFT or DPO.

What each method needs before training starts

SFT prerequisites

• Demonstrations of the target behavior at expert quality, typically hundreds to tens of thousands depending on task diversity
• Coverage of the input distribution, including edge cases and failure-recovery examples
• A held-out set from a separate source, never a random split of the same generation pipeline

DPO prerequisites

• A competent SFT-level policy to sample candidate responses from
• Judges (human or carefully validated AI) whose agreement rate you have measured
• Preference pairs covering the distinctions that matter, including hard negatives, not just easy wins

GRPO prerequisites

• Tasks where the policy already succeeds sometimes: with a near-zero success rate every group is uniformly bad and the group-relative advantage is zero
• A reward or verifier you trust under optimization pressure
• An executable environment with reset and enough throughput for group sampling
• Rollout, serving, and training infrastructure that stay weight-synchronized

The offline-versus-online distinction matters operationally. SFT and DPO run on fixed datasets and finish on a schedule. GRPO generates its own data continuously, so cost and duration depend on sampling throughput and on how quickly the policy improves.

Infrastructure and cost differences

SFT is a standard training job: one model in memory, a dataloader, and an optimizer. Parameter-efficient variants such as Low-Rank Adaptation (LoRA) run on modest hardware. DPO adds a frozen reference model for likelihood comparisons, roughly doubling memory but changing little else.

GRPO couples a training job to an inference fleet. Rollout workers sample from the current policy at high throughput, rewards are computed per response, trajectories flow back to the trainer, and updated weights flow forward to the samplers. Most GRPO engineering effort goes into this loop, not into the loss function, and most GRPO cost is sampling compute rather than gradient compute.

Iteration speed follows the same gradient. An SFT experiment can be rerun in hours. A GRPO experiment couples environment behavior, reward correctness, and training dynamics, so each debugging cycle is slower and observability is worth building up front.

Common failure modes

A deeper treatment of the reinforcement-learning failure modes, including KL divergence behavior, truncation effects, and stale-weight bugs, is in the companion guide on why GRPO training fails.

When the methods should be staged

The methods compose. A typical complete program runs SFT to establish competence and format, DPO to sharpen judgment where experts can rank outputs, and GRPO where an environment and reward exist. Each stage starts from the previous stage's checkpoint and each stage's evaluation gates the next.

Stopping early is a legitimate outcome. If held-out evaluation shows SFT alone meets the capability bar, the preference and reinforcement-learning stages are cost without benefit. The decision to continue should come from measured gaps, such as the model knowing the format but choosing wrong actions, not from a desire to use the fancier method.

Decision matrix

Three example programs

Structured domain analysis

A model must produce schema-valid analyses of domain documents. Experts can write gold outputs and rank near-misses. Program: SFT on expert demonstrations, then DPO on expert-ranked pairs mined from the SFT model's own outputs. GRPO is unnecessary because nothing is interactive.

Tool-using support agent

An agent must call internal tools, recover from errors, and complete multi-step requests. Success is checkable against final state. Program: SFT on curated tool-use traces for format and basic competence, then GRPO in a sandboxed environment with a verifier on end state plus penalties for invalid calls. DPO is optional as a bridge if judged trajectory pairs are cheap.

Code assistant for a proprietary framework

A model must write code against an internal framework where tests exist. Program: continued pretraining on the framework corpus, SFT on worked examples, then GRPO with the test suite as the verifier (the RLVR setting). The test suite must be hardened first, because the policy will find any gap between passing tests and correct code.

Method selection checklist

1. Write down the capability as behavior you can evaluate, with a held-out set from an independent source.
2. Ask whether experts can produce correct outputs directly. If yes, plan SFT and collect demonstrations.
3. Ask whether experts can reliably rank outputs they cannot write. Measure judge agreement before trusting it. If agreement is strong, plan a DPO stage.
4. Ask whether success can be checked programmatically or by executing the result. If yes, GRPO with verifiable rewards is available, and worth it for interactive capabilities.
5. Confirm the policy has nonzero success on the reinforcement-learning tasks before committing to GRPO, or add an SFT warm start until it does.
6. Budget the online stage by sampling cost, not gradient cost.
7. Define stop conditions per stage: what measured gap justifies continuing to the next method.