Rewards and verifiers
In post-training, the reward is the specification. LLMix engineers the learning signal: deterministic verifiers, outcome and process rewards, human and AI feedback, learned reward models, calibration, and reward-hacking analysis.
An optimizer does not pursue your intent. It pursues the number your pipeline produces, with increasing competence, for millions of steps. Every gap between that number and the behavior you actually want will be found and exploited. Reward engineering is therefore specification work: deciding what success means precisely enough that maximizing it is the same as achieving it.
Exact checks on outcomes: tests pass, the answer matches, the final state satisfies constraints, the schema validates. Objective, cheap at scale, and the hardest signal to hack when checks run against world state rather than the model's claims. This is the signal behind reinforcement learning from verifiable rewards (RLVR).
Score the end result of an episode, however reached. Robust to strategy diversity and the default choice when only the destination matters. Their weakness is sparsity on long tasks: one number for many decisions.
Score intermediate steps: valid tool calls, verified milestones, reasoning checks. They densify signal on long horizons and localize credit, at the risk of teaching the process instead of the result when the steps rewarded are not actually necessary.
Experts compare outputs, and the comparisons train the policy directly (through Direct Preference Optimization) or train a reward model (the reinforcement learning from human feedback pipeline, RLHF). The gold source for judgment no program can check, priced accordingly.
A judge model applies rubrics or principles at scale. Cheap and consistent relative to human panels, but a judge is a model with biases (position, length, style), so LLMix treats judges as instruments to validate against human agreement before their scores move weights.
A trained scorer generalizes preference data to unseen outputs and enables online optimization. It is also the component the policy optimizes against most directly, so its blind spots become training targets. Reward models need holdout validation, drift monitoring, and periodic refresh with new comparisons.
Can success be checked exactly (state, tests, match)?
├── yes → deterministic / programmatic verifier
│ (add process rewards if episodes are long)
└── no
├── Can experts rank outcomes reliably (measured agreement)?
│ ├── yes → preference data → DPO, or a reward model for online RL
│ └── no → can a rubric make an AI judge match experts
│ on a validation set?
│ ├── yes → calibrated AI feedback (RLAIF)
│ └── no → the capability is not yet specifiable:
│ redesign the task or the objective
└── Can intermediate progress be verified even if the
outcome cannot?
├── yes → process rewards over verified milestones
└── no → environment redesign or a weaker, checkable objectiveFor a tool-using support agent, a production-shaped reward might decompose like this. The exact weights are program decisions made against measured behavior, and every component is logged separately so exploitation of any one of them is visible.
reward = 1.00 * task_success # verifier on final ticket state
+ 0.10 * milestone_progress # verified sub-goals reached
- 0.05 * invalid_tool_calls # schema-invalid or denied calls
- 0.01 * steps_used # cost pressure, capped
- 1.00 * policy_violation # hard penalty, ends episode
# Caveats engineered alongside the weights:
# - task_success checks world state, never the agent's claim
# - milestone_progress is capped so it cannot outweigh success
# - step cost is small enough that recovery is still worth itBefore any judge (human or AI) supplies training signal, LLMix measures it: agreement rates between judges, consistency across orderings and phrasings, and correlation with ground truth where verifiable cases exist. Disagreement is data. Systematic splits usually mean the rubric is underspecified, and cases where judges cannot agree are removed from training rather than averaged into noise. No LLM judge is treated as objective.
A binary reward on a 40-step task is one bit of signal per episode. Where milestones are verifiable, partial credit densifies learning without changing the goal. Real programs also balance several objectives at once (success, cost, policy compliance, format validity), and the composition is engineered explicitly: named components with logged values and deliberate weights, revisited as training exposes which components the policy trades against each other.
When a measure becomes the target, it stops measuring. Classic exploits include passing the checker without doing the task, style that flatters the judge, length inflation, and confident text over verified state. LLMix red-teams reward systems before training (trying to construct high-scoring failures by hand) and monitors for hacking signatures during it, such as training reward rising while held-out quality stalls or KL divergence spikes toward exploitative modes.
The reward used for training is never reused as the acceptance metric. Release evaluation runs on held-out tasks, with independent checks and where warranted different judges, so that a policy that overfits its training signal still has to prove the capability somewhere it cannot have gamed.
If success is exactly checkable, a verifier: it is cheaper, objective, and harder to hack. Reward models earn their complexity where quality is real but not computable. Many programs use both, a verifier for the objective core and a model or judge for qualities the verifier cannot see.
Sometimes. Verifiable domains (code with tests, checkable end states) can run on programmatic signal alone. Judgment-heavy capabilities need some human signal, if only to validate the AI judge that then scales the labeling.
Before training, by red-teaming the reward with hand-built exploits. During training, by monitoring held-out quality against training reward, per-component reward logs, KL divergence, and length and format statistics. Divergence between the training signal and held-out capability is the alarm.
RL environments for LLM agents
Where the signal is exercised: executable tasks, tools, transitions, and rollout integration.
Why GRPO training fails
The failure-diagnosis guide: most broken runs trace back to the reward pipeline.
Bespoke LLM post-training
The complete program that reward engineering plugs into, from data to validated release.