LLM fine-tuning
LLMix builds supervised adaptation as an engineering discipline: data-mixture design, continued pretraining where justified, supervised fine-tuning, LoRA and QLoRA, preference optimization, distillation, and held-out regression evaluation.
Fine-tuning is a model-development stage, not a complete product. LLMix combines data design, training, evaluation, and inference preparation so that the resulting behavior is reproducible and testable.
Method choice is conditional on the signal available and the deployment constraints, not on a universal ranking. These are the stages LLMix selects from and combines.
Next-token training on a domain corpus before any instruction data. Justified when the domain's language and structure are far from the base model's distribution, and skipped when they are not, because it is the most compute-hungry stage.
Training on demonstrations of the target behavior. The workhorse stage: it sets format, competence, and tone, and establishes the warm start that preference or reinforcement-learning stages build on.
Low-Rank Adaptation trains small adapter matrices instead of all weights, and QLoRA does so over a quantized base. They cut memory and cost sharply and suit most single-capability adaptations. Full-parameter training remains justified when the change is deep or the adapter ceiling shows in evaluation.
Direct Preference Optimization (DPO) and related objectives sharpen judgment where experts can rank outputs but not write ideal ones. A natural second stage once SFT competence exists.
Training a smaller model on the outputs or scores of a stronger teacher, with filtering so mistakes are not inherited. The standard route to serving-cost reduction for a scoped capability.
| Method | Input signal | Primary use | Compute profile | Main failure modes | Typical output |
|---|---|---|---|---|---|
| CPT / DAPT | Raw domain corpus | Close a large domain gap before instruction data | Highest: full training over a corpus | Forgetting general ability, wasted spend when the gap was small | Domain-adapted base checkpoint |
| Full SFT | Demonstrations | Deep behavior change, new formats and policies | High: all parameters | Overfitting narrow data, regression without mixture balance | Full tuned checkpoint |
| LoRA | Demonstrations | Scoped capability at low cost | Low: adapter weights only | Capacity ceiling on deep changes | Adapter weights (mergeable) |
| QLoRA | Demonstrations | LoRA-scale training on constrained hardware | Lowest: quantized base + adapters | Quantization interactions, same ceiling as LoRA | Adapter weights over quantized base |
| DPO | Chosen vs rejected pairs | Sharpen judgment after SFT | Moderate: policy + frozen reference | Judge bias, style collapse | Preference-tuned checkpoint |
| Distillation | Teacher outputs or scores | Smaller model for a scoped task | Moderate: student-sized training | Inheriting teacher errors, coverage gaps | Compact student model |
Selection is conditional: no row dominates the others, and staged combinations (for example LoRA SFT followed by DPO) are the common case. The comparison guide covers the staging logic in depth: SFT vs DPO vs GRPO.
Most fine-tuning outcomes are decided by the dataset before the first training step. LLMix treats data work as the primary engineering surface.
A concrete example of the quality bar: for a structured-analysis capability, near-duplicate demonstrations with conflicting labels are worse than fewer examples, because the loss averages the conflict into confident noise. LLMix adjudicates conflicts before training rather than hoping scale washes them out.
Every program defines held-out capability evaluation and a regression suite over general behavior before training starts. A tuned model ships only when the capability gain is shown on held-out data and the regression suite shows no unacceptable loss elsewhere. Training-set performance is never the acceptance criterion.
A checkpoint that works in the trainer and fails in production is a common and avoidable outcome. Programs include the serving side: chat-template and tokenizer consistency between training and inference, adapter merging or deployment as adapters, quantization where scoped, and conversion for the target serving stack with a final evaluation pass on the served artifact itself.
Some capabilities need more than supervised signal: tool use under consequences, long-horizon behavior, or objectives only checkable by execution. Programs then extend from SFT into preference optimization and online reinforcement learning inside executable environments. That path is the complete post-training program, and it reuses everything the fine-tuning stage produced.
Explore the complete post-training programThere is no universal number. Requirements scale with task diversity, not task difficulty: hundreds of high-quality demonstrations can carry a narrow format task, while broad capabilities need thousands plus preference data. LLMix runs pilot learning curves early so the data budget is measured, not guessed.
Start from constraints, not preference. LoRA and QLoRA cover most scoped adaptations at a fraction of the cost, and evaluation tells you when the adapter ceiling is real. Full-parameter training earns its cost for deep behavioral change or heavy domain adaptation.
Yes, where the provider exposes tuning (and its variants) for that model. The trade-offs are provider lock-in and limited method control: open-weight models allow the full toolkit including preference optimization and later reinforcement learning.
The client. Checkpoints or adapters, datasets built for the program, configurations, and evaluation assets are all delivered, subject only to the base model's own license terms.
Bespoke LLM post-training
The complete program when supervised signal alone is not enough: preferences, rewards, and online RL.
Reward modeling and verifier design
The learning signal that extends fine-tuning into preference and reinforcement-learning stages.
How much data does fine-tuning need
The method comparison guide covers data requirements per stage and how to measure them with pilots.