How to train your local LLM

The moment someone says “we need to train our own LLM,” you already have two problems: the actual technical work, and the fact that most of the time they don’t need training at all. Retrieval fixes recall. Prompting fixes format. Fine-tuning fixes behaviour that a well-designed prompt still can’t reach, and pre-training fixes almost nothing you’ll ever have the budget to fix.

This guide is the long version of that conversation. It walks through what it actually takes to specialise a local LLM to the point of shipping, in the order the decisions arrive. It is opinionated because the alternative is drowning in options: a model registry search returns tens of thousands of checkpoints and half a dozen “trainer” repos, and none of that tells you which knob to turn or in what order.

The thesis: pick the smallest technique that closes the actual behaviour gap, treat the dataset like it’s the product, and gate the release on behavioural evals — not on loss curves.

Everything below is a variation on those three sentences.

1. Decide whether you actually need to train

Before touching a GPU, walk the ladder in order:

Prompt engineering — clearer instructions, few-shot examples, output schemas.
Retrieval-augmented generation (RAG) — grounding on your data.
Structured generation — constrained decoding, grammars, tool calling.
Supervised fine-tuning (SFT) — teaching format, style, or a narrow task.
Preference optimisation — refining tone, calibration, refusal boundaries.
Continued pre-training — new domain vocabulary or language.
Full pre-training — you have eight figures and a research team.

You move up a rung only when the rung below has demonstrably failed against a written eval suite. “It didn’t feel right” is not a signal; a 30% pass rate on a 50-prompt test set is. Most projects that end up in fine-tuning territory should have stopped at rung 2 or 3 and didn’t, because nobody built the eval.

Signs you actually need SFT:

The output format is non-standard and can’t be reliably prompted into (a proprietary DSL, an internal ticketing shape, a rare structured artefact).
The domain vocabulary is genuinely out-of-distribution (legal citations, clinical shorthand, niche engineering acronyms).
Latency or cost forbids a large frontier model and you need a small model to behave like a specialist.
Behaviour must be consistent under adversarial prompting in ways prompt guards can’t guarantee.

If none of the above is true, close this tab. RAG plus a decent instruct model will beat a mediocre fine-tune every time.

2. Pick the smallest technique that fits

The techniques form a rough hierarchy of cost and blast radius. Start at the top and only move down when the tier above provably fails your evals.

QLoRA (SFT) — LoRA adapters on a 4-bit base. Changes ~0.1–1% of parameters. The default for format, style, or narrow-task specialisation on ≤10k well-curated examples. Fits comfortably in 12–16 GB of VRAM for a 7B base.
LoRA (SFT) — same as above, base in bf16 or fp16. Reach for it if QLoRA quality is measurably worse and you have the VRAM headroom.
DPO / KTO / ORPO — preference calibration on top of SFT (or fused with it, in ORPO’s case). Only when SFT is factually right but stylistically off.
Full-parameter SFT — moves every weight. Use only when low-rank adaptation is provably the bottleneck, and be ready for FSDP or ZeRO-3 across multiple GPUs.
Continued pre-training — unlabelled corpus, base checkpoint, no instruction data. For genuine new-language or deep-domain-vocabulary needs.

Rule of thumb: start with QLoRA. If your target is ≤13B and your dataset is ≤10k examples, QLoRA reaches ~90% of the achievable ceiling for ~10% of the cost. Everything above QLoRA is buying diminishing returns unless you have hard evidence you need them.

The memory math worth internalising for planning:

VRAM (SFT, rough) ≈ P × bytes_per_param × k

Where P is parameter count, bytes_per_param is 2 for bf16, 1 for int8, 0.5 for int4, and k is a multiplier for optimiser states, activations, and gradients: ~4× for full-parameter Adam, ~1.2× for QLoRA. A 7B model in QLoRA fits on a single 16 GB card. The same 7B under full-parameter bf16 Adam wants 60+ GB and multi-GPU sharding.

3. Choose a base model that meets you halfway

Base-model choice is a third of the final quality. Pick the one whose native behaviour is closest to what you want, in the smallest size that can plausibly hold the task.

Three axes matter.

Licence. Read it before you touch it. Llama models ship under a community licence with clauses that trip up commercial use at scale (MAU thresholds, attribution rules, acceptable-use policy). Qwen and much of Mistral’s open-weight line are Apache-2.0 — friendlier, but check the specific model card because each release sets its own terms and there are proprietary Mistral products alongside the open ones. Gemma has its own terms. If you’re building a product, get this reviewed by someone who owns the risk.

Instruct vs base. Start from an instruct-tuned checkpoint unless you are retraining behaviour from scratch. The instruction alignment is a huge amount of free work you’d otherwise redo. The exception is continued pre-training on unlabelled domain text, where you want the base checkpoint so you don’t wash out the instruction tuning.

Size. For most specialisation, 7–14B is the sweet spot. Below 3B you start hitting capability floors you can’t train around; above 30B the serving cost usually outweighs the quality gain unless the task is genuinely frontier-hard.

4. Curate the dataset like it’s the product

This is where most fine-tunes fail, silently, and it takes weeks to notice.

The single most important thing to internalise: 500 immaculate examples beat 50,000 mediocre ones. LLMs learn from the modal signal in your data. If 10% of your training examples have a subtly wrong format or a bad tone, the model will reproduce that error faithfully. Bad examples don’t average out — they compound.

A workable dataset process:

Write the target behaviour as an eval before you write any training data. Twenty to fifty held-out prompts with expected properties (not necessarily exact answers). This is the contract.
Draft twenty to fifty gold examples by hand. These set the voice nothing else will replicate.
Scale up by generation and filtering, not by scraping. Ask a stronger model to draft candidates against your gold examples. Reject aggressively. The default should be to throw out 70% of what comes back.
Deduplicate. Exact and near-duplicate removal (MinHash, ~0.85 threshold) at both prompt and completion level. Duplicates cause memorisation and inflate your validation scores.
Scrub PII, secrets, and copyright. Not optional. Do it before the data ever touches a training loop, and keep the scrubbing script in the repo.
Split thoughtfully. Train / validation / held-out behavioural test. The held-out set never sees a training loop, ever — it’s the release gate.

Format matters as much as content. Every example must use the exact chat template the base model was trained with. Mixing templates, forgetting the BOS token, or subtly wrong role tags is the number-one silent-failure cause of fine-tunes that look like they trained but underperform the base model.

The check I run before every training run:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained(BASE_MODEL)

# Roundtrip a sample through the chat template
sample = training_data[0]
rendered = tok.apply_chat_template(
    sample["messages"],
    tokenize=False,
    add_generation_prompt=False,
)
print(rendered)                        # eyeball it

# The template already embeds special tokens (BOS, role tags), so don't let
# the tokenizer add its own — that's how you get a duplicated BOS.
retok = tok(rendered, add_special_tokens=False).input_ids
print(tok.decode(retok))               # should round-trip to `rendered`

If the render doesn’t look right — missing role tags, a doubled BOS, the wrong template entirely — stop and fix it before anything else. You’ll save yourself a wasted training run.

5. Set up the environment for reproducibility

The frameworks worth knowing, all mature and actively maintained:

Axolotl — YAML-driven configs, opinionated defaults, wide model coverage. My default for SFT and preference tuning when I don’t have a strong reason otherwise.
Unsloth — the fastest QLoRA path on a single consumer GPU. Custom Triton kernels, ~2× speedups typical, narrower model coverage than Axolotl.
TorchTune — PyTorch-native, minimal abstraction between you and the training loop. Pick this when you need to understand or modify what’s happening.
TRL — the reference preference optimisation library (DPO, KTO, ORPO, GRPO). Often used underneath Axolotl.

Reproducibility non-negotiables:

Pin every dependency (uv pip freeze > requirements.lock, or the equivalent in your toolchain).
Commit the training config as a file. Do not run from a notebook.
Log the git SHA, the dataset SHA-256, and the base-model revision into the run metadata.
Set every seed you can (torch, numpy, random, transformers, dataloader worker seed).
Save the exact tokenizer alongside the weights. Do not assume the base model’s tokenizer will still be identical six months from now.

A “run” is not a set of hyperparameters; it’s a triple (config, dataset, base) that must be reconstructable from a single commit in six months.

6. Configure the run — the boring numbers that matter

Sensible defaults for a first QLoRA SFT of a 7–8B model on ~5k examples:

# axolotl-style, illustrative
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
load_in_4bit: true
adapter: qlora

lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj

sequence_len: 4096
sample_packing: true
train_on_inputs: false          # loss masked to completions only

num_epochs: 2
micro_batch_size: 2
gradient_accumulation_steps: 8  # effective batch 16
learning_rate: 2.0e-4
lr_scheduler: cosine
warmup_ratio: 0.03
weight_decay: 0.0

bf16: true
gradient_checkpointing: true
flash_attention: true

Three of these matter more than the rest and are the ones people get wrong.

train_on_inputs: false (loss masking). Train only on the assistant completion, not on the user turn. If you leave the loss active on the input, the model learns to reproduce user prompts back at you — a weirdly common silent failure that only surfaces at inference.

lora_r and lora_alpha. Rank 16 with alpha 32 (a 1:2 ratio) is a sane starting point. Higher rank means more capacity and more overfitting risk. If the model looks under-trained, raise the rank before raising the epochs.

Epochs. For high-quality small datasets: 2–3. For larger noisier ones:

More than 3 epochs is almost always overfitting, no matter what the training curve tells you. Trust the held-out eval, not the loss.

Note the learning rate: 1–3e-4 for LoRA/QLoRA is genuinely higher than the 1–5e-5 you’d use for full fine-tuning, because only a small parameter subset is moving. Do not copy an LR from a full fine-tune paper into a LoRA config.

7. Watch the loss, but trust the evals

Training loss going down is necessary and almost meaningless on its own. A model whose loss dropped cleanly can still be worse than the base at every task you actually care about — because it overfitted the dataset’s artefacts, or drifted away from behaviours you needed but never trained.

The evaluation gates worth running, in order:

Behavioural eval on the held-out set. Pass/fail against the properties you defined in step 4. Below 90% pass on properties you consider table-stakes: don’t ship.
Regression suite against the base model. A separate set of prompts unrelated to your task. If the fine-tune regresses more than a couple of points on general instruction-following, you’ve caused catastrophic forgetting and need to mix in general-purpose data or drop the LoRA rank.
LLM-as-judge, cautiously. Useful for pairwise comparisons at scale; dangerous as a single-model absolute score. If you use it, pick a different family than the one you’re fine-tuning, and calibrate on a small human-labelled subset first.
A human read-through. Fifty completions, eyeballed by someone who knows the domain. Non-negotiable before any release. LLM judges will miss subtle-but-obvious problems every time.

Loss curves diagnose training bugs: loss going up, diverging, plateauing too early. They do not measure quality. Never promote a checkpoint on loss alone.

8. Reach for preference optimisation only when SFT plateaus

If your SFT model is factually right but stylistically off — too verbose, wrong register, mis-calibrated refusals, formatting drift — that’s the shape preference optimisation is for.

DPO is the well-worn default, and often good enough. Requires paired chosen/rejected data.
ORPO folds SFT and preference learning into a single stage; worth trying if you’re starting from a base model rather than an SFT checkpoint.
KTO is useful when you have thumbs-up/thumbs-down data rather than pairwise preferences.

What DPO/KTO/ORPO cannot fix:

Factual gaps. The model doesn’t know what it doesn’t know; preference learning can’t teach it. That’s still RAG or SFT territory.
Format failures. If the model can’t produce the schema, preference learning on top won’t rescue it. Go back to SFT.
Capability floors. A 3B model that can’t reason multi-step won’t reason multi-step after DPO. Preference tuning polishes; it does not enlarge.

The preference dataset needs to isolate the axis you’re tuning. Chosen and rejected completions should differ only on the property you care about; otherwise the model will learn incidental shortcuts (longer answers are better, markdown is better, sycophancy is better) that you’ll then have to untrain.

9. Merge, quantize, ship

For LoRA/QLoRA the training output is an adapter. You have two shipping paths:

Serve base plus adapter separately. Both vLLM and TGI support LoRA hot-swap. Useful when you have multiple adapters over one base model, or when you need to A/B specific adapter versions in production.
Merge and quantize. Fold the adapter into the base weights, then quantize the merged model for your target runtime. Simpler to serve; costlier to iterate.

The formats worth knowing:

GGUF — for llama.cpp and its ecosystem (Ollama, LM Studio, others). Best for CPU and consumer-GPU inference. Quantization options run from Q2_K to Q8_0; Q4_K_M and Q5_K_M are the practical quality-per-byte sweet spots for a 7–13B model.
AWQ and GPTQ — for GPU inference under vLLM, TGI, or similar. Faster than GGUF on GPU, slightly better quality per bit than the older bitsandbytes quantization.
BF16 / FP16 — no quantization. For evaluation and for further training.

Always re-run your behavioural eval against the quantized model. Quantization is lossy; a checkpoint that passed at bf16 can fail at Q4. That gate is the difference between “we shipped a specialised model” and “we shipped a broken specialised model that nobody caught before Monday.”

10. Close the loop

A model that ships is not finished; it’s the first observation in a series. The mechanical parts of closing the loop:

Log every completion in production, with structured metadata: input, output, latency, downstream user action if any. Store enough to reconstruct the exact prompt — system message, template version, model revision.
Sample and label. Weekly, pull a stratified sample and have a human rate it against your rubric. The samples that fail become next iteration’s training data.
Watch for drift. Two failure modes matter: input drift (users start asking things you didn’t train for) and behaviour drift (the model changes because something upstream — tokenizer, template, quantization runtime — changed). A stable weekly behavioural eval catches both.
Retrain on a schedule, not on vibes. Monthly or quarterly cadence, with a documented promotion gate. Ad-hoc retraining is how you end up unable to reproduce your production model.

Why this holds up

The ladder keeps you honest. Most “we need to fine-tune” problems dissolve at prompting or RAG. The projects that survive to fine-tuning arrive with a real behaviour gap and a real eval, which is exactly what you need to succeed.
The dataset is the product. Frameworks, hyperparameters, and base models are commodities. A well-curated 2k-example dataset with a matching eval is the moat — and the thing nobody else has.
The evals are the gate. Loss going down is not shipping. Behavioural pass rate against a held-out suite is shipping. That gate is the difference between a model you can defend in a release meeting and one you’re quietly hoping nobody probes.
Small techniques compound. QLoRA plus a tight dataset plus rigorous evals will outperform full-parameter fine-tuning with a mediocre dataset and loss-curve vibes, at 1% of the compute. The frontier here is discipline, not GPUs.

Training a local LLM well is unglamorous. It looks less like a research paper and more like release management: contracts, gates, reproducibility, and a lot of throwing out data that wasn’t good enough. That’s the point. The model is the boring, predictable output of a process that took the interesting work seriously.