notes/brainlift.md

# Brainlift: Training and Optimizing Small Language Models to Reliably Perform Specific Behaviors

## TL;DR
- **For instilling one narrow, falsifiable behavior into a 0.6B–4B model, the dataset is the deliverable:** LIMA (1,000 examples), s1 (1,000 traces), and QLoRA's own ablations converge on quality-and-diversity over quantity, and QLoRA makes single-GPU tuning routine. Capability is mostly *elicited* from a base model that already latently has it — not installed from scratch.
- **QLoRA is the right default precisely because it "forgets less"** (Biderman et al.), and reliability comes from an engineering stack — distilling a teacher's *process*, constrained decoding for output form, and an adversarial base-vs-tuned eval — not from a bigger model or a cleverer prompt.
- **Build the evaluation before you generate a single training row, and make it adversarial**, because small (and large) LMs fail in predictable, testable ways: compositional reasoning collapse (Faith and Fate), irrelevant-clause sensitivity (GSM-Symbolic, up to 65% drop), the reversal curse, lost-in-the-middle, and confident hallucination.

## Purpose
A portable research knowledge base for instilling ONE specific, falsifiable behavior into a small open base model (0.6B–4B) via supervised fine-tuning (QLoRA), using distillation from a frontier teacher to build the dataset, and evaluating with LLM-as-judge plus a base-vs-tuned comparison.

**In scope:** data-centric generation/curation and distillation; PEFT (LoRA/QLoRA); SFT mechanics; preference optimization (DPO and variants); compression/quantization; small-model landscape; evaluation and LLM-as-judge; reliability and structured output; failure modes and limits of (small) LMs; practitioner playbooks.

**Out of scope:** pretraining from scratch; large-scale RLHF infrastructure; multimodal training; serving/infra beyond quantization formats; agent orchestration.

## Key Findings
1. **Small-data alignment works when the capability is latent.** Two independent 1,000-example results (LIMA for general alignment, s1 for reasoning) plus QLoRA's finding that data quality beats size establish that a narrow behavior can be reached cheaply — but LoRA-Learns-Less shows this breaks down for genuinely new hard skills.
2. **Distill the process, not the style.** Orca and DeepSeek-R1 show reasoning traces transfer real capability into small models; Orca explicitly warns that style-only imitation inflates apparent quality on shallow evals.
3. **PEFT trades peak capacity for forgetting-resistance**, which is a feature for narrow-behavior work. Full fine-tuning still wins on hard new skills.
4. **LLM-as-judge is usable (>80% human agreement) but biased** (position, verbosity, self-preference); design evals around the biases.
5. **Reliability = data + constrained decoding + calibration eval.** Neither fine-tuning nor grammar-constrained decoding is individually sufficient.
6. **Small models fail in named, reproducible ways** — build adversarial evals that target exactly those failure modes.

## DOK 4 — Spiky Points of View

**1. For a narrow behavior, ~1,000–2,000 excellent, diverse examples is enough; past that, more data is usually wasted effort.**

- _Why:_ pretraining already installed the capability, so SFT only has to pin a consistent output policy over what the model already knows (Superficial Alignment Hypothesis). Rows past the first ~1k mostly re-teach that and add overfitting risk.
- What still helps is coverage of the behavior's input variety, not raw count.
- Two independent replications on very different tasks (LIMA on alignment, s1 on competition math) landed on the same ~1,000 figure, so it isn't a single-domain fluke.
- _Evidence:_ LIMA (1,000 examples rival RLHF models trained on 52× more data), s1 (7 vs 394 GPU-hours for near-identical accuracy), QLoRA's quality>size ablation.
- _When wrong:_ if the base genuinely lacks the capability (novel reasoning, unfamiliar code), LoRA-Learns-Less shows you need more data AND higher rank or full FT.
- _Abandon when:_ the base-vs-tuned delta plateaus below target despite clean, diverse data; that plateau means the capability is absent, not under-sampled.

**2. Use QLoRA by default and treat "forgets less" as a feature, not a compromise.**

- _Why:_ full FT rewrites high-rank directions across all weights and injects "intruder dimensions" that overwrite unrelated pretrained abilities; LoRA's low-rank cap keeps the update from traveling that far.
- For one-behavior work you don't want to disturb anything else, so LoRA's capacity ceiling is the guardrail you want, not a weakness.
- The alternative fails concretely: Raschka saw a model "unlearn arithmetic" from narrow SFT; Luo et al. show forgetting worsens with scale under continual full tuning.
- QLoRA is single-GPU with no measured quality loss on tested tasks, so the default costs nothing.
- _Evidence:_ Biderman et al. (LoRA forgets less, retains out-of-domain capability), intruder-dimensions paper (the mechanism), QLoRA (single-GPU, no perf loss), Raschka.
- _Counter:_ full FT wins peak in-domain accuracy on genuinely hard new skills.
- _Wrong if:_ your ONLY metric is target-task accuracy and base-capability retention is irrelevant; then use full FT or high-rank (r≈256) LoRA.

**3. Build the eval before you generate a single training row, make it adversarial, and trust base-vs-tuned deltas over any single judge score.**

- Without a pre-committed eval, "we fine-tuned a model" is unfalsifiable; the eval is what turns a vibe into a measurement.
- Make it adversarial: small models fail in named ways (irrelevant clauses, reversed relations, mid-context facts) a happy-path test never triggers, so a fine-tune that clears only easy cases has learned the surface.
- Read the delta, not the absolute score: judge biases (position, verbosity, self-preference) distort raw numbers but largely cancel when one judge compares two outputs with randomized order.
- Prefer binary pass/fail over 1–5: it forces a crisp definition and is far more stable across judge runs.
- _Evidence:_ Husain (evals first, 60–80% of effort; binary > 1–5), Zheng et al. (quantified biases; >80% human agreement), GSM-Symbolic and Reversal Curse (fragilities to target).
- _Spiky corollary:_ a fine-tune shipped with no robustness eval should not be trusted, however good the demos.
- _Wrong if:_ the behavior is code-verifiable by exact-match/regex (skip the judge), or you've validated high judge-human agreement on your task.

**4. Reliability is an engineering stack (data + constrained decoding + a calibration/abstention eval), not a bigger model or a better prompt.**

- Three levers fix three failure sources: fine-tuning raises the odds of correct content but can't guarantee valid form; constrained decoding guarantees form but can push the model off-path and hurt content; a calibration eval surfaces the residual confident-guess errors that only rewarding "I don't know" suppresses.
- The deeper point: reliability is a tail property (the 1-in-50 malformed output breaks the pipeline), and scaling or prompting shifts the average while leaving the tail. Engineer the tail.
- _Evidence:_ constrained-decoding literature (Outlines/XGrammar guarantee form), Why-Models-Hallucinate (residual errors are calibration failures), plus fine-tuning raising correct-behavior probability.
- _Counter:_ constrained decoding adds ~2–5× overhead and can degrade content; over-constraining hides genuine uncertainty.
- _Wrong if:_ the base is already schema-reliable and latency can't absorb decoding overhead.

## DOK 3 — Insights

**1. The dataset is the deliverable; capability is mostly elicited, not installed.**

- LIMA (1,000 examples), s1 (1,000 traces; 394 vs 7 GPU-hours), and QLoRA's quality>size finding converge: a small, diverse, high-quality set beats a large noisy one.
- _Why:_ pretraining did the heavy lifting, so SFT mainly selects a consistent output policy over existing knowledge; beyond a point you pay to re-teach what the model knows, and coverage matters more than count.
- _Tension:_ LoRA-Learns-Less shows that for genuinely NEW capabilities (code, math) low-rank small-data tuning underperforms full FT—elicitation works when the capability is latent, not when it's absent.

**2. Distill the process, not the style.**

- Orca and DeepSeek-R1 show that transferring reasoning traces/explanations (not just final answers) is what moves a real capability into a small model; Orca warns that style-only imitation inflates apparent quality.
- _Why:_ final-answer-only data teaches outputs without the intermediate computation, so it copies tone and collapses when the problem shifts; traces hand over the steps the student can internalize. It's also why style-rewarding evals overrate imitation-trained models.
- _Tension:_ DeepSeek-R1 distillation still leaves persistent gaps for ≤7B students; process distillation narrows but doesn't close the teacher-student gap.

**3. LoRA/QLoRA is the right default for behavior instillation precisely because it forgets less.**

- Biderman et al. and the intruder-dimension paper show LoRA sacrifices peak capacity but preserves base competence—ideal when you want ONE behavior without collateral regression.
- _Why:_ the low-rank cap limits how far the update moves the weights, so unrelated abilities survive; full FT has no such cap. For single-behavior work, preserving everything else is the goal, so the capacity limit is the property you want.
- Raschka's "unlearned arithmetic" and Luo et al.'s forgetting-scales-with-size results show full FT/narrow data risks broad capability loss.
- _Steelman for full FT:_ if the behavior is a hard new skill, full FT (or high-rank LoRA, α=2r, all modules) wins on the target metric.

**4. Evaluation must precede and drive training.**

- Hamel Husain (evals first, 60–80% of effort), LIMA (perplexity ≠ quality), and practitioner consensus all treat the eval harness as the real deliverable.
- _Why:_ loss and perplexity don't track the behavior you care about, so without a behavioral eval you optimize a proxy; the eval is also the only signal for whether the fix is more data, different data, or a different method.
- A base-vs-tuned LLM-judge comparison on a held-out set is the minimum viable measurement.

**5. LLM-as-judge is usable but biased; design around it.**

- Zheng et al. quantify position, verbosity, and self-preference biases (position can swing win-rate 10–15 points; length bias ~+17%).
- _Why it's still workable:_ the biases are systematic, not random, so they distort absolute scores but largely cancel in A/B comparisons under a fixed judge with randomized order.
- Mitigations: randomize order, control for length, don't judge with the same model family, prefer binary pass/fail (Husain).
- _Tension:_ judges agree with humans >80%, so the biases are manageable, not disqualifying.

**6. Reliability comes from data + decoding, not prompting.**

- Fine-tuning raises the probability of correct behavior; constrained decoding (Outlines/XGrammar) guarantees output FORM.
- _Why both:_ they govern different things (content odds vs form validity), and reliability is a tail property, so closing it takes both at once rather than a prompt that only shifts the average.
- Neither alone suffices: constrained decoding can force off-distribution tokens and hurt content; fine-tuning can't guarantee 100% valid structure.

**7. Small models fail in predictable, testable ways; build adversarial evals for exactly those.**

- Named fragilities: GSM-Symbolic (irrelevant-clause drops up to 65%), Faith-and-Fate (multiplication 59%→4%), Reversal Curse (near-0% on reversed facts), Lost-in-the-Middle (U-shaped context use), and the hallucination-calibration work.
- _Why it helps:_ because the failures are named and reproducible, the papers' own perturbations become your robustness suite, turning known weaknesses into a pre-deployment checklist.
- A robustness eval should perturb numbers, add distractor clauses, reverse relations, and move key info to the middle.

**8. "Emergence" is partly a measurement artifact.**

- Schaeffer et al. show apparent emergence tracks metric choice.
- _Why it matters here:_ a behavior that looks absent at small scale under exact-match may just be the metric masking steady progress; continuous metrics (edit distance, token-level, Brier) expose the gradient and prevent premature abandonment.

## DOK 2 — Knowledge Tree

### A. Data-centric AI & distillation

**LIMA: Less Is More for Alignment** — Zhou et al., Meta AI, NeurIPS 2023. https://arxiv.org/abs/2305.11206. *Peer-reviewed / researcher.*
- Fine-tuned LLaMa-65B on only 1,000 curated prompt-response pairs with standard supervised loss, no RLHF.
- In a controlled human study, LIMA responses were equivalent or preferred to GPT-4 in 43% of cases; it beat DaVinci003 (RLHF-trained) and Alpaca-65B (trained on 52× more data). Even GPT-4 preferred LIMA's output over its own 19% of the time.
- Hyperparameters: 15 epochs, LR 1e-5→1e-6, batch 32; perplexity did NOT correlate with generation quality, so checkpoints were selected manually on a 50-example dev set.
- **DOK 2 Summary:** Proposes the "Superficial Alignment Hypothesis" — nearly all knowledge is learned in pretraining and alignment mainly teaches format/style, so a tiny high-quality dataset can suffice. Intellectual foundation for behavior-focused small-data fine-tuning.

**Self-Instruct / Stanford Alpaca** — Wang et al. 2022 (https://arxiv.org/abs/2212.10560); Taori et al. 2023. *Peer-reviewed + technical / practitioner.*
- Self-Instruct bootstraps 52K instructions from 175 seed tasks using a base LLM; improved the SUPER-NATURALINSTRUCTIONS baseline by 33%.
- Alpaca reused the pipeline with text-davinci-003, generated 52K samples, and fine-tuned LLaMA-7B cheaply; AlpaGasus later showed filtering Alpaca's low-quality rows improves the model.
- **DOK 2 Summary:** Established the template for distilling a teacher's behavior into a smaller student via synthetic instruction data — the direct ancestor of the "generate data from a frontier teacher" workflow.

**Evol-Instruct / WizardLM** — Xu et al. 2023. *Peer-reviewed / researcher.*
- Iteratively rewrites simple instructions into more complex/diverse ones (in-depth and in-breadth evolution).
- **DOK 2 Summary:** Complexity and diversity of instructions, not just count, drive instruction-following quality — a concrete lever for dataset design.

**Orca: Progressive Learning from Complex Explanation Traces of GPT-4** — Mukherjee et al., Microsoft 2023. https://arxiv.org/abs/2306.02707. *Peer-reviewed / researcher.*
- 13B model trained to imitate GPT-4 reasoning via explanation traces and step-by-step thought, with ChatGPT as intermediate teacher; ~5M samples.
- Beat Vicuna-13B by >100% on Big-Bench Hard and 42% on AGIEval.
- Explicitly warns: naive imitation makes students copy the STYLE but not the REASONING of teachers, and shallow evaluation overestimates capability.
- **DOK 2 Summary:** To transfer a behavior (not just surface style), distill the process — reasoning traces and explanations — and evaluate rigorously, because style mimicry masks capability gaps.

**Textbooks Are All You Need (phi-1) / phi-1.5** — Gunasekar et al. 2023 (https://arxiv.org/abs/2306.11644); Li et al. 2023 (https://arxiv.org/abs/2309.05463). *Peer-reviewed / researcher.*
- phi-1: 1.3B params, trained on 6B tokens of "textbook-quality" web data + 1B tokens of GPT-3.5 synthetic textbooks/exercises; 50.6% pass@1 on HumanEval, 55.5% on MBPP. phi-1-small (350M) still hit 45% on HumanEval.
- phi-1.5: 1.3B, matches models 5× larger on reasoning; challenges the notion that capability is determined solely by scale.
- **DOK 2 Summary:** Data quality can substitute for scale — carefully curated/synthetic "textbook" data lets tiny models punch far above their weight, reinforcing the data-as-deliverable thesis.

**FineWeb-Edu / Cosmopedia / SmolLM-Corpus** — Hugging Face (Penedo et al.). *Technical / practitioner.*
- Cosmopedia: 25B tokens, 30M synthetic samples (Mixtral-generated textbooks/blogs/stories) — largest open synthetic dataset at release. FineWeb-Edu is an educational-quality filtered subset of FineWeb; FineMath ~50B tokens.
- **DOK 2 Summary:** Open replications confirm that quality-filtering and synthetic generation at scale produce better small models than raw web crawl.

*Collective note:* The synthetic-data surveys ("Best Practices and Lessons Learned on Synthetic Data," https://arxiv.org/abs/2404.07503; "A Survey on Post-training of LLMs," https://arxiv.org/abs/2503.06072) document that the recurring quality levers are complexity, diversity, and scale, and that quality filtering beats raw quantity. Tooling: **distilabel/Argilla** for programmatic synthetic-data and preference-pair pipelines.

### B. PEFT / LoRA

**LoRA: Low-Rank Adaptation** — Hu et al., Microsoft, ICLR 2022. https://arxiv.org/abs/2106.09685. *Peer-reviewed / researcher.*
- Freezes base weights, learns low-rank update matrices A,B; far fewer trainable params and no inference latency once merged.
- **DOK 2 Summary:** The core PEFT method — adapt behavior by learning a small rank-r perturbation instead of all weights.

**QLoRA: Efficient Finetuning of Quantized LLMs** — Dettmers et al., UW, NeurIPS 2023. https://arxiv.org/abs/2305.14314. *Peer-reviewed / researcher.*
- Finetunes a 65B model on a single 48GB GPU. Innovations: 4-bit NormalFloat (NF4), double quantization (~0.37 bits/param saved, ~3GB on a 65B model), paged optimizers. Storage in NF4, compute de-quantized to bf16.
- Guanaco reached 99.3% of ChatGPT on the Vicuna benchmark in 24h on one GPU; the authors trained >1,000 models.
- Found data quality > dataset size for instruction following, and that MMLU does not predict chatbot quality.
- **DOK 2 Summary:** Makes single-GPU fine-tuning of small models routine, with no measured performance loss vs 16-bit for the tested tasks.

**LoRA Learns Less and Forgets Less** — Biderman et al., Columbia/Databricks Mosaic, TMLR 2024. https://arxiv.org/abs/2405.09673. *Peer-reviewed / researcher.*
- On code and math, standard low-rank LoRA substantially underperforms full fine-tuning; full FT learns perturbations 10–100× higher rank than typical LoRA configs.
- But LoRA forgets less (better retains out-of-domain capability) and maintains more diverse generations; mitigates forgetting more than weight decay/dropout. Example: code IFT full-FT scored 0.414 vs LoRA r=64's 0.509 on retention.
- Best practices: target all modules (attention + MLP), use α=2r, rank ~16 as a starting point, ≥4 epochs; LoRA is highly LR-sensitive.
- **DOK 2 Summary:** LoRA trades peak in-domain capacity for regularization/forgetting-resistance — a favorable trade when instilling a narrow behavior without wrecking base competence.

**LoRA vs Full Fine-tuning: An Illusion of Equivalence** — Shuttleworth et al. 2024. https://arxiv.org/abs/2410.21228. *Peer-reviewed / researcher.*
- LoRA introduces "intruder dimensions" — high-ranking singular vectors dissimilar to the pretrained weights — that full FT does not; causal intervention on them shows they drive forgetting.
- **DOK 2 Summary:** LoRA and full FT reach structurally different solutions; that difference explains both LoRA's forgetting-resistance and its capacity limits.

**Sebastian Raschka — Practical Tips for Finetuning LLMs Using LoRA** (Ahead of AI, 2023). https://magazine.sebastianraschka.com/p/practical-tips-for-finetuning-llms. *Technical blog / practitioner.*
- QLoRA: ~33% memory savings for ~33% runtime increase. Optimizer choice barely matters. In his sweep r=256/α=512 gave the best performance.
- Apply LoRA to ALL layers, not just K/V. Multiple epochs on small instruction sets can overfit and hurt.
- Observed a model "unlearned arithmetic" because Alpaca lacked arithmetic examples — a concrete instance of narrow capability regression.
- **DOK 2 Summary:** Hard-won operational defaults for LoRA; the "unlearned arithmetic" anecdote grounds the forgetting risk from narrow data.

### C. SFT mechanics
- **Loss masking / train-on-completions-only**: standard TRL SFTTrainer practice, but "Instruction Tuning With Loss Over Instructions" (Shi et al. 2024) questions always masking the instruction.
- **Chat templates**: model-specific special tokens (LIMA introduced an explicit EOT token distinct from EOS); mismatched templates silently degrade behavior.
- **Hyperparameters**: small datasets overfit fast; LIMA found perplexity uncorrelated with quality, so gate on behavioral eval, not loss. Use packing for throughput; watch effective batch size and LR.
- **DOK 2 Summary (collective):** For narrow-behavior SFT the choices that matter most are correct chat-template/token handling, masking the prompt so loss applies only to completions, and early stopping by behavioral eval rather than loss.

### D. Preference optimization

**DPO: Direct Preference Optimization** — Rafailov et al., Stanford, NeurIPS 2023. https://arxiv.org/abs/2305.18290. *Peer-reviewed / researcher.*
- Reparameterizes the RLHF reward so the optimal policy has a closed form; trains directly on preference pairs with a simple classification loss — no separate reward model, no sampling during training.
- Matches or beats PPO-based RLHF on sentiment control, summarization, and single-turn dialogue while being far simpler and more stable.
- **DOK 2 Summary:** Makes preference tuning accessible on small hardware; the go-to "stretch" step after SFT when you have chosen/rejected pairs.

**ORPO, KTO, SimPO, IPO** — Hong et al. 2024; Ethayarajh et al. 2024; Meng et al. 2024 (https://arxiv.org/abs/2405.14734); Azar et al. 2024. *Peer-reviewed / researcher.*
- ORPO: reference-model-free, folds preference into SFT via an odds-ratio term (one model, one dataset), more efficient but hyperparameter-sensitive.
- KTO: learns from unpaired binary (good/bad) signals — no preference pairs required.
- SimPO: reference-free, length-normalized reward + target margin; reported to outperform DPO and ORPO in its own experiments while tolerating noisier labels.
- IPO: addresses DPO overfitting with a theoretically grounded objective.
- **DOK 2 Summary:** A menu trading off data format (paired vs unpaired), compute (reference model or not), and stability; choose by what feedback you can cheaply generate.

*Collective note:* Argilla's RLHF overview (https://argilla.io/blog/mantisnlp-rlhf-part-9/) and post-training-stack write-ups stress that DPO alone optimizes generic preference; controllable/domain behavior often needs SFT first, then a preference pass, and that all these methods are β- and length-term–sensitive.

### E. Compression / quantization & reasoning distillation

**Distilling the Knowledge in a Neural Network** — Hinton, Vinyals, Dean, 2015. https://arxiv.org/abs/1503.02531. *Peer-reviewed / researcher.* Origin of soft-target knowledge distillation.

**DeepSeek-R1 (distilled models)** — DeepSeek-AI, 2025. https://arxiv.org/abs/2501.12948. *Technical / researcher-practitioner.*
- Fine-tuned six dense students (1.5B–70B, Qwen2.5 & Llama) on 800K reasoning traces from R1; SFT only, no RL stage.
- Per the R1 technical report, DeepSeek-R1-Distill-Qwen-32B scores 72.6% on AIME 2024, 94.3% on MATH-500, and 57.2% on LiveCodeBench — results that significantly outperform previous open-source models and are comparable to o1-mini.
- Key claim: distilling from a strong teacher beats running large-scale RL directly on a small model.
- **DOK 2 Summary:** Landmark evidence that a hard capability (long-CoT reasoning) transfers into small models purely via SFT on teacher traces — though persistent gaps remain for ≤7B students.

**s1: Simple Test-Time Scaling** — Muennighoff et al., Stanford, EMNLP 2025. https://arxiv.org/abs/2501.19393. *Peer-reviewed / researcher.*
- Per the paper: "After supervised finetuning the Qwen2.5-32B-Instruct language model on s1K and equipping it with budget forcing, our model s1-32B exceeds o1-preview on competition math questions by up to 27% (MATH and AIME24)"; budget forcing (appending "Wait") scaled AIME24 from 50% to 57%.
- s1K = 1,000 traces selected for difficulty, diversity, quality. Training on the full 59K vs the 1K cost 394 vs 7 H100-hours for marginal gains — data selection dominates.
- Explicitly invokes LIMA's Superficial Alignment Hypothesis.
- **DOK 2 Summary:** A second independent confirmation that ~1,000 well-chosen examples can activate a latent capability; the behavior is elicited, not taught from scratch.

**Quantization formats** — GPTQ, AWQ, bitsandbytes (NF4), GGUF/llama.cpp. *Docs / practitioner.*
- **DOK 2 Summary (collective):** Post-training quantization (GGUF for llama.cpp/CPU, AWQ/GPTQ for GPU) lets a tuned small model deploy on edge; 4-bit is the practical sweet spot.

### F. Small-model landscape

**SmolLM3-3B** — Hugging Face, released July 8, 2025. https://huggingface.co/HuggingFaceTB/SmolLM3-3B. *Technical / practitioner.*
- 3B, trained on 11.2T tokens; dual reasoning (think/no_think); 64k context (128k via YaRN); fully open (data mixture, configs, 100+ intermediate checkpoints). Trained on 384 H100s for 24 days.
- Outperforms Llama-3.2-3B and Qwen2.5-3B; competitive with Qwen3-4B and Gemma3-4B. Instruct variant optimized for reasoning and tool use.
- **DOK 2 Summary:** Best current fully-open small base for tuning, with a documented recipe and a strong instruct variant.

*Collective note:* **Qwen3 (0.6/1.7/4B), Llama 3.2 (1B/3B), Gemma 3 small, Phi family, SmolLM2 (135M/360M/1.7B)** are the practical base pool. Instruct variants ship chat templates and instruction-following priors; base variants give a cleaner slate but need format teaching. Tokenizer/number-tokenization differences matter (Goat attributed arithmetic gains to LLaMA's consistent digit tokenization).

### G. Evaluation & LLM-as-judge

**Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena** — Zheng et al. 2023. https://arxiv.org/abs/2306.05685. *Peer-reviewed / researcher.*
- Per the paper: "strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans" (validated on ~3K controlled expert votes + ~3K crowdsourced votes).
- Documents position bias (most judges favor the first answer; only GPT-4 stays consistent in >60% of swapped cases), verbosity bias, and self-enhancement bias (judges favor their own outputs). Independent studies measure length bias around +17% for LLM judges vs ~+13% for humans.
- **DOK 2 Summary:** LLM-as-judge is scalable and roughly human-aligned but carries measurable biases — always randomize position, control length, and avoid judging with the same model family.

**Hamel Husain — Your AI Product Needs Evals / LLM-as-a-Judge / Evals FAQ** (hamel.dev). https://hamel.dev/blog/posts/evals/. *Technical blog / practitioner.*
- Error analysis first: read ~100 traces; stop when ~20 traces reveal no new failure mode ("theoretical saturation").
- Use a single domain-expert "benevolent dictator"; build a custom, friction-free data-viewing tool. Prefer binary pass/fail over 1–5 scales ("Critique Shadowing"); pass rate is a product decision, not necessarily 100%.
- Per the Evals FAQ: "In the projects we've worked on, we've spent 60-80% of our development time on error analysis and evaluation"; he notes a ~70% pass rate can indicate an eval that is actually stress-testing the app.
- **DOK 2 Summary:** Build evals before/alongside the model; the eval harness IS the improvement flywheel and the debugging infrastructure.

*Collective note:* **G-Eval** (Liu et al. 2023) uses CoT + form-filling for judge scoring; **JSONSchemaBench** (https://arxiv.org/abs/2501.10868) shows models still struggle with real-world schemas. Watch for contamination, overfitting to the judge, and benchmark gaming.

### H. Reliability & structured output

**Constrained decoding — Outlines, XGrammar, Guidance, llama.cpp GBNF; JSONSchemaBench** — Willard & Louf 2023; Dong et al. 2024 (XGrammar); https://arxiv.org/abs/2501.10868. *Docs + peer-reviewed / practitioner.*
- FSM/token-masking sets invalid-token logits to −∞, guaranteeing schema-valid output; Outlines compiles JSON schemas for ~O(1) valid-token lookup per step.
- Overhead ranges from minimal to ~2–5× for naive implementations; XGrammar is current SOTA for low overhead. Output quality can suffer when constraints force the model off its preferred tokens, but constrained decoding sometimes IMPROVES task performance.
- **DOK 2 Summary:** For structured-output behaviors, constrained decoding guarantees form while fine-tuning improves the model's tendency to produce correct content within that form — complementary, not substitutes.

### I. Failure modes & limits (dedicated strand)

**GSM-Symbolic** — Mirzadeh et al., Apple, 2024 (ICLR 2025). https://arxiv.org/abs/2410.05229. *Peer-reviewed / researcher.*
- All tested models drop accuracy when only numbers change in a template; performance degrades further as clause count rises.
- Per the paper: "adding seemingly relevant but ultimately irrelevant information to problems, we demonstrate substantial performance drops (up to 65%) across all state-of-the-art models" — Phi-3-mini experienced over a 65% drop on the GSM-NoOp variant.
- Interpretation: reasoning resembles sophisticated pattern matching, not robust logic. **Contested:** Ivanova and others critique the statistical rigor; treat as evidence of fragility, not proof of "no reasoning."

**Faith and Fate: Limits of Transformers on Compositionality** — Dziri et al., NeurIPS 2023. https://arxiv.org/abs/2305.18654. *Peer-reviewed / researcher.*
- GPT-4 zero-shot: 59% on 3×3-digit multiplication, dropping to 4% on 4×4; a few-shot scratchpad lifts 3×3 to 92% but stays near 0 on the hardest cases.
- Transformers reduce compositional reasoning to "linearized subgraph matching"; under stated assumptions the probability of error converges toward 1 as problem size grows. Exhaustive fine-tuning (~1.8M multiplication pairs) generalized in-distribution but "utterly failed" out-of-distribution.

**Why Language Models Hallucinate** — Kalai, Nachum, Vempala, Zhang (OpenAI/Georgia Tech), 2025. https://arxiv.org/abs/2509.04664. *Researcher.*
- Frames hallucination as statistical error: generative error rate is lower-bounded by the "Is-It-Valid" binary misclassification rate — even with clean data, generating valid outputs is statistically harder than classifying validity.
- Post-training persists/worsens hallucination because benchmarks reward confident guessing over "I don't know"; the proposed fix is reworking eval metrics to reward calibrated uncertainty/abstention.

**Lost in the Middle: How Language Models Use Long Contexts** — Liu et al., TACL 2024. https://arxiv.org/abs/2307.03172. *Peer-reviewed / researcher.*
- U-shaped accuracy: models use info at the start/end of context well and degrade sharply in the middle, even in explicitly long-context models.

**Are Emergent Abilities of Large Language Models a Mirage?** — Schaeffer, Miranda, Koyejo, Stanford, NeurIPS 2023. https://arxiv.org/abs/2304.15004. *Peer-reviewed / researcher.*
- Core claim: apparent "emergence" appears "due to the researcher's choice of metric rather than due to fundamental changes in model behavior with scale." Nonlinear/discontinuous metrics (exact match) manufacture sharp jumps; linear/continuous metrics (token edit distance, Brier score) show smooth, predictable scaling. They even *induce* apparent emergence in vision models by choosing metrics.

**The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"** — Berglund et al., ICLR 2024. https://arxiv.org/abs/2309.12288. *Peer-reviewed / researcher.*
- Models trained on "A is B" do not generalize to "B is A": "near 0% accuracy on reversals" across GPT-3-350M, Llama-7B, and GPT-3-175B; the log-probability of the correct reversed name is no higher than a random name. Data augmentation with paraphrases did not fix it. (In-context, models CAN reverse; the curse is about stored/trained knowledge.)

**An Empirical Study of Catastrophic Forgetting in LLMs During Continual Fine-tuning** — Luo et al., 2023 (EMNLP 2025 proceedings). https://arxiv.org/abs/2308.08747. *Peer-reviewed / researcher.*
- Catastrophic forgetting observed across 1B–7B models, and severity INTENSIFIES with scale during continual instruction tuning. Decoder-only BLOOMZ forgets less than encoder-decoder mT0. General instruction tuning (Alpaca vs LLaMA) mitigates forgetting; continual tuning can also reduce some biases (e.g., gender bias).

**Alignment tax** — Ouyang et al. (InstructGPT), NeurIPS 2022. https://arxiv.org/abs/2203.02155; confirmed by Lin et al., EMNLP 2024, https://arxiv.org/abs/2309.06256.
- InstructGPT coined the term: RLHF "comes at the cost of lower performance on certain tasks," with regressions on SQuAD, DROP, HellaSwag, and WMT'15 Fr→En. Mitigation: PPO-ptx (mixing pretraining-distribution gradients into PPO) largely closes the gap without hurting labeler scores. Lin et al. independently confirm the tax on OpenLLaMA-3B/Mistral-7B and note DPO induces less tax than other RLHF algorithms.

**DOK 2 Summary (collective for I):** Small (and large) LMs fail systematically at compositional/multi-step reasoning, robustness to irrelevant info, symmetric fact retrieval, middle-of-context use, and calibrated uncertainty. Fine-tuning a narrow behavior can cause narrow forgetting and an alignment tax. Design evals to probe these exact fragilities.

### J. Practitioner playbooks
- **Hamel Husain** (evals), **Sebastian Raschka** (LoRA experiments; "Build a Reasoning Model From Scratch"), **Unsloth / Axolotl** docs (fast QLoRA recipes; HF alignment-handbook), **Argilla / distilabel** (data + preference pipelines).
- **DOK 2 Summary:** The consensus workflow — define behavior + eval → generate/curate teacher data → QLoRA SFT → base-vs-tuned judge eval → optional DPO → quantize/deploy.

### K. Forums / community
- r/LocalLLaMA, Hugging Face forums, Unsloth/Axolotl Discords, Hacker News threads on QLoRA/DeepSeek-R1. Useful for hardware-specific gotchas, chat-template bugs, and reproductions; treat single-poster claims as anecdotes to verify.

## Recommendations

**Stage 0 — Define and instrument (before any training).** Write the ONE behavior as a falsifiable spec. Build the eval harness first: a held-out behavioral set plus an adversarial set that perturbs numbers, injects irrelevant clauses (GSM-Symbolic style), reverses relations (reversal curse), and shifts key info to the middle (lost-in-the-middle). Decide the judge protocol now — binary pass/fail, randomized answer order, a different model family as judge. *Benchmark that gates progress:* baseline the untuned model on this harness; the whole project is measured as the base-vs-tuned delta.

**Stage 1 — Data.** Generate ~1,000–2,000 examples from a frontier teacher, distilling the *process* (reasoning traces/explanations), then filter for quality and diversity (AlpaGasus-style). Use distilabel/Argilla for pipelines. *Threshold to add more data:* only if the base-vs-tuned delta plateaus below target with clean data — that signals a missing latent capability, not a data-quantity problem.

**Stage 2 — Train.** QLoRA (NF4, double quant) on a fully-open small base (SmolLM3-3B or Qwen3-1.7B/4B). Defaults: target all modules, α=2r, r≈16 (raise to 64–256 only if the target metric is capacity-bound), ≥3–4 epochs, sweep LR (LoRA is LR-sensitic). Mask the prompt (loss on completions only); verify the exact chat template. Early-stop on the behavioral eval, not loss. *Escalate to full FT or high-rank LoRA if:* the behavior is a genuinely new hard skill and low-rank LoRA underperforms full FT on the target metric.

**Stage 3 — Preference tuning (stretch).** Only after SFT clears most of the target. Build chosen/rejected pairs (or unpaired good/bad for KTO) and run DPO (or ORPO to fold it into one stage). Re-run the full eval to check for an alignment tax — regression on base capabilities. *Threshold to keep it:* preference tuning must improve the target behavior without dropping retained-capability checks.

**Stage 4 — Reliability & deploy.** Add constrained decoding (Outlines/XGrammar) for any structured output; measure the quality cost vs the validity gain. Quantize to GGUF (llama.cpp) or AWQ/GPTQ for deployment. *Kill-switch:* if constrained decoding measurably degrades content quality and the base is already schema-reliable, drop it.

## Caveats
- **Contested reasoning claims.** GSM-Symbolic's "not genuine reasoning" framing is disputed (Ivanova and others critique statistical rigor); treat it as strong evidence of *fragility*, not proof of no reasoning.
- **Vendor vs independent.** DeepSeek-R1, SmolLM3, and QLoRA headline numbers are self-reported by the authors/vendors; the s1, LIMA, MT-Bench, GSM-Symbolic, Faith-and-Fate, and LoRA-Learns-Less findings are peer-reviewed but some rely on GPT-4-as-judge, which carries the biases documented in Section G.
- **Alignment-tax magnitude is method-dependent.** InstructGPT quantifies regressions qualitatively; the headline drop is largely mitigated by PPO-ptx and DPO, so the "tax" is real but not fixed — measure it on your own benchmarks.
- **Scale gap in the evidence.** Several marquee results (LIMA-65B, s1-32B, Guanaco-65B, Faith-and-Fate on GPT-4) are on models much larger than the 0.6B–4B target range; the *mechanisms* (superficial alignment, forgetting, compositional limits) transfer, but exact numbers will not. Small models show these failure modes at least as strongly.
- **Fast-moving field.** Model releases (SmolLM3, Qwen3, DeepSeek-R1) and preference methods (SimPO/ORPO/KTO) are recent; recheck for newer bases and recipes before committing.
- **Two searches were cut short by budget** (a direct pull of the emergent-abilities and reversal-curse primary PDFs); those facts were verified via a dedicated sub-search against the primary arXiv abstracts and are cited accordingly.