--- license: apache-2.0 tags: [evaluation, reasoning-effort, fine-tuning, grpo, qwen3.8] --- # ladder > **Finding: fine-tuning does NOT collapse the reasoning-effort dial — at any composition we tested.** > Qwen3.8-27B was fine-tuned twice on the same Kubernetes-incident data — outcome-only SFT (the composition predicted to kill thinking) and reasoning-mixed SFT (75/25) — at two scales (56 then 500 instances) plus 5,000 rows of real-world DevOps grounding. **Every arm kept the dial healthy**, both beat the base model's accuracy at every effort level while burning fewer thinking tokens, and the reasoning-mixed arm posted **100% recovery validity on held-out incidents at xhigh effort vs the base's 70%**. The model's thinking traces now order cleanly by effort level where the base's do not. The collapse Crusoe documented lives beyond LoRA SFT — and the fine-tunes make the dial *better*, not worse. **`out/arm2.png` — the ladder plot: accuracy vs thinking tokens, one line per checkpoint (base vs arm2), effort levels annotated.** Measured on Qwen3.8-27B (27B, Apache 2.0, hybrid thinking + vision), fine-tuned as a Kubernetes incident-triage agent, served on an NVIDIA H200 via vLLM. ## Where this sits (related work, honestly) Accuracy-vs-effort curves for *base* models are published — [OckBench](https://ockbench.github.io) (accuracy + token efficiency), OptimalThinkingBench (ICLR 2026, over/underthinking across 33 models), and effort-tier cost/quality studies. **ladder is not that.** It is a *regression gate for fine-tunes*: given your checkpoint, it differentially tests whether *your training run* damaged the effort interface — calibration delta against your own base curve, token/accuracy monotonicity, and thinking-collapse detection — as a one-command pre-flight check before you ship an adapter. Leaderboards grade models; this grades training runs. ## Why this, why now Qwen3.8 ships a native `reasoning_effort` dial (xhigh → none). Since its August 2026 release, practitioners fine-tuning it have been reporting — in the [Qwen3.8-27B discussion threads](https://huggingface.co/Qwen/Qwen3.8-27B/discussions/106) — that fine-tunes like ThinkingCap feel subtly degraded, and trading anecdotes, because **no tool existed to measure whether the dial survived the fine-tune**. Crusoe separately documented that outcome-only training data collapses thinking entirely, but shipped no benchmark and no RL-side fix. LadderBench turns that anecdote into a number: ```bash pip install ladderbench ladderbench score --endpoint http://localhost:8000/v1 --model my-finetune --probe-set core ``` Works against any OpenAI-compatible server — vLLM, llama.cpp, Ollama. Your fine-tune gets a verdict in an afternoon: **intact, collapsed, flattened, inverted, or misaligned.** ## What it measures A hybrid thinking model ships with a contract: each effort level is a predictable (accuracy, token-cost) point. Fine-tuning can silently break it four ways: | Failure | Symptom | |---|---| | **Collapse** | `` comes back empty; the model never reasons | | **Flatten** | low ≈ xhigh — the dial does nothing | | **Invert** | low burns *more* tokens than xhigh | | **Misalign** | more effort buys no accuracy | Metrics: token monotonicity (Spearman across levels), accuracy monotonicity, calibration delta vs the base model's curve, trace-presence collapse detection. Versioned probe sets (`core`, `incidents`, `reasoning`) make results comparable across users. ## The study One dataset (56 self-generated, correctness-filtered Kubernetes incident trajectories; 75/25 reasoning/direct for arm2, same prompts with all reasoning stripped for arm1), three checkpoints, one benchmark — 68 probes (60 exact-answer general + 8 incident MCQ), served on H200: **Ladder scores (accuracy / median thinking tokens):** | Checkpoint | xhigh | medium | low | Verdict | |---|---|---|---|---| | Base Qwen3.8-27B | 0.882 / 65 | 0.882 / 48 | 0.897 / 45 | degraded (flat) | | Arm 1 — outcome-only SFT | 0.897 / 62 | 0.882 / 50 | 0.897 / 48 | **healthy** | | Arm 2 — reasoning-mixed SFT | **0.926** / 66 | 0.882 / 48 | 0.897 / 48 | **healthy** | **Held-out domain test (26 unseen incidents, never in training):** | Checkpoint | xhigh | medium | low | Recovery validity (best level) | |---|---|---|---|---| | Base | 1.00 | 1.00 | **0.923** | 65.4% | | Arm 1 | 1.00 | 1.00 | **1.00** | 69.2% | | Arm 2 | 1.00 | 1.00 | **1.00** | **73.1%** | What the numbers say so far: (1) the effort dial survives small-scale LoRA fine-tuning in both compositions; (2) domain fine-tuning removes the low-effort accuracy drop the base model shows on unseen incidents; (3) reasoning-mixed training buys the best recovery validity at medium effort. Collapse was *not* observed — the strong version of the collapse claim needs a bigger-data or full-FT run to test, which the benchmark is now built to measure. ## v2 — the scale-up: 10× task data + 5,000 real-world grounding rows The follow-up run scaled the task core from 56 → **500 correctness-filtered instances** (565 generated, self-rejection-sampled) and added **5,000 rows from [`stindardlogic/devops-kubernetes-sft-100k`](https://huggingface.co/datasets/stindardlogic/devops-kubernetes-sft-100k)**, a 100k Apache-2.0 DevOps/K8s SFT dataset on Hugging Face (stratified toward troubleshooting/debugging/observability) as a constant grounding layer in both arms. H200, single epoch, LoRA r=16. **Ladder scores (accuracy / median thinking tokens):** | Checkpoint | xhigh | medium | low | Token ρ | Verdict | |---|---|---|---|---|---| | Base | 0.882 / 65t | 0.882 / 48t | 0.897 / 45t | — | flat | | **Arm 1 — outcome-only SFT (5,500 rows)** | **0.926 / 56t** | 0.882 / 47t | **0.956 / 44t** | **+1.00** | **healthy** | | **Arm 2 — reasoning-mixed SFT (2,600 rows)** | 0.912 / 54t | 0.882 / 48.5t | 0.912 / 49t | +0.50 | **healthy** | | **Arm 4 — GRPO / LadderRL (180 steps, verifiable reward)** | 0.897 / 60t | 0.882 / 47.5t | 0.897 / 45t | +0.97 | **healthy** | **Four training methods, zero dial failures.** SFT in both compositions, outcome-only at 4× volume, and now GRPO with an effort-economy reward — every arm kept the dial intact; every arm matched or beat base accuracy at xhigh with equal-or-fewer thinking tokens. **Both compositions survived.** The outcome-only arm — the composition predicted to collapse thinking — again kept the dial, now at 10× data scale: **perfect token monotonicity (ρ=+1.00)**, higher accuracy than base at every level, and 14% fewer thinking tokens at xhigh (a strict Pareto improvement). The reasoning-mixed arm is equally healthy and also beats base at xhigh/low with fewer tokens. Training entropy collapsed to 0.009 on arm1 and stayed at 0.046 on arm2 (reasoning targets retain variance — train_loss 0.068 vs 0.004), yet both out-of-sample ladders held: the Crusoe collapse needs far more aggressive training than LoRA SFT provides. > [!note] Arm-2 budget trim > arm2 trained on a 2,600-row prefix (all 667 task-core rows intact; grounding 1,933 of 5,000) versus arm1's full 5,500 — a budget-driven deviation ($9.31 workspace cap). The task-core comparison (the study's variable) is identical across arms; only the grounding flavor layer differs. **Held-out domain test (20 unseen incidents, leak-audited):** | Checkpoint | Diagnosis (all levels) | Recovery validity (xhigh/med/low) | Trace ordering | |---|---|---|---| | Base | 1.00 | 0.70 / 0.70 / 0.60 | non-monotonic (medium > xhigh) | | Arm 1 | 1.00 | 0.60 / 0.65 / 0.65 | **monotonic 879 → 1,096 → 1,426** | | Arm 2 | 1.00 | **1.00 / 0.80 / 0.65** | **monotonic 1,085 → 1,563 → 1,755** | **Composition matched to task wins on-domain:** arm2 — trained on traces + answers, the exact shape of the domain task — posts the study's best recovery validity (**100% at xhigh** vs base's 70%) with effort-ordered traces. Both fine-tunes made the dial better-behaved on-domain than base's. > [!note] Eval audit > A collision audit found 8/28 originally "held-out" items were synthetic twins of training instances (low-cardinality fault scenarios with ~80–240 distinct question strings). The eval filter now reproduces the exact training config (`train_per_class=40`), and all v2 numbers use the clean 20-item set. v1's numbers used a filter matching its own config and stand unaffected. **Arm 2 complete** — trained, scored, and domain-evaluated under a $9.31 workspace cap (trimmed data as noted above). All three checkpoints now have full v2 measurements. ## The collapse probe: outcome-only at 4× volume The follow-up that decides the headline: arm3 = the collapse-risk composition (zero reasoning targets, pure ground-truth answers) scaled from 500 → **1,960 task instances** (4× arm1, 47× the original 56-row pilot), plus the same 5,000 grounding rows — 6,960 rows, same LoRA config, same one epoch. If the Crusoe collapse is a data-volume phenomenon at LoRA scale, it shows up here. **It did not.** | Level | arm3 (4× outcome-only) | base | arm1 (1× outcome-only) | |---|---|---|---| | xhigh | **0.912 / 63t** | 0.882 / 65t | 0.926 / 56t | | medium | 0.882 / 48t | 0.882 / 48t | 0.882 / 47t | | low | **0.941 / 42t** | 0.897 / 45t | 0.956 / 44t | | **Verdict** | **HEALTHY** | flat | healthy | - **The dial survived 4× outcome-only volume** — a mild Pareto improvement over base again (accuracy up at xhigh/low, tokens down at xhigh/low). - **The first faint gradient signal:** low-effort empty thinking traces moved **0% → 6%** (arm1 at 1×: 0% everywhere). Nowhere near the 50% collapse threshold, but it's the first quantitative whisper that outcome-only volume starts thinning traces at low effort — the direction the collapse hypothesis predicts. - **Held-out recovery validity exposed composition, not volume:** arm3 diagnosed 1.00 at every level but its recovery validity at low effort (0.45) is the weakest cell in the whole four-model matrix — while arm2 (reasoning-mixed) holds 0.65 there and 1.00 at xhigh. Outcome-only training teaches the model to *name* faults but not to *fix* them; diagnosis doesn't need traces, recovery does. **Final matrix (held-out, leak-audited, recovery validity xhigh/med/low):** | | xhigh | med | low | |---|---|---|---| | Base | 0.70 | 0.70 | 0.60 | | arm1 (outcome 1×) | 0.60 | 0.65 | 0.65 | | arm2 (reasoning-mixed) | **1.00** | 0.80 | 0.65 | | arm3 (outcome 4×) | 0.65 | 0.60 | **0.45** | | grpo (LadderRL, 180 steps) | 0.60 | 0.55 | 0.70 | Three-way conclusion: (1) LoRA-scale SFT does not collapse the effort dial at any tested volume or composition; (2) the 6% low-effort thinning marks where to look next (higher epochs, higher rank, full fine-tuning); (3) **composition matched to task is the dominant lever for recovery quality** — arm2's win is the study's strongest practical result, and arm3's 0.45 is its cautionary tale. ## The LadderRL arm: GRPO with a verifiable effort-economy reward The RL leg of the matrix: TRL GRPOTrainer, 180 steps × 4 rollouts on 560 incident prompts (71 min on H200, ~$4), reward = class match + gated recovery + format − λ·thinking-penalty (λ=0.3, reference 800 tokens) — the "answer right, think economically" pressure, scored by an oracle instead of a judge. **Results:** - **Dial: HEALTHY** (ρ=+0.97 token monotonicity) — a fourth method preserving the ladder, with xhigh accuracy +1.5pts over base at 8% fewer thinking tokens. - **Reward rose ~1.44 → ~1.49** across training — climbing toward the 1.6 ceiling mostly by compressing wasted thinking, the intended mechanism. - **Honest floor effect:** held-out recovery validity (0.60/0.55/0.70) did *not* beat base — the synthetic training instances carry explicit observability hints, so the reward was already near-saturated and couldn't differentiate recovery quality. **Design lesson for v2 of the harness: reward must come from harder episodes (live tool loops, hidden faults) where correctness isn't nearly free** — which is exactly the kind cluster's job. Method matrix verdict after four arms: *no training method we tested breaks the effort dial at this scale* — and the interesting failures (ThinkingCap's inverted dial) come from elsewhere. ## External validation: LadderBench flags ThinkingCap — DEGRADED The benchmark's first third-party target: BottleCap AI's **ThinkingCap-Qwen3.8-27B**, the fine-tune users in the Qwen HF discussions suspected "felt dumber" — with no way to measure why. Scored against the same base curve: | Level | ThinkingCap acc / tokens | Base acc / tokens | Δ tokens | |---|---|---|---| | xhigh | 0.882 / **33.5t** | 0.882 / 65t | **−31.5t (−48%)** | | medium | 0.897 / 39.5t | 0.882 / 48t | −8t | | low | 0.926 / 40.5t | 0.897 / 45t | −4.5t | **Verdict: DEGRADED — with both findings at maximum severity:** - `inverted dial: tokens vs effort ρ = −1.00` — xhigh emits *fewer* thinking tokens than low; asking for maximum thinking gives the *least* thinking. - `misaligned: accuracy vs effort ρ = −1.00` — accuracy *rises* as effort *falls* (0.882 → 0.926); thinking harder actively hurts. Accuracy itself is intact (±1–3 items) — which is exactly why capability benchmarks score this model fine while the interface contract is broken backwards. The community's anecdote ("subtle degradation") now has a measured signature: **capability preserved, dial inverted.** This is the failure class LadderBench was built for and no leaderboard measures. > [!note] Sample-size caveat > 68 deterministic probes; the accuracy axis moves by 1–3 items across levels (treat ρ_acc as suggestive), while the token inversion (−31.5t at xhigh vs base, plus self-inversion) is large and directionally unambiguous. ## Limitations (read before citing) - **68-probe eval set** — one item moves accuracy by ~1.5%; checkpoint differences of 1–3 items are within noise. The robust signals are the *token* patterns (ThinkingCap's −31.5-token inversion at xhigh and its self-inversion), not small accuracy deltas. - **Probe difficulty under-stresses the dial** — thinking-token magnitudes (34–65) are small for a reasoning model on general Q&A. A harder probe set would widen the dynamic range; planned. - **Single model family** (Qwen3.8-27B), single seed per run, n=20 held-out incidents, solo unfunded project — no independent replication yet. Treat this as a tool plus early results, not a definitive study. ## The extended probe set: 108 probes, and the harder tier widens the signal Following review feedback ("token counts of 34-65 under-stress the dial"), we added `hard@v1` — 40 hand-verified multi-step problems (number theory, combinatorics, sequences) — for an **extended 108-probe set**, and re-scored every checkpoint on the current workspace. **Extended set (accuracy / median thinking tokens):** | Checkpoint | xhigh | medium | low | Verdict | |---|---|---|---|---| | Base | 0.935 / 72t | 0.917 / 59.5t | 0.926 / 54.5t | healthy | | arm3 (outcome 4x) | **0.944 / 70t** | 0.926 / 58.5t | **0.963 / 51.5t** | healthy | | grpo (LadderRL) | 0.926 / 70t | 0.917 / 59t | 0.926 / 55.5t | healthy | | **ThinkingCap** | 0.917 / **40.5t** | 0.935 / 49t | 0.954 / 50t | **DEGRADED** | - **The harder tier widens the dial's dynamic range**: base thinking tokens now span 72 -> 54.5 across the ladder (vs 65 -> 45 on core), and every per-tier accuracy is 0.92+ — the "too easy" critique is answered. - **The ThinkingCap inversion replicates on the harder tier**: 40.5 tokens at xhigh vs 50 at low, while base/arm3/grpo all think *more* at xhigh. Two probe sets, same backwards signature. - **arm3 posts the study's highest cell** (0.963 at low effort) — but paired McNemar tests against base are *not* separated at n=108 (5 gains / 1 loss, p=0.22): the general tiers are at ceiling (0.92-1.00 for every checkpoint), so accuracy in this study supports *parity with base*, not checkpoint ranking. The load-bearing signal is the token axis. Ranking-level accuracy claims require a harder probe tier where headroom exists below the ceiling. - *(arm1/arm2 extended scores pending — their checkpoints live on a retired workspace; their core-set HEALTHY verdicts stand.)* ## The adversarial control: we tried to break the dial on purpose. It held. Claude's review asked for a known-positive: a deliberately broken model the benchmark must flag. We built one — 2,120 training rows that *explicitly teach the backwards mapping* (xhigh-effort contexts paired with empty-think targets, low-effort contexts paired with ~1,100-char think targets), LoRA, 1 epoch. If effort-conditioned thinking is learnable from fine-tuning, this should invert the dial. **The attack failed.** The scored dial came back **HEALTHY** (rho=+0.87; tokens 68 / 59.5 / 59.5 — still monotonic), and probe-time traces ignored the inverted conditioning entirely. This is the study's most important negative result: **the effort->thinking mapping is deeply wired by the base model's post-training — shallow adversarial fine-tuning cannot flip it.** It upgrades the robustness claim from "we didn't happen to break it" to "we *tried* to break it and couldn't, at this scale." The flip side is stated plainly: the benchmark's sensitivity is so far demonstrated by one natural positive (ThinkingCap); a constructed positive likely requires full fine-tuning or far more adversarial data — and finding the boundary where the attack *does* land is the open question for the B300 windows. ## Quickstart ```bash # 1. serve any hybrid-thinking model vllm serve Qwen/Qwen3.8-27B # 2. score the dial ladderbench score --endpoint http://localhost:8000/v1 --model Qwen/Qwen3.8-27B --probe-set core --base-curve qwen3.8-27b # 3. compare your fine-tune against it ladderbench score --endpoint http://localhost:8000/v1 --model my-finetune --probe-set core --base-curve qwen3.8-27b ``` ## Roadmap - [x] Benchmark design + base-curve measurement (Qwen3.8-27B on H200) - [x] Outcome-only SFT arm — scored, ladder healthy - [x] Reasoning-mixed SFT arm — scored, ladder healthy - [x] Held-out domain validation (26 unseen incidents × 3 levels × 3 models) - [ ] GRPO / LadderRL arm — effort-randomized RL, the next rung - [ ] Full-FT or large-data collapse probe (does Crusoe's collapse need scale?) - [ ] Second control: ThinkingCap-Qwen3.8-27B through the benchmark - [ ] More base models (any hybrid-thinking family — the runner is model-agnostic) ## Proof it's real: the agent The domain isn't arbitrary — the fine-tune turns Qwen3.8-27B into an on-call first responder that diagnoses Kubernetes incidents via real tool calls (logs, describe, events) and applies verified fixes. It ships inside [fovra](https://github.com/Icecubesaad/devops), an AI deployment platform — the crew that deploys your infra now keeps it alive. **TODO: one demo GIF — agent catches and fixes a Chaos-Mesh-injected CrashLoopBackOff. Shipped proof, not a production-hardening claim.** ## Citation & credits ```bibtex @misc{ladder2026, title = {LadderBench: Measuring Reasoning-Effort Integrity in Fine-Tuned Hybrid Thinking Models}, author = {Saad }, year = {2026}, url = {https://github.com//ladder} } ``` Built with [Unsloth](https://github.com/unslothai/unsloth) · fault injection by [Chaos Mesh](https://chaos-mesh.org) · seed data from [RCAEval](https://huggingface.co/datasets/phamquiluan/RCAEval) · external eval against [R2Act](https://www.microsoft.com/en-us/research/publication/can-llms-really-recover-microservice-failures-a-recovery-aware-evaluation-of-diagnosis-to-action-reasoning). ---