Model card: compare against TwIL-LM3/LM2 and external models; drop internal lambda-checkpoint comparisons

README.md

@@ -27,36 +27,41 @@ A 2.5B reasoning model for **formal logic** tasks, built from
 through LoRA supervised fine-tuning, WiSE-FT weight interpolation (λ = 0.25) and entropy-weighted
 GRPO reinforcement learning (MGPO, step 1400).
 
-It is the **balanced** point of its training arm. On the in-domain macro gate it scores **0.4178**,
-within 0.004 of TwIL-LM3 (0.4218) at about 18% fewer parameters, and it keeps a 10-dataset
-held-out macro of **0.6759** — second only to TwIL-LM3 (0.7339) among the models compared below.
-It also decodes **1.57x faster than TwIL-LM3** under identical forced work.
-
-The price of that balance is structured-output strictness. Its strict-7 score (0.1214) and strict
-MCQ accuracy (0.0000) are well below the λ = 0.5 sibling from the same arm (0.2457 and 0.2850),
-which gives back roughly 0.044 of held-out macro to get them. This release is the held-out-friendly
-end of that trade; the other end was not published. See [Limitations](#limitations-and-caveats).
+On the in-domain macro gate it scores **0.4178** — fourth of the twelve models with a reported
+gate, behind only TwIL-LM3 (0.4218), Qwen3-8B (0.5336) and Gemma-4-26B-A4B-it (0.6344). It is ahead
+of the other eight, including LFM2.5-8B-A1B (0.3757), which has about three times its total
+parameters, and Granite-4.1-3B (0.3435). It also decodes **1.57x faster than TwIL-LM3** under
+identical forced work.
+
+It is not a strong strict-output or general-benchmark model. Its strict-7 score (0.1214) ranks
+eleventh of twelve, its `lean_critic` accuracy (0.3100) is the lowest of all thirteen models
+compared, and its 10-dataset held-out macro (0.6759) is below every model of comparable size in the
+comparison. No paired evaluation against its own base is included, so this card makes no claim
+about what the fine-tune did to held-out capability. See [Results](#results) and
+[Limitations](#limitations-and-caveats).
 
 ## Highlights
 
-* **Gate parity with TwIL-LM3 at a smaller size.** Macro gate 0.4178 against 0.4218, at 2.53B
-  against 3.08B parameters. Read the parity as approximate: TwIL-LM3's figure is understated by
-  truncation (4.4% of its Track A rows hit the token cap), so its true gate is somewhat higher.
-* **Held-out capability is largely retained.** 10-dataset Track B macro 0.6759, against 0.6317 for
-  the λ = 0.5 sibling and 0.4333 for the SmolLM2-1.7B-based TwIL-LM2. The gap to the λ = 0.5 arm is
-  widest on the math sets: GSM8K 0.7867 against 0.6767 and GSM-Symbolic 0.7233 against 0.5033.
-* **A cleaner measurement.** Only 0.9% of Track A generations hit the 2048-token cap, under the 2%
-  threshold our protocol requires to mark a comparison `rankable`. TwIL-LM3 (4.4%) and the
-  SmolLM2-1.7B-based model (6.9%) are both above it.
+* **Fourth on the in-domain gate.** 0.4178 against 0.4218 for TwIL-LM3, 0.3927 for the
+  SmolLM2-1.7B-based TwIL-LM2 and 0.3757 for LFM2.5-8B-A1B. TwIL-LM3's figure is understated by
+  truncation (4.4% of its Track A rows hit the token cap), so read the gap to it as approximate.
+* **Close to TwIL-LM3 at a smaller size.** 0.004 behind on the gate at 2.53B against 3.08B
+  parameters — about 18% fewer.
+* **Formal-logic lanes where it is competitive.** `lean_formalize` token-F1 0.5159 is fourth of
+  thirteen, ahead of Qwen3-8B (0.4022), Gemma-4-26B-A4B-it (0.4107) and LFM2.5-8B-A1B (0.4655).
+  `rule_induction` 0.3292 is ahead of TwIL-LM3 (0.3192) and of Granite-4.1-3B (0.2476).
+* **Lowest `lm_corpus` perplexity in the comparison** (1.9808), and third on `math_corpus`
+  (3.3073). Read these with the tokenizer caveat under [Limitations](#limitations-and-caveats).
 * **Fast.** 21,369 decode tokens/s on one H100 in a controlled bench — 1.57x TwIL-LM3 and 1.22x the
   SmolLM2-1.7B-based model — because Granite's architecture decodes quickly, not because it answers
   short.
-* **Lowest `lm_corpus` perplexity in the comparison** (1.9808), and 3.3073 on `math_corpus`. Read
-  these with the tokenizer caveat under [Limitations](#limitations-and-caveats).
+* **A cleaner measurement.** Only 0.9% of Track A generations hit the 2048-token cap, under the 2%
+  threshold our protocol requires to mark a comparison `rankable`.
 * **Runs anywhere.** 2.53B parameters in bf16 (4.72 GiB), with a Q4\_K\_M GGUF at 1.44 GiB for CPU.
 
-It is not a general assistant, and it is not the strongest structured-output model here:
-`lean_critic` accuracy is 0.3100 against 0.6600 for TwIL-LM3.
+Where it is weak: strict-7 (0.1214, eleventh of twelve), strict MCQ accuracy (0.0000), `lean_critic`
+(0.3100, last of thirteen), and held-out benchmarks (10-dataset macro 0.6759, eleventh of thirteen).
+It is not a general assistant.
 
 ## Model Details
 
@@ -85,90 +90,118 @@ inherited from the base and was not evaluated for this card.
 
 ## Results
 
-Every column below is a checkpoint we trained or released, scored through the same harness and
-decoding settings described under [Evaluation protocol](#evaluation-protocol):
-
-* **this model** — the λ = 0.25, MGPO step-1400 checkpoint in this repository;
-* **λ = 0.25, step 300** — the next-best gate in the same arm;
-* **λ = 0.5, step 800** — the arm's highest Track A gate, with weaker held-out results (not released);
-* **TwIL-LM3** — [`webAI-Official/TwIL-LM3`](https://huggingface.co/webAI-Official/TwIL-LM3),
-  SmolLM3-3B, WiSE-FT λ = 0.25, MGPO step 2071;
-* **TwIL-LM2 (SmolLM2-1.7B)** — the earlier, SmolLM2-1.7B-Instruct-based TwIL-LM2 (WiSE-FT
-  λ = 0.75, MGPO step 1680), using its published figures.
+Every model below was scored through the same harness, prompts and decoding settings described
+under [Evaluation protocol](#evaluation-protocol). The columns are this model; the two other TwIL
+models — [TwIL-LM3](https://huggingface.co/webAI-Official/TwIL-LM3) (SmolLM3-3B) and the earlier,
+SmolLM2-1.7B-Instruct-based TwIL-LM2 — with their published figures; the two SmolLM bases; and the
+external models reported alongside them on those cards.
 
 ### Track A — in-domain formal logic
 
-| lane / metric              | **This model** | λ=0.25 s300 | λ=0.5 s800 | TwIL-LM3   | TwIL-LM2 (SmolLM2-1.7B) |
-| -------------------------- | -------------- | ----------- | ---------- | ---------- | ----------------------- |
-| parameters                 | 2.53B          | 2.53B       | 2.53B      | 3.08B      | 1.7B                    |
-| `lean_formalize` token-F1  | 0.5159         | 0.5455      | **0.7175** | 0.5869     | 0.6199                  |
-| `rule_induction` derivation | 0.3292        | 0.3471      | **0.5516** | 0.3192     | 0.5136                  |
-| `entailment_label` accuracy | 0.5300        | 0.5000      | **0.6250** | 0.5750     | 0.5850                  |
-| `mcq_answer` accuracy      | 0.0000         | 0.0000      | **0.2850** | 0.1100     | 0.1600                  |
-| `semantic_parse` token-F1  | 0.4013         | 0.4171      | 0.6171     | 0.4416     | **0.8428**              |
-| `lean_critic` accuracy     | 0.3100         | 0.3400      | 0.5350     | **0.6600** | 0.5250                  |
-| `procedural` accuracy      | 0.0100         | 0.0150      | **0.0750** | 0.0300     | 0.0100                  |
-| `fol_translation` primary  | 0.0000         | 0.0000      | 0.1000     | 0.0000     | **0.1150**              |
-| `lm_corpus` perplexity ↓   | **1.9808**     | 1.9948      | 1.9982     | 2.8972     | 2.2981                  |
-| `math_corpus` perplexity ↓ | 3.3073         | 3.3731      | 3.2403     | 3.8229     | **3.0390**              |
-| average, 6 lanes           | 0.3477         | 0.3583      | **0.5552** | 0.4488     | 0.5410                  |
-| **macro gate**             | 0.4178         | 0.4114      | **0.4393** | 0.4218     | 0.3927                  |
-| **strict-7**               | 0.1214         | 0.1221      | **0.2457** | 0.1971     | 0.2386                  |
-| macro\_primary             | 0.4400         | 0.4275      | 0.4113     | **0.4475** | 0.3625                  |
-| mean generation length ↓ (tokens) | 517     | 505         | **213**    | 443        | 460                     |
-| truncation at 2048 tokens  | 0.9%           | 0.6%        | 0.5%       | 4.4%       | 6.9%                    |
-| `rankable` (< 2% truncation) | yes          | yes         | yes        | no         | no                      |
-
-`average, 6 lanes`, `macro gate`, `macro_primary` and `strict-7` are the harness aggregates defined
-on the [TwIL-LM3 model card](https://huggingface.co/webAI-Official/TwIL-LM3); this card does not
-redefine them. They are not interchangeable, and the ordering changes between them — this model
-leads neither `macro_primary` nor `strict-7`.
-
-**Reading it.** This is a mid-table formal-logic model whose main asset is held-out retention.
-Against TwIL-LM3 it wins on `rule_induction` (0.3292 against 0.3192) and on both perplexities, and
-loses on the rest, heavily on `lean_critic` (0.3100 against 0.6600). Its strict-7 (0.1214) is well
-below TwIL-LM3's (0.1971), so the near-parity is on the gate, which credits loose matches on some
-lanes, and not on strict scoring.
-
-The λ = 0.5 sibling is the stronger *formal-logic* model — it leads nine of the fifteen scored rows
-above — and the SmolLM2-1.7B-based model is the semantic-parsing specialist at 0.8428. Neither
-keeps held-out capability the way this checkpoint does; see Track B.
-
-Two things are worth stating plainly. Strict MCQ accuracy is **0.0000** for both λ = 0.25
-checkpoints: they do not emit the exactly-requested answer form on that lane. And harness
-answers-per-second (12.07 for this model, 17.57 for λ = 0.5) reflects how long each model chooses
-to write, not how fast it decodes; use the controlled bench below for speed.
+| lane / metric | TwIL-LM2 (this model) | TwIL-LM3 | TwIL-LM2 (SmolLM2-1.7B) | SmolLM3-3B base | SmolLM2-1.7B base | LFM2.5-1.2B-Thinking | LFM2-2.6B | Llama-3.2-3B | Granite-4.1-3B | LFM2.5-8B-A1B | Qwen3-8B | Gemma-4-26B-A4B-it | gpt-oss-120b ‡ |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| parameters | 2.53B | 3.08B | 1.7B | 3.08B | 1.7B | 1.2B | 2.6B | 3B | 3B | 8B (1B active) | 8B | 26B (4B active) | 120B |
+| `lean_formalize` token-F1 | 0.5159 | 0.5869 | 0.6199 | 0.4347 | 0.1087 | 0.1890 | 0.1321 | 0.3690 | 0.2652 | 0.4655 | 0.4022 | 0.4107 | **0.6306** |
+| `rule_induction` derivation | 0.3292 | 0.3192 | 0.5136 | 0.1029 | 0.1350 | 0.0837 | 0.0615 | 0.0825 | 0.2476 | 0.1936 | 0.3680 | **0.7319** | 0.6518 |
+| `entailment_label` accuracy | 0.5300 | 0.5750 | 0.5850 | 0.3750 | 0.2450 | 0.4700 | 0.4700 | 0.3300 | 0.4900 | 0.5400 | 0.5800 | 0.6200 | **0.7750** |
+| `mcq_answer` accuracy | 0.0000 | 0.1100 | **0.1600** | 0.0000 | 0.0000 | 0.0000 | 0.0150 | 0.0000 | 0.0100 | 0.0750 | 0.0000 | 0.0200 | 0.0700 |
+| `semantic_parse` token-F1 | 0.4013 | 0.4416 | **0.8428** | 0.4149 | 0.2155 | 0.4439 | 0.3665 | 0.3102 | 0.1953 | 0.3778 | 0.4257 | 0.4567 | 0.4331 |
+| `lean_critic` accuracy | 0.3100 | 0.6600 | 0.5250 | 0.6500 | 0.4950 | 0.5450 | 0.5900 | 0.5300 | 0.5150 | 0.5500 | **0.7950** | 0.7500 | 0.5550 |
+| `lm_corpus` perplexity ↓ | **1.9808** | 2.8972 | 2.2981 | 3.1818 | 2.5845 | 5.0065 | 4.3815 | 2.8478 | 2.4736 | 4.9472 | 2.5440 | 16.1145 | 912.23 § |
+| `math_corpus` perplexity ↓ | 3.3073 | 3.8229 | **3.0390** | 4.0685 | 3.2670 | 7.7402 | 6.7472 | 4.7531 | 4.1162 | 8.3323 | 4.0083 | 59.7838 | 1045.63 § |
+| average, 6 lanes | 0.3477 | 0.4488 | **0.5410** | 0.3296 | 0.1999 | 0.2886 | 0.2725 | 0.2703 | 0.2872 | 0.3670 | 0.4285 | 0.4982 | 0.5192 |
+| **macro gate** | 0.4178 | 0.4218 | 0.3927 | 0.3466 † | 0.2590 † | 0.3067 | 0.3473 | 0.2925 | 0.3435 | 0.3757 | 0.5336 | **0.6344** | — |
+| **strict-7** | 0.1214 | 0.1971 | **0.2386** | 0.1493 | 0.1071 | 0.1450 | 0.1579 | 0.1229 | 0.1507 | 0.1714 | 0.2093 | 0.2050 | — |
+| macro\_primary | 0.4400 | 0.4475 | 0.3625 | 0.4075 | 0.2900 | 0.3625 | 0.4188 | 0.3450 | 0.3675 | 0.4213 | 0.5750 | **0.6100** | — |
+
+† The base columns come from the external-comparison run rather than the paired base-versus-TwIL
+run, hence SmolLM3-3B 0.3466 here against 0.3356 in its paired run and SmolLM2-1.7B 0.2590 against
+0.2630. The paired run is the correct basis for an improvement claim.
+
+‡ **gpt-oss-120b** runs MXFP4 weights at tensor-parallel 2 — quantized and multi-GPU, so it is not
+directly comparable to the single-GPU bf16 columns. Its `procedural` lane and the loose-match
+scorings were not collected, so its gate, strict-7 and macro\_primary cannot be computed; the —
+cells mean that, not zero.
+
+§ The 120B's perplexities are three orders of magnitude off every other model because its harmony
+response format and tokenizer make the corpus lanes score a different quantity. They are reported
+for completeness and excluded from the perplexity ranking.
+
+Throughput and generation-length rows are left out of this table: the source cards report them
+from different runs, so they cannot be put in one column. The controlled decode bench under
+[Speed](#speed) is the like-for-like speed comparison. `average, 6 lanes`, `macro gate`,
+`macro_primary` and `strict-7` are the harness aggregates defined on the
+[TwIL-LM3 model card](https://huggingface.co/webAI-Official/TwIL-LM3); this card does not redefine
+them. They are not interchangeable, and the ordering changes between them.
+
+**Reading it.** Among the three TwIL models this is second on the gate — 0.4178, between 0.4218 for
+TwIL-LM3 and 0.3927 for the SmolLM2-1.7B-based model — but it is well behind both on strict-7
+(0.1214 against 0.1971 and 0.2386). The gate credits loose matches on some lanes and
+strict-7 credits none, so the near-parity is on the gate, not on strict scoring.
+
+Against the wider set it is fourth on the gate and on `macro_primary`, behind TwIL-LM3, Qwen3-8B
+and Gemma-4-26B-A4B-it, and ahead of everything else with a reported gate. It is fourth of thirteen
+on `lean_formalize`, fifth on `rule_induction` and seventh on `entailment_label`. It is eighth on
+`semantic_parse` (0.4013, against 0.8428 for the SmolLM2-1.7B-based model) and last on
+`lean_critic` (0.3100, against 0.6600 for TwIL-LM3 and 0.7950 for Qwen3-8B).
+
+Strict MCQ accuracy is 0.0000. That is not unique to this model — Qwen3-8B, Llama-3.2-3B,
+LFM2.5-1.2B-Thinking and both SmolLM bases also score 0.0000, because they answer the lane without
+emitting the requested form — but it contributes to a strict-7 that is eleventh of twelve; only
+SmolLM2-1.7B base (0.1071) is lower. On its Track A `procedural` lane it scores 0.0100 and on
+`fol_translation` 0.0000.
+
+It does not beat the two largest models with reported gates: Qwen3-8B leads it 0.5336 to 0.4178 and
+Gemma-4-26B-A4B-it 0.6344 to 0.4178. Much of the Qwen gap is loose-match credit rather than
+capability (Qwen3-8B answers MCQ correctly but almost never in the requested format), though Gemma
+also genuinely leads on rule induction (0.7319), which no scoring convention explains away.
 
 ### Track B — held-out benchmarks
 
-Nothing in this suite was trained on. Ten chain-of-thought datasets, 300 randomly sampled examples
-each, identical rows for every column.
-
-| dataset                     | **This model** | λ=0.25 s300 | λ=0.5 s800 | TwIL-LM3   | TwIL-LM2 (SmolLM2-1.7B) |
-| --------------------------- | -------------- | ----------- | ---------- | ---------- | ----------------------- |
-| `gsm8k`                     | 0.7867         | 0.8000      | 0.6767     | **0.8733** | 0.4633                  |
-| `svamp`                     | 0.8200         | 0.8400      | 0.7333     | **0.8500** | 0.3833                  |
-| `gsm_symbolic`              | 0.7233         | 0.7400      | 0.5033     | **0.7567** | 0.2600                  |
-| `arc_cot`                   | 0.7367         | 0.7300      | 0.7067     | **0.8467** | 0.5200                  |
-| `logicbench`                | 0.6900         | 0.6933      | 0.6833     | **0.7167** | 0.5400                  |
-| `strategyqa`                | 0.6933         | 0.6733      | **0.7233** | 0.6500     | 0.5900                  |
-| `drop`                      | 0.5833         | 0.6000      | 0.5700     | **0.7467** | 0.4367                  |
-| `csqa`                      | 0.6900         | 0.6767      | 0.7233     | **0.7367** | 0.4333                  |
-| `musr` (mean of 3)          | 0.4957         | 0.4877      | 0.4904     | **0.4958** | 0.3131                  |
-| `mmlu_redux`                | 0.5400         | 0.5167      | 0.5067     | **0.6667** | 0.3933                  |
-| **macro (10 CoT datasets)** | 0.6759         | 0.6758      | 0.6317     | **0.7339** | 0.4333                  |
-
-**TwIL-LM3 leads this table**, on nine of ten datasets and on the macro. This model is second on
-the macro (and ties with the s300 checkpoint), ahead of the λ = 0.5 arm by 0.044 and of the
-SmolLM2-1.7B-based model by 0.243. It is close to TwIL-LM3 on `musr` (0.4957 against 0.4958) and
-`logicbench` (0.6900 against 0.7167), and furthest behind on `drop` (−0.163) and `mmlu_redux`
-(−0.127).
-
-The 14-dataset macro (adding `ifeval`, `rudas_ood`, `bbh_logic` and `math500`) was not run for the
-Granite checkpoints in a directly comparable tree, so it is not reported here, and **no
-instruction-following result is claimed**. It is also not known from these runs how this model
-compares with its own base, `granite-3.3-2b-instruct`: no paired base-versus-tuned evaluation is
-included in this card.
+Nothing in this suite was trained on. All models are scored by the same aggregation over 300
+randomly sampled, model-identical examples per dataset.
+
+| dataset | TwIL-LM2 (this model) | TwIL-LM3 | TwIL-LM2 (SmolLM2-1.7B) | SmolLM3-3B base | SmolLM2-1.7B base | LFM2.5-1.2B-Thinking | LFM2-2.6B | Llama-3.2-3B | Granite-4.1-3B | LFM2.5-8B-A1B | Qwen3-8B | Gemma-4-26B-A4B-it | gpt-oss-120b ‡ |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| `gsm8k` | 0.7867 | 0.8733 | 0.4633 | 0.8833 | 0.4800 | 0.8400 | 0.8767 | 0.8300 | 0.9100 | 0.9133 | 0.9567 | 0.9733 | **0.9767** |
+| `svamp` | 0.8200 | 0.8500 | 0.3833 | 0.8567 | 0.4867 | 0.9167 | 0.9000 | 0.8200 | 0.9000 | 0.9133 | 0.9367 | **0.9500** | 0.9400 |
+| `gsm_symbolic` | 0.7233 | 0.7567 | 0.2600 | 0.7633 | 0.2200 | 0.6867 | 0.9767 | 0.8067 | 0.9533 | 0.9267 | 0.8133 | **0.9967** | 0.8467 |
+| `arc_cot` | 0.7367 | 0.8467 | 0.5200 | 0.8400 | 0.5100 | 0.8300 | 0.8667 | 0.7967 | 0.8633 | 0.9033 | 0.9633 | **0.9767** | 0.9667 |
+| `logicbench` | 0.6900 | 0.7167 | 0.5400 | 0.6467 | 0.5067 | 0.6700 | 0.6267 | 0.5733 | 0.7367 | 0.7200 | 0.8567 | **0.8667** | 0.8533 |
+| `strategyqa` | 0.6933 | 0.6500 | 0.5900 | 0.6333 | 0.6000 | 0.5933 | 0.6433 | 0.6533 | 0.6333 | 0.6667 | 0.7400 | 0.7700 | **0.7867** |
+| `drop` | 0.5833 | 0.7467 | 0.4367 | 0.7000 | 0.4233 | 0.6667 | 0.6900 | 0.6733 | 0.7600 | 0.6633 | **0.8833** | 0.7933 | 0.8500 |
+| `csqa` | 0.6900 | 0.7367 | 0.4333 | 0.7067 | 0.3967 | 0.6100 | 0.7433 | 0.7500 | 0.7633 | 0.7700 | **0.8633** | **0.8633** | 0.8367 |
+| `musr` | 0.4957 | 0.4957 | 0.3131 | 0.4997 | 0.4223 | 0.5227 | 0.4867 | 0.4932 | 0.5669 | 0.5703 | 0.6301 | 0.6369 | **0.6852** |
+| `mmlu_redux` | 0.5400 | 0.6667 | 0.3933 | 0.6633 | 0.4100 | 0.6400 | 0.7133 | 0.6000 | 0.6800 | 0.8367 | 0.8500 | **0.9633** | 0.9467 |
+| `ifeval` | — | 0.6433 | 0.4300 | 0.6767 | 0.4700 | 0.8233 | 0.7300 | 0.7167 | 0.7967 | **0.8900** | 0.8400 | 0.8733 | 0.7900 |
+| `rudas_ood` | — | 0.0365 | 0.0289 | 0.0209 | 0.0128 | 0.0089 | 0.0017 | 0.0733 | 0.0355 | 0.0061 | 0.0468 | **0.1547** | 0.0000 ¶ |
+| `bbh_logic` | — | 0.6633 | 0.2373 | 0.6667 | 0.2447 | 0.5327 | 0.5713 | 0.5333 | 0.7727 | 0.7700 | 0.6367 | 0.9940 | **0.9980** |
+| `math500` | — | 0.6900 | 0.2100 | 0.7000 | 0.1900 | 0.6867 | 0.7133 | 0.4233 | 0.6067 | 0.7800 | 0.6100 | **0.9000** | 0.8433 |
+| **macro (10 CoT datasets)** | 0.6759 | 0.7339 | 0.4333 | 0.7193 | 0.4456 | 0.6976 | 0.7523 | 0.6997 | 0.7767 | 0.7884 | 0.8493 | **0.8790** | 0.8689 |
+| **macro (all 14)** | — | 0.6694 | 0.3742 | 0.6612 | 0.3838 | 0.6448 | 0.6814 | 0.6245 | 0.7127 | 0.7378 | 0.7591 | **0.8366** | 0.8086 |
+
+‡ **gpt-oss-120b**: MXFP4 weights, tensor-parallel 2 — quantized and multi-GPU, so not directly
+comparable to the single-GPU bf16 columns. ¶ 74% of its `rudas_ood` generations hit the length cap,
+so that cell is a truncation artefact rather than a measured score and is excluded from the bolding.
+
+The — cells for this model are lanes that were **not run** for this checkpoint (`ifeval`,
+`rudas_ood`, `bbh_logic`, `math500`, and therefore the 14-dataset macro); no instruction-following
+or 14-dataset result is claimed. Qwen3-8B `svamp` is 0.9367 as on the earlier TwIL-LM2 card; the
+TwIL-LM3 card lists 0.9400, which does not reproduce that card's own Qwen3-8B macros.
+
+**Reading it.** On the 10-dataset macro this model scores 0.6759, eleventh of thirteen. It is ahead
+of only the two SmolLM2-1.7B entries (0.4333 and 0.4456) and behind every model of comparable
+size: LFM2-2.6B (0.7523), SmolLM3-3B base (0.7193), Llama-3.2-3B (0.6997), LFM2.5-1.2B-Thinking
+(0.6976) and Granite-4.1-3B (0.7767). TwIL-LM3, the strongest TwIL model here, scores 0.7339.
+
+It is strongest on `strategyqa` (0.6933, fourth of thirteen, ahead of TwIL-LM3 at 0.6500, Llama-3.2-3B
+and Granite-4.1-3B) and holds a mid-table position on `logicbench` (0.6900, seventh of thirteen). It
+ranks eleventh of thirteen on `gsm8k` (0.7867), `arc_cot` (0.7367), `drop` (0.5833) and
+`mmlu_redux` (0.5400).
+
+The comparison that would say whether the fine-tune moved held-out performance is the one against
+`granite-3.3-2b-instruct` itself, and it is not in this card. Granite-4.1-3B, in the tables above, is
+a different and larger model, not this model's base.
 
 ### Speed
 
@@ -177,40 +210,18 @@ an actual speed comparison, each model ran alone on one idle H100 (vLLM 0.19.1,
 transformers 5.15.0) over the same 128 prompts with `ignore_eos` and a hard 512-token cap, so every
 model emitted exactly 65,536 output tokens.
 
-| controlled decode bench         | **This model** | TwIL-LM3 | TwIL-LM2 (SmolLM2-1.7B) |
-| ------------------------------- | -------------- | -------- | ----------------------- |
-| decode tokens/s                 | **21,369**     | 13,623   | 17,542                  |
-| 512-token completions/s         | **41.7**       | 26.6     | 34.2                    |
-| decode wall seconds (65,536 tok) | **3.07**      | 4.81     | 3.74                    |
-| relative to this model          | 1.00x          | 0.64x    | 0.82x                   |
+| controlled decode bench          | **This model** | TwIL-LM3 | TwIL-LM2 (SmolLM2-1.7B) |
+| -------------------------------- | -------------- | -------- | ----------------------- |
+| decode tokens/s                  | **21,369**     | 13,623   | 17,542                  |
+| 512-token completions/s          | **41.7**       | 26.6     | 34.2                    |
+| decode wall seconds (65,536 tok) | **3.07**       | 4.81     | 3.74                    |
+| relative to this model           | 1.00x          | 0.64x    | 0.82x                   |
 
-The three Granite checkpoints decode within 1.1% of one another (21,236 to 21,467 tokens/s), so
-the speed advantage comes from the architecture, not from a particular checkpoint. The shared
+Decode speed here is set by the architecture, not by anything the fine-tune changed. The shared
 prompt file is Granite-templated, so prefill differs slightly by tokenizer (30.7K tokens for
 Granite and SmolLM3, 38.2K for SmolLM2); it is a small share of the forced output and, if anything,
-slightly handicaps the SmolLM2-based model.
-
-### Choosing a checkpoint within the arm
-
-All four evaluated λ = 0.25 checkpoints sit within 0.0006 of one another on Track B, so held-out
-performance cannot separate them; the ordering comes from Track A and the probe.
-
-| MGPO step | Track B R10 | Track A macro gate |
-| --------- | ----------- | ------------------ |
-| 300       | 0.6758      | 0.4114             |
-| 1100      | 0.6764      | 0.4019             |
-| **1400**  | 0.6759      | **0.4178**         |
-| 1700      | 0.6761      | 0.4059             |
-
-Step 1400 is the checkpoint chosen by probe Pass@1 and has the highest gate of the four.
-
-### Where it sits in the wider Granite family
-
-For context, an internal, **unreleased** Granite 4.1 3B MGPO checkpoint (fusion + WiSE-FT λ = 0.5,
-step 600), scored on the same Track A and Track B protocol, reaches a macro gate of 0.5105,
-strict-7 of 0.2886 and Track B R10 of 0.7761 — 9.3, 16.7 and 10.0 percentage points above this
-model respectively. This model is the smaller, faster option; it is not the strongest Granite
-checkpoint we have trained.
+slightly handicaps the SmolLM2-based model. The other models in the tables above were not run in
+this bench.
 
 ## Usage
 
@@ -286,28 +297,29 @@ Three stages on top of the base model:
    SFT recipe.
 2. **WiSE-FT interpolation** toward the pretrained base,
    `W = (1 − λ)·W_base + λ·W_finetuned` with **λ = 0.25** — only a quarter of the fine-tuned delta
-   is retained. This conservative λ is the main reason held-out capability survives; the λ = 0.5
-   sibling retains twice the delta and shows it in the strict formal-logic lanes and in the loss of
-   held-out math.
+   is retained. λ was chosen to keep as much held-out capability as possible while still gaining
+   in-domain.
 3. **MGPO** — entropy-weighted GRPO reinforcement learning against a programmatic verifier, with
    partial credit for loose matches and token-F1 so that all-fail prompt groups still produce
-   gradient. Published checkpoint is **step 1400**.
+   gradient. Published checkpoint is **step 1400**, chosen by probe Pass@1.
 
-Unlike TwIL-LM3, there is no checkpoint-fusion stage between SFT and WiSE-FT in this arm.
+Unlike TwIL-LM3, there is no checkpoint-fusion stage between SFT and WiSE-FT in this model.
 
 ## Limitations and caveats
 
-**Strict output form.** Strict MCQ accuracy is 0.0000 and `procedural` accuracy 0.0100;
-`fol_translation` primary score is 0.0000; strict-7 is 0.1214. The model reasons near the required
-form without reliably emitting it. If you need exactly-formatted formal objects, the λ = 0.5
-checkpoint's profile (strict-7 0.2457, MCQ 0.2850) is better, at the cost of held-out math.
+**Held-out benchmarks.** The 10-dataset Track B macro is 0.6759, eleventh of thirteen and below
+every model of comparable size in the table. No paired base-versus-tuned run is included, so this
+card cannot say whether that is a regression from `granite-3.3-2b-instruct` or the base's level.
+Four Track B lanes (`ifeval`, `rudas_ood`, `bbh_logic`, `math500`) were not run.
 
-**Lean critique.** `lean_critic` accuracy is 0.3100, well below TwIL-LM3 (0.6600) and the
-SmolLM2-1.7B-based model (0.5250). This is the weakest of the Track A lanes relative to its peers.
+**Strict output form.** Strict MCQ accuracy is 0.0000, `procedural` accuracy 0.0100,
+`fol_translation` primary score 0.0000 and strict-7 0.1214 (eleventh of twelve). The model reasons
+near the required form without reliably emitting it. If you need exactly-formatted formal objects,
+the SmolLM2-1.7B-based TwIL-LM2 (strict-7 0.2386, semantic parsing 0.8428) is the stronger option
+in this comparison.
 
-**No base comparison.** This card does not include a paired evaluation of
-`granite-3.3-2b-instruct` on the same harness, so it makes no claim about how much the tuning
-improved or regressed the base model on either track.
+**Lean critique.** `lean_critic` accuracy is 0.3100, the lowest of the thirteen models compared and
+well below TwIL-LM3 (0.6600).
 
 **Perplexity across tokenizers.** `lm_corpus` and `math_corpus` perplexity is a per-token quantity,
 and the columns use different tokenizers (Granite and SmolLM2 have about 49K entries each but
@@ -315,15 +327,15 @@ distinct vocabularies; SmolLM3 has 128K). The perplexity rows are informative wi
 only indicative across families.
 
 **Result trees.** For Track B, TwIL-LM3 and the SmolLM2-1.7B-based model are scored from the
-results tree that matches their published cards (rope-fixed), and the Granite checkpoints from the
-default tree, with vLLM 0.19.1; the two TwIL macros reproduce their published values exactly
-(0.7339 and 0.4333). Track A figures for all five columns come from GATE 2 reports at n = 200 per
-lane and a 2048-token cap.
+results tree that matches their published cards (rope-fixed), and this model from the default tree,
+with vLLM 0.19.1; the two TwIL macros reproduce their published values exactly (0.7339 and 0.4333).
+Track A figures for this model come from GATE 2 reports at n = 200 per lane and a 2048-token cap.
 
-**Truncation of the comparison arms.** TwIL-LM3 and the SmolLM2-1.7B-based model are formally
-`rankable: false` at 4.4% and 6.9% truncation. A truncated response scores zero regardless of
-reasoning quality, so their Track A figures are understated: wherever this model is ahead of them
-the true margin is smaller, and wherever it is behind, the true deficit is larger.
+**Truncation in the comparison.** At the 2048-token cap, TwIL-LM3 (4.4%), the SmolLM2-1.7B-based
+TwIL-LM2 (6.9%), SmolLM3-3B base (17.4%) and SmolLM2-1.7B base (11.7%) are all above the 2%
+threshold and formally `rankable: false`. A truncated response scores zero regardless of reasoning
+quality, so their Track A figures are understated: wherever this model is ahead of them the true
+margin is smaller, and wherever it is behind, the true deficit is larger.
 
 **Scope.** Tuned for formal logic. The Track B suite reported here does not cover code generation
 or tool use, and no claim is made about either. Granite's base tool-calling and document-grounded
@@ -343,7 +355,8 @@ evaluated.
   backend. `musr` is the mean of the murder, object and team splits.
 * Controlled decode bench: one idle H100 per model, 128 shared prompts, `ignore_eos`, hard 512-token
   cap, engine initialisation excluded from the rate.
-* All columns are scored on the same sampled rows within each track.
+* All models are scored on the same sampled rows within each track. The TwIL and external-model
+  figures are those published on the TwIL-LM3 and earlier TwIL-LM2 cards.
 
 Track B is sampled at 300 examples per dataset for compute reasons. Absolute scores can shift on
 the full sets, but the comparative ordering across models is expected to be stable.
@@ -351,11 +364,11 @@ the full sets, but the comparative ordering across models is expected to be stab
 ## Relationship to TwIL-LM
 
 **TwIL-LM3** ([`webAI-Official/TwIL-LM3`](https://huggingface.co/webAI-Official/TwIL-LM3)) is the
-SmolLM3-3B-based member of the family. It is stronger on held-out benchmarks and on `lean_critic`;
-this model matches its in-domain gate at a smaller size and decodes 1.57x faster.
+SmolLM3-3B-based member of the family. It is stronger on held-out benchmarks, on strict-7 and on
+`lean_critic`; this model is 0.004 behind it on the gate at a smaller size and decodes 1.57x faster.
 
 The name **TwIL-LM2** was previously used for a SmolLM2-1.7B-Instruct-based model. It is the
-semantic-parsing specialist in the tables above, and the weakest of these models on held-out
+semantic-parsing specialist in the tables above, and the weakest of the TwIL models on held-out
 benchmarks. This repository is a different model — Granite-based, 2.5B — that carries the TwIL-LM2
 name; the SmolLM2-1.7B-based figures above are that model's published numbers, included as a
 reference point and not as this model's results.