Model card: UK CoM/CCTS attribution in structured metadata; win/loss by quartile; comparator roster; drop step count

README.md

@@ -79,6 +79,10 @@ line, and it is released as a research artifact.
 
 **Result: 28 wins, 16 losses, 50 mid-pool.**
 
+📖 **Every test is defined in [Test definitions](#test-definitions) below** — what it measures, on what
+data, and which direction is better. The tier codes (T1, T1b, T2 …) used in the tables are all
+explained there.
+
 Every comparator was trained on **1.3–153× more data** than KOS-V5 (0.3–36T tokens vs our 0.235T).
 
 ### The comparator pool (16 external models + KOS-V5 = 17)
@@ -110,6 +114,8 @@ Every comparator was trained on **1.3–153× more data** than KOS-V5 (0.3–36T
 
 ### Wins — top quartile (28)
 
+*Tier codes (T1, T2 …) are defined in [Test definitions](#test-definitions).*
+
 | Metric | Value | Rank |
 |---|---|---|
 | **BPB, 5-corpus mean** (held-out medical text) | **0.4635** | **1 / 17** |
@@ -138,6 +144,8 @@ placements are therefore conservative, not inflated.
 
 ### Losses — bottom quartile (16)
 
+*Tier codes are defined in [Test definitions](#test-definitions).*
+
 | Metric | Value | Rank |
 |---|---|---|
 | **MedHALT false-confidence test (T8-FCT)** | **0.0280** | **16 / 17** — worst placement |
@@ -156,6 +164,8 @@ are MedHALT-FCT, two general-domain reasoning sets, one HoC probe and one RadGra
 
 ### Mid-pool — neither a win nor a loss (50)
 
+*Tier codes are defined in [Test definitions](#test-definitions).*
+
 Reported because they are frequently mistaken for one or the other:
 
 | Metric | Value | Rank |
@@ -171,6 +181,137 @@ above KOS-V4. It is a post-training problem, not a token-budget one.
 
 ---
 
+## Test definitions
+
+What each tier actually measures. **"dir"** is the direction that is better. Every metric is ranked
+against the same 17-model pool unless marked *reference-only*.
+
+### Block A — general LM diagnostics (58 metrics)
+
+#### T1 · Bits-per-byte on held-out medical text — *the headline* (dir: lower)
+Negative log-likelihood divided by **UTF-8 bytes**, not tokens — so it is **tokenizer-agnostic** and
+directly comparable across models with different vocabularies. Measured on **five held-out medical
+corpora**, none of which appear in training: `radiology_reports`, `chest_xray_reports`,
+`clinical_narratives`, `biomedical_literature`, `medical_textbooks`, plus their mean.
+*Caveat we state against ourselves:* a model distilled from a larger teacher can print lower BPB
+regardless of true quality, which **flatters the trillion-token externals, not a from-scratch model** —
+so our BPB placements are conservative.
+
+#### T1b · Tokenizer statistics (dir: STRR higher; fragment & tokens-per-byte lower)
+- **STRR** (*Single-Token Rate Ratio*) — the fraction of medical terms the tokenizer encodes as **one
+  token**, over a union vocabulary of **21,978 medical terms** across the 5 domains. Reported per
+  domain, plus a macro mean and a union-weighted aggregate.
+- **fragment_score** — average number of subword pieces per medical term.
+- **tokens_per_byte** — raw compression efficiency (how compactly the tokenizer encodes text).
+
+#### T2 · Attention pathology (dir: lower for sinks/collapse; higher for min-entropy)
+Per-(layer, head) attention diagnostics over medical text:
+- **collapsed_head_frac** — fraction of heads whose attention has degenerated to a near-constant pattern.
+- **bos_sink_mass** — attention mass dumped on the first token (the classic "attention sink").
+- **layers_with_sink** — fraction of layers exhibiting a sink.
+- **entropy_norm_min** — the lowest normalised attention entropy in the model (higher = no head has
+  collapsed to a single position).
+
+#### T3 · Representation flow (dir: lens accuracy higher; dead-neuron lower)
+- **peak_lens / final_lens accuracy** — logit-lens probe accuracy: how well the correct token can be
+  read out of intermediate layers (peak across depth, and at the final layer).
+- **lens_AUC** — area under the depth-vs-accuracy curve.
+- **max_dead_neuron_frac** — the worst layer's fraction of neurons that never activate.
+
+#### T4 · Representation geometry (dir: RankMe higher; isotropy lower = less anisotropic)
+Measured at **5 proportional depths**:
+- **RankMe** — the *effective rank* (soft rank of the singular-value spectrum) of the hidden
+  representations. Higher = the model uses more of its representational space rather than collapsing
+  into a low-dimensional cone.
+- **isotropy** — directional uniformity of the embedding cloud.
+
+#### T5 · WeightWatcher power-law α — structural maturity
+Fits a power law to each layer's weight spectrum. **α ∈ [2, 6] is the healthy band**; α > 6 means
+under-trained, α < 2 over-trained.
+- **safe_layer_ratio** (dir: higher) — fraction of layers inside the healthy band.
+- **under_ratio** (dir: lower) — fraction of layers that are under-trained.
+- **median_α / min_α** — **reference-only, deliberately unranked**: α is a *target band*, not a
+  monotone quantity, so "more is better" is meaningless. (Ranking it higher-is-better would literally
+  reward being under-trained — and would have *improved* our reported result from rank 15 to rank 3. We
+  declined to take the favourable reading of an ill-defined metric.)
+
+#### T9 · Calibration (dir: lower)
+**Expected Calibration Error (ECE)**, averaged over **15 multiple-choice calibration tasks**: the gap
+between the model's confidence and its actual accuracy. Low ECE = the model's confidence means something.
+
+#### T10 · Context-length scaling + needle-in-a-haystack
+- **Bucket BPB** (dir: lower) — BPB measured at {L/8, L/4, L/2, L} where L = min(model's max position,
+  4096), so every model is judged inside its own usable window. A separate KOS-only extension probes
+  the full **32,768**-token window.
+- **Needle retrieval** (dir: higher) — a numeric medical fact is hidden at depth **0.1 / 0.5 / 0.9**
+  inside a long context and must be retrieved, at fixed lengths **512 / 1,024 / 2,048 / 4,096** tokens.
+  Reported per length, per depth, and as a 3-depth mean.
+
+#### T11 · Demographic bias (dir: disparity lower; frac_stereotyped closer to 0.5)
+**CrowS-Pairs**: for each minimally-different sentence pair (stereotyping vs anti-stereotyping), which
+does the model prefer? `|disparity|` measures how strongly it prefers one; `frac_stereotyped` is the
+fraction of pairs where the stereotyping sentence wins (0.5 = no preference).
+
+### Block B — medical knowledge and entity extraction (32 metrics)
+
+#### T6 · Closed-book multiple choice (dir: higher)
+**16 log-likelihood MCQ tasks**, scored with the standard lm-evaluation-harness (no chain-of-thought,
+no prompting tricks — this is a base model):
+- *Medical specialist:* **MedQA** (4-option, USMLE-style), **MedMCQA**, **PubMedQA**, and the **8 MMLU
+  medical subjects** (anatomy, clinical knowledge, college biology, college medicine, high-school
+  biology, medical genetics, nutrition, professional medicine, virology).
+- *Augmented / regional / expert:* **MedXpertQA-Text**, **AfriMed-QA**.
+- *General-reasoning controls:* **PIQA**, **HellaSwag**.
+
+#### T6b · MedThink-LL (dir: rank-1 higher; margin lower)
+A PubMedQA-substitute: the correct answer must be ranked above **4 distractors** by log-likelihood.
+**rank1_acc** = how often it is; **mean_logprob_margin** = the confidence gap.
+
+#### T7 · BLURB linear probes (dir: higher)
+A **linear probe** is trained on the frozen model's representations — this reads the *semantic geometry*
+of the upper layers, independent of any generation ability:
+- **BC5CDR-Chem** — chemical NER (macro / micro F1).
+- **NCBI-Disease** — disease NER (macro / micro F1).
+- **BIOSSES** — biomedical sentence-similarity (Pearson / Spearman).
+- **HoC** — Hallmarks-of-Cancer document classification (micro / macro / samples F1).
+- **BLURB mean** — the headline: mean of BC5CDR-macro, NCBI-macro, BIOSSES-Spearman, HoC-micro.
+
+#### T8 · Med-HALT — hallucination resistance (dir: higher)
+- **FCT** (*false-confidence test*) — the question's **premise is wrong**; the model should decline.
+  A base model has no refusal prior, which is exactly why this is our worst placement.
+- **NOTA** (*none-of-the-above*) — the correct answer is "none of the above".
+- **FQT** (*fake-question test*) — the question is fabricated; it should be rejected.
+- **PMID2Title** — recall a paper's title from its PubMed ID (a memorisation probe; **degenerate** —
+  every model in the pool scores 0.0, so it is excluded from the tally).
+
+### Block C — radiology-specific (6 metrics)
+
+#### DR.1 / DR.2 / DR.3 · RadGraph entity + relation extraction (dir: higher)
+Structured extraction of clinical entities and their relations from radiology reports, scored as
+macro-F1 and micro-F1:
+- **DR.1 — RadGraph-XL**: chest CT + abdomen-pelvis CT + brain MRI.
+- **DR.2 — RadGraph-CXR**: chest x-ray (50 MIMIC + 50 CheXpert).
+- **DR.3 — RadGraph2-CXR**: adds the CHAN (change) relation class.
+
+#### DPM.1 · PMR-Q&A (reference-only)
+English physiotherapy / rehabilitation Q&A, rank-1 accuracy. **Unranked** — too few comparators have a
+score for it to be a meaningful ranking.
+
+### How ranks are computed
+
+- **Pool = KOS-V5 + the 16 external models = 17.** KOS-V4 is *excluded* from the pool — it is our own
+  prior model, and is reported separately in its own head-to-head below.
+- Each metric is ranked **in its own correct direction** (lower-is-better for BPB, ECE, sinks,
+  fragmentation; higher-is-better for accuracy, F1, RankMe, STRR).
+- **Denominators vary** where a comparator has no score for a metric: Gemma-2-9B has no T10 result and
+  BioMedLM is architecturally capped at 1,024 tokens (so it is skipped in the longer needle tables).
+  That is why some ranks read "of 16" or "of 12" rather than "of 17" — the rank is always taken over
+  whichever models actually have that metric.
+- **Two metrics are reference-only** (the WeightWatcher α pair) and **one is degenerate** (PMID2Title,
+  where all 17 models score 0.0), leaving **93 metrics in the tally**.
+
+---
+
 ## KOS-V5 vs KOS-V4 — a corpus trade-off, now settled
 
 Both models are ours, both are finished, and both were run on the identical harness. **KOS-V5 is
@@ -254,12 +395,12 @@ neither quantify the cost nor claim there is none. The tokenizer is baked into t
 embedding table, so it is not fixable by fine-tuning — removing it requires retraining with a
 digit-splitting pre-tokenizer. Treat any numeric medical task on this model with that in mind.
 
-**2. Low medical single-token rate (STRR — bottom quartile, rank 15/17).** Deliberate: BBPE spends its
+**2. Low medical single-token rate ([STRR](#t1b--tokenizer-statistics-dir-strr-higher-fragment--tokens-per-byte-lower) — bottom quartile, rank 15/17).** Deliberate: BBPE spends its
 32k vocabulary on byte-level compression rather than memorising whole medical terms. It single-tokens
 only ~19% of radiology and ~23% of chest-xray terms. The trade appears sound — tokens-per-byte is a
 top-quartile 4/17 and BPB is 1/17 — but it is a trade, and it is the same trade as the digit issue.
 
-**3. MedHALT-FCT is the worst placement in the model (rank 16/17).** A base model with no refusal prior
+**3. MedHALT-FCT ([false-confidence test](#t8--med-halt--hallucination-resistance-dir-higher)) is the worst placement in the model (rank 16/17).** A base model with no refusal prior
 has essentially no mechanism for declining a question whose premise is false, and KOS-V5 answers
 anyway. This is the axis a grounded medical model most needs to improve, and it is a target for the
 instruction/RL stage rather than for more pretraining.