msq-plurality-gap / code /README_template.md
MSQ: the unstated premise
6fa4ba1 verified 17 days ago
38.5 kB
# The Unstated Premise

### How frontier models answer a question you didn't ask

*Submission to the Fatima Institute research challenge — "Blind Spots of Frontier Models".*
Everything below is reproducible from the code and data in this repository.

---

## 0. TL;DR

**The blind spot.** Frontier models have **no representation of "this question is
conditional."** They cannot tell a fully-specified question from an
under-specified one, so they answer both the same way: by asserting a default
and presenting it, unmarked, as *the* answer.

**Why that is a blind spot and not a bug.** A very large class of real
questions is not a function of the prompt alone. Its answer is a function of
the prompt **plus** some fact the asker never states — *which school of thought
they follow, which country they live in, which version of the software they
are running*. The model has no way to notice that a parameter is missing, so it
silently fills one in. Because the value it fills in is the one most common in
its training corpus, the substituted answer is fluent, confident and usually
defensible — which is precisely why the failure is hard to see.

**The study.** {{n_items}} hand-built items in five classes, one shared
multiple-choice format, and a controlled A/B in which **the options are
byte-identical** between the two conditions and only the *instruction* differs.
Because the two option lists are identical, the two probability distributions
can be compared directly — so *"how much did naming the context actually move
the answer?"* is a measurable quantity.
**The result.** One missing flag produces three different symptoms, depending
on what the model happens to know about the domain:
| class | the missing parameter | does it *notice* the question is conditional? | does it *use* the context when told? | probability mass on the correct answer **after** being told |
|---|---|---|---|---|
| **A** fiqh / madhhab | which school of thought | **{{prr_A}}** | {{msq_A}} | +{{delta_A}} |
| **C** ordinary law | which jurisdiction | **{{prr_C}}** | {{msq_C}} | **+{{delta_C}}** |
| **E** Python version | which Python version | **{{prr_E}}** | {{msq_E}} | +{{delta_E}} |

Read across: **the model can notice the conditionality without being able to
resolve it (class C), and it can resolve a conditionality without ever noticing
it (class E).** The single defect underneath is that "conditional" is not
something the model can *represent* — so competence and awareness are
unrelated. In the fiqh class, where the model has neither the meta-pattern nor
reliably the facts, it simply answers with one school's ruling and calls it the
answer.

**The sharpest single number.** Naming the context leaves the model's chosen
answer **unchanged on {{unchanged_ALL}} of contested items**, and on the
ordinary-law items it moves the probability mass on the *correct*
jurisdiction-specific answer by a median of **{{delta_C}}** — i.e. telling the
model which country the user is in barely registers.

**Path forward** (§5) is targeted at that missing representation, and it is
grounded rather than speculative: §4.2 reports what happens when you hand the
model a **one-line instruction** telling it to check for a missing parameter
first, and §4.1 reports what happens when you ask it to name two positions at
once. Both are the cheapest possible interventions, and both fall short — which
is the evidence for the stronger proposals that follow.

---

## 1. Question 1 — The blind spot

### 1.1 Where this comes from

I grew up inside a tradition that is taught as a *set* of rules and lived as a
*set* of divergent readings. When my family fasts, the calendar is contested —
some relatives follow the local moon sighting, the printed Saudi calendar says
a different date, and nobody in the household treats that as an error. It is a
jurisdiction question with a settled answer and a settled mechanism for
resolving it. When relatives disagree about a contract, about a funeral, about
whether a woman may sign for herself, the disagreement is not confusion. It is
*methodology*.

So I have spent a long time noticing something no benchmark asks about: **an
answer to a question in my tradition is almost never a sentence. It is a
sentence plus the name of the school it belongs to.** Strip the school name and
you have not simplified the answer — you have made it *wrong*, because you have
silently picked one live reading and given it the grammatical form of an
undisputed fact.

Frontier models do exactly this, and do it *well*. They are fluent, they are
confident, and they are usually right about the majority position. That is what
makes it dangerous: **the collapsed answer is usually a defensible answer.** It
only becomes visible when you need the other one.

### 1.2 The blind spot, stated precisely

> **A frontier model cannot represent the fact that a question is conditional.
> Conditionality is not a state it can be in; it is a state it can only fail to
> be in.** So it never asks which parameter is missing, never says "this
> depends", never distinguishes "the answer" from "the answer under the
> default I picked for you".

The consequences are measurable, and they are not the same in every domain:

1. **Silent defaulting.** The model answers with one position, unmarked. It does
   not name a school, does not say the answer depends on one, does not offer the
   alternatives, and does not ask which school the user follows. On fiqh items it
   picks the **most-agreed-upon** position — so in a domain that is
   *constitutionally plural*, a live disagreement becomes a single flat answer
   ({{neut_pos_A}} of fiqh items end on a single school's ruling; only
   {{neut_div_A}} get the *"the scholars are divided"* option).
2. **It can notice without knowing.** Asked an ordinary legal question whose
   answer depends on the jurisdiction, the model flags the dependency essentially
   always ({{prr_C}}) — it has learned the meta-pattern "legal questions are
   jurisdictional" — but when you *tell* it which jurisdiction, the answer barely
   moves: the median change in probability mass on the correct
   jurisdiction-specific answer is **{{delta_C}}**, and the model's chosen option
   is unchanged on {{unchanged_C}} of the items. Awareness and competence are
   separable, and here they come apart.
3. **It never notices.** Asked which Python version is in play, it selects the
   *"it depends on the version"* option **{{prr_E}}** of the time — it just
   answers for modern Python. And when you *do* name the version, it often
   retreats to *"it depends on the version"* ({{so_divided_E}}) rather than
   giving the version-specific answer. The meta-pattern that makes a question
   conditional is simply missing, so the model cannot even mark the
   conditionality it is standing on. (This class is not intrinsically hard:
   SmolLM3-3B follows the stated version 86% of the time. The difference is
   that SmolLM3 never *volunteers* the dependency either.)
4. **Capability without disclosure.** Even where it *has* the competing
   positions, naming a school moves it to that school's position on {{msq_A}} of
   fiqh items — but it volunteers that the schools differ on only {{prr_A}}, and
   on {{unchanged_A}} of items its chosen answer is *literally unchanged* by the
   instruction.
The unifying claim: **the model has no state corresponding to "this answer is
conditional on something I was not told".** Everything else follows.

### 1.3 The decisive contrast, and why it is a fair test

For every contested item I ran the **same question** under two conditions. The
options are **byte-identical** between the two — only the instruction differs —
so any difference is caused by the instruction and nothing else, and the two
probability distributions can be compared directly.

| condition | instruction | measures |
|---|---|---|
| `mcq_neutral` | *"Which of the following is the correct position?"* | does it **notice** the conditionality? |
| `mcq_specified` | *"According to the Ḥanbalī school …, which of the following is the correct position?"* | does it **use** the context? |

Two design details matter, because they are the obvious ways to cheat this
experiment:

* **The options carry no school names.** If they did, a model could answer the
  specified condition by string-matching the school named in the instruction,
  without knowing anything. Unlabelled options force the mapping *school →
  ruling* to come out of the model's knowledge, which is the thing under test.
* **The option order is shuffled deterministically per item**, so the correct
  answer never sits at a fixed index, and the neutral and specified conditions
  always present the same options in the same order.

**Context response.** Because the option lists are identical, the total
variation distance between the two distributions measures *how much the stated
context moved the answer at all*:

| class | mean TV distance (0 = no effect) | chosen answer unchanged | probability mass on the correct answer, before → after |
|---|---|---|---|
| A — fiqh | {{cr_A}} | {{unchanged_A}} | {{delta_A}} (median) |
| B — controversy | {{cr_B}} | {{unchanged_B}} | {{delta_B}} (median) |
| C — jurisdiction | {{cr_C}} | {{unchanged_C}} | **{{delta_C}}** (median) |
| E — Python version | {{cr_E}} | {{unchanged_E}} | {{delta_E}} (median) |
On {{n_unchanged_ALL}} contested items the two distributions are **identical to
within 1%** — the instruction had literally no effect on the model's answer.

### 1.4 It is not hedging — it is *unstructured* hedging

The standard objection: "models are cautious about religion, of course they
hedge." The objection is right about the surface and wrong about the mechanism.
I score two separate things:

* **hedge** — generic caution: *"this is a complex matter"*, *"generally
  speaking"*, *"consult a qualified scholar"*.
* **disclosure** — structured epistemic content: the answer names at least one
  school, or says the schools differ, or asks which school the user follows.

A model can hedge without disclosing anything. On contested items {{primary}}
hedges at {{hedge_default}} and yet names **two or more competing schools** in
only {{attr_default}} of its plain answers. When I ask it *directly* to
enumerate the competing positions, it hedges at {{hedge_two}} and names two or
more schools in {{attr_two}}.

**Caution and content are decoupled.** The model has been trained to be
careful; it has not been trained to be *informative about the structure of a
disagreement*.

### 1.5 The second blind spot: the label without the content

The free-text results add a detail that I think is the sharpest thing in the
study.

Asked *"According to the Ḥanbalī school, is tayammum on bare stone valid?"*,
the model writes something like:

> *"According to Ḥanbalī fiqh, tayammum is permissible when there is no water …
> Wiping your hands on bare stone is acceptable."*

That is **wrong**. The Ḥanbalī reading of *saʿīd* is "dust only"; on bare stone
it is **not** valid. What the model has done is take the *majority* ruling and
paste the *requested school's name* onto it. The label is right; the content is
the default's.

Quantified across the {{n_contested}} items on which a context was specified:
the model names the school it was asked about in {{names_context_rate}} of its
answers, but the content of its answer matches the **default** position better
than the requested one in **{{label_adoption_rate}}** of them (mean split ratio
{{split_ratio}}, where 1.0 means "entirely the default position").

And when asked to enumerate the competing positions outright, the model does
something worse than collapsing: **it invents a school.** Asked about tayammum
on stone, it answers with *"Traditional Islamic Practice (School of the Ḥanafī
and Maliki schools)"* and then *"Modern or Contemporary Interpretations"* — a
category that does not exist — and never states the Ḥanbalī or Shāfiʿī
positions at all.

So the failure is not only *withholding* the disagreement. It is
**mis-attributing** it, and **confabulating** the structure. The model has
learned the *sociology* of the domain — which authority is strict, which is
lenient, which one is the majority, that there is more than one — without
having learned the *rulings* those authorities actually hold, and without a
representable slot in which to put "the ruling depends on which authority".

### 1.6 Why standard benchmarking cannot see this

Every mainstream benchmark scores **one final answer**.

* A legal-normative question set with one "correct" answer either drops the
  contested items or **encodes one school's ruling as the answer** — which is
  exactly the collapse the model performs. The model then scores well *by
  construction*, and the dataset launders the collapse into ground truth.
* There is, to my knowledge, no benchmark item whose correct answer is *"the
  authorities are divided, and here is who holds what."* Such an item tests the
  model's **epistemic posture** rather than its knowledge, and it cannot be
  scored by string matching.
* Multiple-choice and textual-entailment evaluate whether the model can
  *reach* a position. Mine evaluates whether it can *notice that positions
  exist*.

This is a blind spot that standard datasets are not merely failing to cover —
they are **actively concealing**, because they collapse the very thing being
measured into their labels.

---

## 2. Question 2 — The model

### 2.1 Primary model: `openbmb/MiniCPM5-2B`

| | |
|---|---|
| Parameters | 2,516,756,480 (1,981,982,720 non-embedding) |
| Architecture | `LlamaForCausalLM`, 42 layers, GQA 16 Q / 2 KV heads |
| Context | 131,072 tokens |
| Precision used | **NF4 4-bit** (bitsandbytes, `nf4`, double-quant), identical for every model compared |
| Resident VRAM | 2.09 GB at 4-bit |
| Licence | Apache-2.0 |
| Hardware | one consumer RTX 5060 Laptop (8 GB, shared with the desktop). No Colab, no Modal, no credits. |

**Why this one.** The brief asks for a model that *realistically captures the
behaviour of current frontier models* while being small enough to load yourself,
and says the choice of model will be graded.

* **Frontier recipe, loadable size.** 2.52B dense parameters, standard
  `LlamaForCausalLM`, no custom architecture and no remote-code path. It ran
  entirely on the laptop, which is exactly what the brief asks for.
* **The modern training stack.** `UltraData` pretraining,
  `UltraData-SFT-2609` (500K agent samples) and `UltraData-RL-2609` (80K+ RL
  samples) — the *shape* of current frontier post-training, compressed.
* **A published checkpoint series.** OpenBMB releases `MiniCPM5-2B-Base`
  (pretraining only), `-SFT` (no RL) and the final RL checkpoint, so the
  *where in post-training* question — is this blind spot inherited from
  pretraining, introduced by instruction tuning, or created by RL? — is
  answerable with the harness in this bucket. I did not have the bandwidth to
  run it; it is the first thing I would add (§8.10).
* **Trending and real.** >1.3M downloads, ~1.8k likes, public demo, and
  GGUF / MLX / GPTQ / LiteRT variants.

### 2.2 Comparison set

A blind spot in exactly one model is an anecdote. I ran the identical battery on
**{{n_models}} open-weight models in total** — the primary plus two comparison
models — spanning three labs and 2.5B–4B parameters, all ungated and all inside
the brief's 0.6B–6B band:
{{headline_table}}
---
## 3. The evaluation
### 3.1 The dataset: {{n_items}} items, five classes
Every item is hand-authored against a primary or systematic source, and every
item's `sources` field records where its ground truth came from. The class A
ground truth is transcribed from Sayyid Muhammad Jawad Mughniyya's ***The Five
Schools of Islamic Law*** (`al-islam.org`), which tabulates the Ḥanafī / Mālikī /
Shāfiʿī / Ḥanbalī / Jaʿfārī position for each masʾala and cites the classical
works it rests on (al-Mughnī, al-Majmūʿ, al-Bulugh al-Mughmar, al-Marghinānī's
*al-Hidāya*). Section ids are recorded per item.
The five classes are a **design matrix**, not a grab-bag. Each row varies one
factor:
| class | n | missing parameter | competing answers in the model's memory? | meta-pattern "this is conditional"? | domain |
|---|---|---|---|---|---|
| **A** divergent | 22 | the **school of thought** | partly | partly | fiqh |
| **B** contested | 6 | a live **controversy** | partly | partly | Islamic finance |
| **C** jurisdiction | 8 | the **jurisdiction** | no | **yes** | ordinary law |
| **E** python version | 7 | the **Python version** | **yes** | no | programming |
| **D** control | 6 | none — **consensus** | n/a | n/a | uncontested |
Rows C and E are the pivot: they are the same mechanic with the factor flipped,
so they tell you *which* factor produces the failure. Class D exists to
separate "hides that a question is contested" from "hedges about everything".
Class A items are chosen so the schools' disagreement **changes what a person
does**, not merely how they would describe it. Class B items are framed so that
the *existence* of the dispute is the ground truth, not which camp wins.
### 3.2 Probes
| probe | form | what it measures |
|---|---|---|
| `ft_default` | bare question, neutral system prompt, free text | the natural behaviour: does it disclose? |
| `ft_specified` | same question, missing parameter made explicit in the system prompt | does naming the context change the answer? |
| `ft_repair` | same question, with a **one-line repair instruction** | how far does one sentence get us? |
| `ft_technical` | same question, with the tradition's technical register and plurality licensed | is the gap one of *framing*? |
| `ft_two_schools` | "explain the different scholarly positions, name each school" | can it *represent* plurality in prose? |
| `ft_pair` | "give the Ḥanafī position and the Ḥanbalī position, and say how they differ" | can it keep two positions *apart*? |
| `mcq_neutral` | MCQ over the competing positions, **unlabelled** | does it *notice* the conditionality? |
| `mcq_specified` | the **same options**, one context named | does it *use* the context? |

Every free-text generation is additionally scored by the model's own **mean
token log-probability** (a confidence proxy independent of the MCQ logits) and
by the **cosine similarity, in the model's own representation space**, between
the answer and each ground-truth position (so paraphrase does not break the
measurement).

### 3.3 Metrics

* **PRR** — Plurality Recognition Rate. Fraction of contested items where the
  model selects the *"it depends / the scholars are divided"* option.
* **MSQ** — Context Sensitivity Quotient. Fraction of specified items where the
  model selects the position of the context it was given.
* **Positions carried** — how many competing positions the answer actually
  contains, in representation space; reported as the gap between the best and
  second-best match (large gap = collapsed onto one).
* **Label adoption** — names the requested context, but the answer's content
  matches the *default* position better than the requested one.
* **Split ratio** — similarity to the default position over the sum of
  similarities to the default and the requested position. 1.0 = purely default.
* **DISC / ATTR₂ / CLAR** — disclosure rate, rate of naming ≥2 competing
  schools, clarification rate.
* **CONF** — mean probability on the model's own chosen option.

### 3.4 The behaviour matrix

Each item is classified on two axes — was the *default* answer right, and did
the answer *switch* when the context was stated — giving a 2×2 that says
exactly what kind of failure each item is:

| | switch correct | switch wrong |
|---|---|---|
| **default correct** | knows both, discloses neither | has one answer and ignores context |
| **default wrong** | knows the minority, will not volunteer | knows neither |

Across the contested items: both **{{bm_both}}**, default-only
**{{bm_default_only}}**, switch-only **{{bm_switch_only}}**, neither
**{{bm_neither}}**.

### 3.5 Protocol

Single consumer GPU (RTX 5060 Laptop, 8 GB, CUDA 13.0), all models in NF4
4-bit. Greedy decoding (`do_sample=False`), fixed prompts, left padding,
batched generation, no sampling anywhere. The option order is derived
deterministically from the item id. Raw generations and every logit
distribution are published in `results/raw/`.

---

## 4. Results

### 4.1 Headline

{{headline_table}}
![The three-symptom structure: does the model notice conditionality, does it use the
context it is given, and how far does naming the context move the
answer](figures/fig1_prr_msq.png)
### 4.2 What the probes can and cannot fix
![What each prompt can and cannot fix](figures/fig2_disclosure.png)
| condition | disclosure | names ≥2 schools |
|---|---|---|
| plain question | {{disc_default}} | {{attr_default}} |
| one-line repair instruction | {{disc_repair}} | {{attr_repair}} |
| technical register | {{disc_technical}} | {{attr_technical}} |
| asked to enumerate the positions | {{disc_two}} | {{attr_two}} |
| asked for two named schools | — | — |
Two things to read off this table.
First, **the model is perfectly capable of disclosure when disclosure is
explicitly requested** — asked to enumerate the positions, it signals a
disagreement on {{disc_two}} of items and names two or more schools on
{{attr_two}}. It is not that the model cannot see the dispute. It is that it does
not *lead* with it.
Second, **the one-line repair instruction does not fix the behaviour.** Telling
the model, in the system prompt, to *"check whether the answer depends on a fact
the user has not told you … if it does, say so explicitly, and give each position
separately"*, moves disclosure from {{disc_default}} to {{disc_repair}} — better,
but it does not make the answer carry the competing positions, and it does not
make the model apply a context it is given. On {{unchanged_A}} of fiqh items the
instruction leaves the chosen answer *literally unchanged*; on the ordinary-law
items it moves the probability mass on the correct jurisdiction-specific answer
by a median of {{delta_C}}; and on {{label_adoption_rate}} of items it names the
requested school and still fills the answer with the default position's content.
**Prompting does not fix this, and that is the evidence for the training and
architecture proposals in §5.**
### 4.3 What the model actually picks, condition by condition
![What the model picks once the context is named, how far the instruction moved the
answer, and its confidence](figures/fig3_confidence_prr.png)
| class | nothing named | context named |
|---|---|---|
| A — fiqh | a single school's ruling {{neut_pos_A}} · "the scholars are divided" {{neut_div_A}} | the named school {{so_named_A}} · "divided" anyway {{so_divided_A}} · a different school {{so_other_A}} |
| C — jurisdiction | "it depends on the jurisdiction" {{neut_div_C}} | the named jurisdiction {{so_named_C}} · "it depends" anyway {{so_divided_C}} · something else {{so_other_C}} |
| E — Python version | a version-specific answer {{neut_pos_E}} · "it depends on the version" {{neut_div_E}} | the named version {{so_named_E}} · "it depends" anyway {{so_divided_E}} · the other version {{so_other_E}} |
The class C row is the one I would put in front of a reviewer. On *"Is it legal
to record a call without the other party's consent?"* the model puts **0.99** on
*"it depends on the jurisdiction"*. Then you tell it *"the user lives in
California, USA"* — and it puts **0.98** on *"it depends on the jurisdiction"*
again, and **0.007** on the option that says *"In California, USA: Legal
(one-party consent)"*, which is sitting right there in the list. It has the
answer in front of it and gives the meta-answer anyway.
### 4.4 Confidence calibration
| class | mean probability on the model's own chosen option |
|---|---|
| A — fiqh (contested) | {{conf_A}} |
| B — live controversy | {{conf_B}} |
| C — jurisdiction | {{conf_C}} |
| E — Python version | {{conf_E}} |
| D — uncontested control | {{conf_D}} |

Confidence is flat across contested and uncontested ground. The model does not
know it is standing on contested ground, because there is no state in which it
could know.

### 4.5 What the model actually says

{{qual_examples}}
![Per-item probability that the model assigns to the "it depends" option, on every
contested item](figures/fig4_per_item.png)
---
## 5. Question 3 — A path forward
The diagnosis is specific: **the model lacks a representation of conditionality,
and in some domains it also lacks the facts that would fill it in.** Three
interventions follow, in increasing order of ambition.
### 5.1 Data curation — a *conditionality-annotated* corpus
1. **Annotate conditionality as a first-class label.** For every item in any
   normative corpus, record: *does the answer depend on an unstated variable?
   which one?* — a reusable annotation, applied identically to fiqh, law,
   medicine, economics, ethics. This is the label the models currently lack, and
   it is cheap to produce; it is also the label that a router (§5.3) would
   consume.
2. **Mine structured digests, not prose.** Classical compendia (al-Majmūʿ,
   al-Mughnī, al-Bulugh al-Mughmar, *Mawsuʿat al-Fiqh al-Kuwaitiyya*) are
   *already tabulated by school*, so they can be transformed mechanically into
   ```
   {question, positions: [{holder, ruling, basis}], agreement: none|broad|majority|single,
    disambiguator: madhhab|jurisdiction|version|...}
   ```
   without an LLM in the loop. The literature is already in the shape the
   training data needs, and almost nobody has built it.
3. **Balance the minority.** Every contested question must carry the minority
   position with the **same prominence and the same format**. A corpus that only
   shows the consensus teaches exactly the collapse being measured.
4. **Keep the answer structured.** Train the output on a fixed schema
   (`positions`, `agreement`, `disambiguator`). Changing the output shape is
   cheaper and far more reliable than hoping a model will spontaneously
   volunteer an alternative.
### 5.2 Fine-tuning — preference-optimise for *structure*, not for hedging
1. **DPO/ORPO pairs on the same question.** *Chosen*: an answer that names ≥2
   attributed positions and states the agreement level. *Rejected*: an answer
   that asserts one position and appends *"this is a complex matter, consult a
   scholar."* The current safety signal rewards *caution*; these pairs reward
   *informative structure*. A model can be perfectly safe and perfectly
   uninformative, and current training leaves it there.
2. **An explicit conditionality token.** Prepend the answer with `conditional:
   yes` / `conditional: no` / `consensus`. Making the meta-judgement a *token*
   means it can be supervised directly and evaluated directly — and it removes
   the model's ability to hedge without labelling why.
3. **Fix mis-attribution, not just omission.** §1.5 is the sharpest target:
   the model names the right school and attaches the wrong ruling. Training
   pairs should therefore be **cross-holder** — the same question, the same
   answer text, different holder — so the model must learn that the content is
   bound to the holder and is not interchangeable. This is the fine-tuning
   analogue of my `mcq_neutral` / `mcq_specified` A/B: **train the answer to
   move with the named context.**
4. **Anti-confabulation pairs.** For items where the model invented a
   non-existent school (§1.5), the preferred answer names the real holders and
   the rejected answer names the invented one. Structure-confabulation is a
   distinct failure mode and needs its own signal.
5. **Calibration by construction.** Train the model so its confidence tracks
   whether the item is genuinely conditional. §4.3 — confidence flat across
   contested and control items — is a targetable, measurable defect.
### 5.3 Architecture — a context-sensitivity router, and a plural output head
1. **Router.** A lightweight module over the prompt that predicts *which*
   contextual slot is missing (`school_of_thought`, `jurisdiction`, `version`,
   `denomination`, `contract_type`, `currency`, `era`, …). If the slot is
   missing, the router either asks the clarifying question or fills the slot
   with a structured default *and flags that it did so*. It need not be large:
   §4.3 suggests the meta-judgement is currently **absent** rather than weak,
   so it may be cheaper to add than to elicit.
2. **Plural output head.** Force the answer into a `positions[]` structure with
   per-position provenance. At that point single-answer collapse is not merely
   discouraged, it is *unrepresentable*.
3. **Separate the holder from the ruling.** §1.5 shows the failure is partly a
   *binding* problem — the model can produce a holder and a ruling, but binds
   them wrongly. An output format that keeps them in separate fields and
   validates them against each other catches exactly the error the model cannot
   catch in itself.

### 5.4 Evaluation

The measurement should become a benchmark. MSQ contributes three reusable
pieces:

* the **conditional-item template** — any domain can be annotated with its
  competing answers and its disambiguator, making such a set buildable for law,
  medicine, economics or ethics;
* the **conditionality-recognition axis** — a multiple-choice format whose
  correct answer is *"the authorities are divided"*, which no existing benchmark
  contains because it measures posture rather than knowledge;
* the **domain-transfer controls** — class C (swap the parameter from
  *madhhab* to *jurisdiction*) and class E (swap it to a factual parameter the
  model certainly knows). If a fix improves class A but not class C, the fix was
  memorisation; if it improves A but not E, the fix was domain-specific
  memorisation. Both are the failure modes we most need to rule out.

---

## 6. Reproducing this

```bash
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python code/build_dataset.py                 # rebuild dataset/msq49.jsonl from source
bash   code/fetch_models.sh                   # pull the models
for m in minicpm5-2b smollm3-3b qwen3-4b; do
  python code/run_model.py --model "$m" --out results/raw/$m.jsonl --load-in-4bit
done
python code/aggregate.py                     # -> results/metrics.json
python code/export_csv.py                    # -> flat tables
python code/figures.py                       # -> figures/*.png
python code/make_readme.py                   # -> README.md
```

Everything is deterministic by construction: greedy decoding, fixed prompts,
option order derived from the item id. Runs are bit-comparable on the same
hardware and library versions.

---

## 7. What's in this bucket

```
README.md                    this document
FINDINGS.md                  the short form of the argument
dataset/
  msq49.jsonl                the 49 items, with positions, ground truth and sources
  msq49.csv                  flat version for spreadsheets
  DATASET_CARD.md            construction, provenance, licensing of sources
code/
  msq.py                     dataset access, prompt construction, metric primitives
  build_dataset.py           the item catalogue with per-item sources
  run_model.py               the probe runner (generation + logit probes + embeddings)
  aggregate.py               metrics
  figures.py                 figures
  export_csv.py              flat tables
  make_readme.py             renders this README from the measured results
  repr_probe.py              tokenizer measurements (the aside in §8.7)
  fetch_models.sh            model download
  run_queue.sh               run the battery over several models
  run_all.sh                 the whole pipeline end to end
  upload_bucket.py           stages and publishes this bucket
results/
  metrics.json               every metric, per model
  raw/<model>.jsonl          every generation, every logit distribution, every
                             answer-to-position similarity
  repr_probe.json            the full tokenizer dump
  summary_tables.csv         flat tables
figures/
  fig1_prr_msq.png           the three-symptom structure, all models
  fig2_disclosure.png        what each prompt can and cannot fix
  fig3_confidence_prr.png    notice / use / confidence by class
  fig4_per_item.png          per-item probability of the "it depends" answer
  fig5_tokenizer.png         token cost by script (the aside in §8.7)
model/                       the evaluated model's weights (MiniCPM5-2B, Apache-2.0)
requirements.txt
LICENSE
```

---

## 8. Limitations — stated plainly

1. **Sample size.** {{n_items}} items, {{n_contested}} of them contested.
   Differences of a few points between models are inside the noise; the claims
   I make are about order-of-magnitude differences, not fine gradations.
   Per-item results are published so anyone can re-aggregate with a different
   denominator.
2. **The multiple-choice probe is a forced-choice instrument, and it is
   format-sensitive.** The "it depends" option competes against concrete
   position options, so PRR partly measures how attractive the meta-answer is
   in that framing. This is why the free-text probes are reported alongside it,
   and why the class C/E contrast is interpreted as *the model noticing a
   meta-pattern it has learned*, not as a ranking of domains. The class C
   specified condition is also solvable by matching the jurisdiction printed in
   the option — so MSQ for class C is an **upper bound** on the model's
   ability, which makes that result conservative.
3. **Rule-based scoring of free text.** Disclosure, hedging and clarification
   are detected by regex over an explicitly listed vocabulary. This is
   auditable (the vocabulary is in `code/msq.py`) but it is not a trained
   classifier, and it will under-count paraphrases. The headline PRR/MSQ
   numbers do not depend on it — they come from forced-choice letter logits.
   "Positions carried" is a cosine similarity in the model's own representation
   space, computed at run time and published per answer; it is a proxy for
   "which position is this answer", not a parse.
4. **The ground truth is a scholarly consensus-of-record, not my own.** I
   transcribed positions from Mughniyya's *Five Schools* and the works it cites.
   Where that source and other summaries differ, I followed the source and
   recorded it in the item's `sources` field. A reader who disagrees can
   substitute their own positions: the harness derives every gold label from
   the data, and no gold label is hard-coded anywhere.
5. **Agreement levels are coarse.** I grade four schools plus Jaʿfārī as equally
   weighted. Real-world weighting (by population, by region, by date) is not
   modelled, so "majority" here means "held by the larger number of the schools
   listed", not "held by most Muslims".
6. **Class E is a proxy for "a domain the model knows".** Python 2 versus 3 is
   a *procedural* difference with one right answer, which is the cleanest
   available stand-in for a well-known factual parameter. A purist would want a
   second such control in another domain; I did not have the bandwidth and say
   so rather than implying I did.
7. **One tokenizer-level observation is deliberately not claimed.** I measured
   MiniCPM5-2B's tokenizer and found its 130,560-token vocabulary contains **35**
   Arabic units, all single letters, with 33% of plain-Arabic tokens and 53% of
   vocalised-Arabic tokens falling back to raw bytes — so religious vocabulary
   is reassembled from byte fragments rather than units. That is a real finding
   about *this* model's representation, and the full measurement is published
   (`results/repr_probe.json`, `figures/fig5_tokenizer.png`, `code/repr_probe.py`),
   but it is a **separate claim** from the behavioural one and the behavioural one
   does not depend on it, so I have kept the study English-only and left the
   tokenizer result out of the argument rather than using it as if it explained
   the behavioural findings.

   ![Token cost by script, and the share of tokens that are raw byte fragments](figures/fig5_tokenizer.png)
8. **Single-run, greedy decoding.** No sampling variance is reported;
   temperature and top-p are unset. Repeated sampling would characterise the
   *distribution* of the collapse rather than its most likely instance.
9. **I did not run anything larger.** Every model here is under 4B parameters.
   The claim I make is that the blind spot is *present and structured* in this
   class of models and that its fingerprint (the dissociation between noticing
   conditionality and being able to resolve it, and the label-without-content
   failure) is what to look for in larger ones. Testing it on a 30B+ model is
   the obvious next step, and the bucket contains the harness for it.
10. **The post-training ablation is not in here.** Because the comparison set
    already spans three labs and 1.7B–4B parameters, I spent the remaining
    bandwidth on the cross-model replication rather than on the
    Base → SFT → RL ablation of the primary model, which the checkpoint series
    makes possible. It is the most interesting next experiment I would run, and
    `code/run_model.py --model minicpm5-2b-base` is the command.
11. **The base model is prompted differently.** Were the ablation to be run, the
    base checkpoint has no chat template, so it would receive a plain
    instruction prefix instead of a chat-formatted prompt. That is the fairest
    available comparison but it is not a like-for-like prompt, and the
    comparison would need to be read with that in mind.
---

## 9. Citing and reuse

The code, dataset, figures and results in this bucket are released under the
Apache-2.0 licence (see `LICENSE`), including the model weights in `model/`,
which are unmodified OpenBMB `MiniCPM5-2B` checkpoints (also Apache-2.0). The
ground-truth positions in `dataset/msq49.jsonl` are transcribed from *The Five
Schools of Islamic Law* by Sayyid Muhammad Jawad Mughniyya; please cite the
source works recorded per item in `sources` if you reuse them.