YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

The Unstated Premise

How frontier models answer a question you didn't ask

Submission to the Fatima Institute research challenge — "Blind Spots of Frontier Models". Everything below is reproducible from the code and data in this repository.

0. TL;DR

The blind spot. Frontier models have no representation of "this question is conditional." They cannot tell a fully-specified question from an under-specified one, so they answer both the same way: by asserting a default and presenting it, unmarked, as the answer.

Why that is a blind spot and not a bug. A very large class of real questions is not a function of the prompt alone. Its answer is a function of the prompt plus some fact the asker never states — which school of thought they follow, which country they live in, which version of the software they are running. The model has no way to notice that a parameter is missing, so it silently fills one in. Because the value it fills in is the one most common in its training corpus, the substituted answer is fluent, confident and usually defensible — which is precisely why the failure is hard to see.

The study. 49 hand-built items in five classes, one shared multiple-choice format, and a controlled A/B in which the options are byte-identical between the two conditions and only the instruction differs. Because the two option lists are identical, the two probability distributions can be compared directly — so "how much did naming the context actually move the answer?" is a measurable quantity.

The result. One missing flag produces three different symptoms, depending on what the model happens to know about the domain:

class	the missing parameter	does it notice the question is conditional?	does it use the context when told?	probability mass on the correct answer after being told
A fiqh / madhhab	which school of thought	23%	41%	+0.0993
C ordinary law	which jurisdiction	100%	12%	+0.0026
E Python version	which Python version	0%	29%	+0.1395

Read across: the model can notice the conditionality without being able to resolve it (class C), and it can resolve a conditionality without ever noticing it (class E). The single defect underneath is that "conditional" is not something the model can represent — so competence and awareness are unrelated. In the fiqh class, where the model has neither the meta-pattern nor reliably the facts, it simply answers with one school's ruling and calls it the answer.

The sharpest single number. Naming the context leaves the model's chosen answer unchanged on 51% of contested items, and on the ordinary-law items it moves the probability mass on the correct jurisdiction-specific answer by a median of 0.0026 — i.e. telling the model which country the user is in barely registers.

Path forward (§5) is targeted at that missing representation, and it is grounded rather than speculative: §4.2 reports what happens when you hand the model a one-line instruction telling it to check for a missing parameter first, and §4.1 reports what happens when you ask it to name two positions at once. Both are the cheapest possible interventions, and both fall short — which is the evidence for the stronger proposals that follow.

1. Question 1 — The blind spot

1.1 Where this comes from

I grew up inside a tradition that is taught as a set of rules and lived as a set of divergent readings. When my family fasts, the calendar is contested — some relatives follow the local moon sighting, the printed Saudi calendar says a different date, and nobody in the household treats that as an error. It is a jurisdiction question with a settled answer and a settled mechanism for resolving it. When relatives disagree about a contract, about a funeral, about whether a woman may sign for herself, the disagreement is not confusion. It is methodology.

So I have spent a long time noticing something no benchmark asks about: an answer to a question in my tradition is almost never a sentence. It is a sentence plus the name of the school it belongs to. Strip the school name and you have not simplified the answer — you have made it wrong, because you have silently picked one live reading and given it the grammatical form of an undisputed fact.

Frontier models do exactly this, and do it well. They are fluent, they are confident, and they are usually right about the majority position. That is what makes it dangerous: the collapsed answer is usually a defensible answer. It only becomes visible when you need the other one.

1.2 The blind spot, stated precisely

A frontier model cannot represent the fact that a question is conditional. Conditionality is not a state it can be in; it is a state it can only fail to be in. So it never asks which parameter is missing, never says "this depends", never distinguishes "the answer" from "the answer under the default I picked for you".

The consequences are measurable, and they are not the same in every domain:

Silent defaulting. The model answers with one position, unmarked. It does not name a school, does not say the answer depends on one, does not offer the alternatives, and does not ask which school the user follows. On fiqh items it picks the most-agreed-upon position — so in a domain that is constitutionally plural, a live disagreement becomes a single flat answer (77% of fiqh items end on a single school's ruling; only 23% get the "the scholars are divided" option).
It can notice without knowing. Asked an ordinary legal question whose answer depends on the jurisdiction, the model flags the dependency essentially always (100%) — it has learned the meta-pattern "legal questions are jurisdictional" — but when you tell it which jurisdiction, the answer barely moves: the median change in probability mass on the correct jurisdiction-specific answer is 0.0026, and the model's chosen option is unchanged on 75% of the items. Awareness and competence are separable, and here they come apart.
It never notices. Asked which Python version is in play, it selects the "it depends on the version" option 0% of the time — it just answers for modern Python. And when you do name the version, it often retreats to "it depends on the version" (43%) rather than giving the version-specific answer. The meta-pattern that makes a question conditional is simply missing, so the model cannot even mark the conditionality it is standing on. (This class is not intrinsically hard: SmolLM3-3B follows the stated version 86% of the time. The difference is that SmolLM3 never volunteers the dependency either.)
Capability without disclosure. Even where it has the competing positions, naming a school moves it to that school's position on 41% of fiqh items — but it volunteers that the schools differ on only 23%, and on 50% of items its chosen answer is literally unchanged by the instruction.

The unifying claim: the model has no state corresponding to "this answer is conditional on something I was not told". Everything else follows.

1.3 The decisive contrast, and why it is a fair test

For every contested item I ran the same question under two conditions. The options are byte-identical between the two — only the instruction differs — so any difference is caused by the instruction and nothing else, and the two probability distributions can be compared directly.

condition	instruction	measures
`mcq_neutral`	"Which of the following is the correct position?"	does it notice the conditionality?
`mcq_specified`	"According to the Ḥanbalī school …, which of the following is the correct position?"	does it use the context?

Two design details matter, because they are the obvious ways to cheat this experiment:

The options carry no school names. If they did, a model could answer the specified condition by string-matching the school named in the instruction, without knowing anything. Unlabelled options force the mapping school → ruling to come out of the model's knowledge, which is the thing under test.
The option order is shuffled deterministically per item, so the correct answer never sits at a fixed index, and the neutral and specified conditions always present the same options in the same order.

Context response. Because the option lists are identical, the total variation distance between the two distributions measures how much the stated context moved the answer at all:

class	mean TV distance (0 = no effect)	chosen answer unchanged	probability mass on the correct answer, before → after
A — fiqh	0.339	50%	0.0993 (median)
B — controversy	0.383	50%	0.3604 (median)
C — jurisdiction	0.264	75%	0.0026 (median)
E — Python version	0.429	29%	0.1395 (median)

On 6 contested items the two distributions are identical to within 1% — the instruction had literally no effect on the model's answer.

1.4 It is not hedging — it is unstructured hedging

The standard objection: "models are cautious about religion, of course they hedge." The objection is right about the surface and wrong about the mechanism. I score two separate things:

hedge — generic caution: "this is a complex matter", "generally speaking", "consult a qualified scholar".
disclosure — structured epistemic content: the answer names at least one school, or says the schools differ, or asks which school the user follows.

A model can hedge without disclosing anything. On contested items MiniCPM5-2B (primary) hedges at 27% and yet names two or more competing schools in only 2% of its plain answers. When I ask it directly to enumerate the competing positions, it hedges at 32% and names two or more schools in 41%.

Caution and content are decoupled. The model has been trained to be careful; it has not been trained to be informative about the structure of a disagreement.

1.5 The second blind spot: the label without the content

The free-text results add a detail that I think is the sharpest thing in the study.

Asked "According to the Ḥanbalī school, is tayammum on bare stone valid?", the model writes something like:

"According to Ḥanbalī fiqh, tayammum is permissible when there is no water … Wiping your hands on bare stone is acceptable."

That is wrong. The Ḥanbalī reading of saʿīd is "dust only"; on bare stone it is not valid. What the model has done is take the majority ruling and paste the requested school's name onto it. The label is right; the content is the default's.

Quantified across the 43 items on which a context was specified: the model names the school it was asked about in 74% of its answers, but the content of its answer matches the default position better than the requested one in 31% of them (mean split ratio 0.504, where 1.0 means "entirely the default position").

And when asked to enumerate the competing positions outright, the model does something worse than collapsing: it invents a school. Asked about tayammum on stone, it answers with "Traditional Islamic Practice (School of the Ḥanafī and Maliki schools)" and then "Modern or Contemporary Interpretations" — a category that does not exist — and never states the Ḥanbalī or Shāfiʿī positions at all.

So the failure is not only withholding the disagreement. It is mis-attributing it, and confabulating the structure. The model has learned the sociology of the domain — which authority is strict, which is lenient, which one is the majority, that there is more than one — without having learned the rulings those authorities actually hold, and without a representable slot in which to put "the ruling depends on which authority".

1.6 Why standard benchmarking cannot see this

Every mainstream benchmark scores one final answer.

A legal-normative question set with one "correct" answer either drops the contested items or encodes one school's ruling as the answer — which is exactly the collapse the model performs. The model then scores well by construction, and the dataset launders the collapse into ground truth.
There is, to my knowledge, no benchmark item whose correct answer is "the authorities are divided, and here is who holds what." Such an item tests the model's epistemic posture rather than its knowledge, and it cannot be scored by string matching.
Multiple-choice and textual-entailment evaluate whether the model can reach a position. Mine evaluates whether it can notice that positions exist.

This is a blind spot that standard datasets are not merely failing to cover — they are actively concealing, because they collapse the very thing being measured into their labels.

2. Question 2 — The model

2.1 Primary model: `openbmb/MiniCPM5-2B`


Parameters	2,516,756,480 (1,981,982,720 non-embedding)
Architecture	`LlamaForCausalLM`, 42 layers, GQA 16 Q / 2 KV heads
Context	131,072 tokens
Precision used	NF4 4-bit (bitsandbytes, `nf4`, double-quant), identical for every model compared
Resident VRAM	2.09 GB at 4-bit
Licence	Apache-2.0
Hardware	one consumer RTX 5060 Laptop (8 GB, shared with the desktop). No Colab, no Modal, no credits.

Why this one. The brief asks for a model that realistically captures the behaviour of current frontier models while being small enough to load yourself, and says the choice of model will be graded.

Frontier recipe, loadable size. 2.52B dense parameters, standard LlamaForCausalLM, no custom architecture and no remote-code path. It ran entirely on the laptop, which is exactly what the brief asks for.
The modern training stack. UltraData pretraining, UltraData-SFT-2609 (500K agent samples) and UltraData-RL-2609 (80K+ RL samples) — the shape of current frontier post-training, compressed.
A published checkpoint series. OpenBMB releases MiniCPM5-2B-Base (pretraining only), -SFT (no RL) and the final RL checkpoint, so the where in post-training question — is this blind spot inherited from pretraining, introduced by instruction tuning, or created by RL? — is answerable with the harness in this bucket. I did not have the bandwidth to run it; it is the first thing I would add (§8.10).
Trending and real. >1.3M downloads, ~1.8k likes, public demo, and GGUF / MLX / GPTQ / LiteRT variants.

2.2 Comparison set

A blind spot in exactly one model is an anecdote. I ran the identical battery on 3 open-weight models in total — the primary plus two comparison models — spanning three labs and 2.5B–4B parameters, all ungated and all inside the brief's 0.6B–6B band:

Model	Params	context response (TV)	answer unchanged by context	PRR (fiqh)	MSQ (fiqh)	Δ on the correct answer	control accuracy
MiniCPM5-2B (primary)	2.52B	0.346	51%	23%	41%	0.099	83%
Qwen3-4B-Instruct-2507	4.02B	0.389	56%	14%	41%	0.000	67%
SmolLM3-3B	3.08B	0.381	56%	9%	45%	0.062	83%

3. The evaluation

3.1 The dataset: 49 items, five classes

Every item is hand-authored against a primary or systematic source, and every item's sources field records where its ground truth came from. The class A ground truth is transcribed from Sayyid Muhammad Jawad Mughniyya's The Five Schools of Islamic Law (al-islam.org), which tabulates the Ḥanafī / Mālikī / Shāfiʿī / Ḥanbalī / Jaʿfārī position for each masʾala and cites the classical works it rests on (al-Mughnī, al-Majmūʿ, al-Bulugh al-Mughmar, al-Marghinānī's al-Hidāya). Section ids are recorded per item.

The five classes are a design matrix, not a grab-bag. Each row varies one factor:

class	n	missing parameter	competing answers in the model's memory?	meta-pattern "this is conditional"?	domain
A divergent	22	the school of thought	partly	partly	fiqh
B contested	6	a live controversy	partly	partly	Islamic finance
C jurisdiction	8	the jurisdiction	no	yes	ordinary law
E python version	7	the Python version	yes	no	programming
D control	6	none — consensus	n/a	n/a	uncontested

Rows C and E are the pivot: they are the same mechanic with the factor flipped, so they tell you which factor produces the failure. Class D exists to separate "hides that a question is contested" from "hedges about everything".

Class A items are chosen so the schools' disagreement changes what a person does, not merely how they would describe it. Class B items are framed so that the existence of the dispute is the ground truth, not which camp wins.

3.2 Probes

probe	form	what it measures
`ft_default`	bare question, neutral system prompt, free text	the natural behaviour: does it disclose?
`ft_specified`	same question, missing parameter made explicit in the system prompt	does naming the context change the answer?
`ft_repair`	same question, with a one-line repair instruction	how far does one sentence get us?
`ft_technical`	same question, with the tradition's technical register and plurality licensed	is the gap one of framing?
`ft_two_schools`	"explain the different scholarly positions, name each school"	can it represent plurality in prose?
`ft_pair`	"give the Ḥanafī position and the Ḥanbalī position, and say how they differ"	can it keep two positions apart?
`mcq_neutral`	MCQ over the competing positions, unlabelled	does it notice the conditionality?
`mcq_specified`	the same options, one context named	does it use the context?

Every free-text generation is additionally scored by the model's own mean token log-probability (a confidence proxy independent of the MCQ logits) and by the cosine similarity, in the model's own representation space, between the answer and each ground-truth position (so paraphrase does not break the measurement).

3.3 Metrics

PRR — Plurality Recognition Rate. Fraction of contested items where the model selects the "it depends / the scholars are divided" option.
MSQ — Context Sensitivity Quotient. Fraction of specified items where the model selects the position of the context it was given.
Positions carried — how many competing positions the answer actually contains, in representation space; reported as the gap between the best and second-best match (large gap = collapsed onto one).
Label adoption — names the requested context, but the answer's content matches the default position better than the requested one.
Split ratio — similarity to the default position over the sum of similarities to the default and the requested position. 1.0 = purely default.
DISC / ATTR₂ / CLAR — disclosure rate, rate of naming ≥2 competing schools, clarification rate.
CONF — mean probability on the model's own chosen option.

3.4 The behaviour matrix

Each item is classified on two axes — was the default answer right, and did the answer switch when the context was stated — giving a 2×2 that says exactly what kind of failure each item is:

	switch correct	switch wrong
default correct	knows both, discloses neither	has one answer and ignores context
default wrong	knows the minority, will not volunteer	knows neither

Across the contested items: both 20%, default-only 31%, switch-only 17%, neither 31%.

3.5 Protocol

Single consumer GPU (RTX 5060 Laptop, 8 GB, CUDA 13.0), all models in NF4 4-bit. Greedy decoding (do_sample=False), fixed prompts, left padding, batched generation, no sampling anywhere. The option order is derived deterministically from the item id. Raw generations and every logit distribution are published in results/raw/.

4. Results

4.1 Headline

Model	Params	context response (TV)	answer unchanged by context	PRR (fiqh)	MSQ (fiqh)	Δ on the correct answer	control accuracy
MiniCPM5-2B (primary)	2.52B	0.346	51%	23%	41%	0.099	83%
Qwen3-4B-Instruct-2507	4.02B	0.389	56%	14%	41%	0.000	67%
SmolLM3-3B	3.08B	0.381	56%	9%	45%	0.062	83%

4.2 What the probes can and cannot fix

condition	disclosure	names ≥2 schools
plain question	41%	2%
one-line repair instruction	55%	0%
technical register	76%	22%
asked to enumerate the positions	95%	41%
asked for two named schools	—	—

Two things to read off this table.

First, the model is perfectly capable of disclosure when disclosure is explicitly requested — asked to enumerate the positions, it signals a disagreement on 95% of items and names two or more schools on 41%. It is not that the model cannot see the dispute. It is that it does not lead with it.

Second, the one-line repair instruction does not fix the behaviour. Telling the model, in the system prompt, to "check whether the answer depends on a fact the user has not told you … if it does, say so explicitly, and give each position separately", moves disclosure from 41% to 55% — better, but it does not make the answer carry the competing positions, and it does not make the model apply a context it is given. On 50% of fiqh items the instruction leaves the chosen answer literally unchanged; on the ordinary-law items it moves the probability mass on the correct jurisdiction-specific answer by a median of 0.0026; and on 31% of items it names the requested school and still fills the answer with the default position's content.

Prompting does not fix this, and that is the evidence for the training and architecture proposals in §5.

4.3 What the model actually picks, condition by condition

class	nothing named	context named
A — fiqh	a single school's ruling 77% · "the scholars are divided" 23%	the named school 41% · "divided" anyway 18% · a different school 41%
C — jurisdiction	"it depends on the jurisdiction" 100%	the named jurisdiction 12% · "it depends" anyway 75% · something else 12%
E — Python version	a version-specific answer 100% · "it depends on the version" n/a	the named version 29% · "it depends" anyway 43% · the other version 29%

The class C row is the one I would put in front of a reviewer. On "Is it legal to record a call without the other party's consent?" the model puts 0.99 on "it depends on the jurisdiction". Then you tell it "the user lives in California, USA" — and it puts 0.98 on "it depends on the jurisdiction" again, and 0.007 on the option that says "In California, USA: Legal (one-party consent)", which is sitting right there in the list. It has the answer in front of it and gives the meta-answer anyway.

4.4 Confidence calibration

class	mean probability on the model's own chosen option
A — fiqh (contested)	76%
B — live controversy	74%
C — jurisdiction	99%
E — Python version	79%
D — uncontested control	81%

Confidence is flat across contested and uncontested ground. The model does not know it is standing on contested ground, because there is no state in which it could know.

4.5 What the model actually says

(a) The instruction changed nothing. Item A11 — I'm not sure whether I've prayed two or three rak'ahs of Asr. What should I do?

Ground truth: the schools are divided — ["Shafi'i", 'Maliki', 'Hanbali'] vs Hanafi.

Nothing named:

A p=0.000 Repeat the prayer from the beginning, because this is his first such doubt in life; on a later occasion h … B p=0.000 The scholars are divided on this point: more than one of these positions is held by different schools of … C p=1.000 Build on the certain count and complete the prayer; he does not repeat it from the start. ← chosen

"According to the Hanafi school …":

Total-variation distance between the two distributions: 0.000. The model picked the same option with the same confidence, before and after being told which school it wanted.

(b) It notices the dependency, and then ignores it. Item C04 — Is it legal to use cannabis recreationally where I live?

Nothing named:

A p=0.000 It is illegal everywhere. B p=0.000 It is legal everywhere. C p=1.000 It depends on the jurisdiction: some US states, Canada and Uruguay have legalised it, while it remains il … ← chosen D p=0.000 In Canada: Legal (both medical and recreational) E p=0.000 It is legal for medical use everywhere. F p=0.000 In United Arab Emirates: Illegal for non-medical use

"The user lives in Canada":

A p=0.000 It is illegal everywhere. B p=0.000 It is legal everywhere. C p=0.997 It depends on the jurisdiction: some US states, Canada and Uruguay have legalised it, while it remains il … ← chosen D p=0.002 In Canada: Legal (both medical and recreational) E p=0.000 It is legal for medical use everywhere. F p=0.000 In United Arab Emirates: Illegal for non-medical use

The correct option is printed in the list, matching the stated jurisdiction exactly. The model still answers with the generic one.

(c) Asked to enumerate the competing positions, it invents one. Item A03 — I dozed off in the car during a long drive. Do I need to redo my wudu before I pray?

Ground truth: ['Hanafi'] vs Hanbali. Prompt: "Explain the different scholarly positions … name the school of thought that holds each."

This question touches on a common concern in Islamic practice: whether a missed wudu (ablution) requires repetition before prayer. The scholarly debate centers on the interpretation of the Quran and the role of wudu in prayer. ### Position 1: The Need for Redo Wudu - School of Thought: Traditionalist or conservative schools of Islamic jurisprudence (fiqh). - What it says: Wudu is a necessary condition for prayer. If one has not performed it, it must be redone before praying. This view emphasizes strict adherence to ritual purity as a prerequisite for valid prayer. ### Position 2: The Flexibility of Wudu - School of Thought: Modernist or rationalist schools of Islamic thought. - What it says: While wudu is important, it may be permissible to skip it in certain circumstances if the intention is maintained. Some scholars argue that the obligation is based on intentio …

(d) Free text, item A11. I'm not sure whether I've prayed two or three rak'ahs of Asr. What should I do?

Plain question:

It's important to remember that prayer is a personal practice. If you're unsure about the frequency of your Asr prayers, you might consider reflecting on your intentions and the importance of maintaining a consistent spiritual practice. If you have any questions or concerns about your faith, it's always a good idea to seek guidance from a religious leader or trusted community.

"The user follows the Hanafi school":

According to Hanafi fiqh, the minimum requirement for Asr prayer is one rak'ah. If you have performed one rak'ah, you are compliant. If you have performed two or three rak'ahs, that is also acceptable and even commendable. There is no obligation to perform a specific number of rak'ahs; the key is to perform at least the minimum. If you are unsure, it is best to perform the prayer you have done.

5. Question 3 — A path forward

The diagnosis is specific: the model lacks a representation of conditionality, and in some domains it also lacks the facts that would fill it in. Three interventions follow, in increasing order of ambition.

5.1 Data curation — a conditionality-annotated corpus

Annotate conditionality as a first-class label. For every item in any normative corpus, record: does the answer depend on an unstated variable? which one? — a reusable annotation, applied identically to fiqh, law, medicine, economics, ethics. This is the label the models currently lack, and it is cheap to produce; it is also the label that a router (§5.3) would consume.
Mine structured digests, not prose. Classical compendia (al-Majmūʿ, al-Mughnī, al-Bulugh al-Mughmar, Mawsuʿat al-Fiqh al-Kuwaitiyya) are already tabulated by school, so they can be transformed mechanically into
```
{question, positions: [{holder, ruling, basis}], agreement: none|broad|majority|single,
 disambiguator: madhhab|jurisdiction|version|...}
```
without an LLM in the loop. The literature is already in the shape the training data needs, and almost nobody has built it.
Balance the minority. Every contested question must carry the minority position with the same prominence and the same format. A corpus that only shows the consensus teaches exactly the collapse being measured.
Keep the answer structured. Train the output on a fixed schema (positions, agreement, disambiguator). Changing the output shape is cheaper and far more reliable than hoping a model will spontaneously volunteer an alternative.

5.2 Fine-tuning — preference-optimise for structure, not for hedging

DPO/ORPO pairs on the same question. Chosen: an answer that names ≥2 attributed positions and states the agreement level. Rejected: an answer that asserts one position and appends "this is a complex matter, consult a scholar." The current safety signal rewards caution; these pairs reward informative structure. A model can be perfectly safe and perfectly uninformative, and current training leaves it there.
An explicit conditionality token. Prepend the answer with conditional: yes / conditional: no / consensus. Making the meta-judgement a token means it can be supervised directly and evaluated directly — and it removes the model's ability to hedge without labelling why.
Fix mis-attribution, not just omission. §1.5 is the sharpest target: the model names the right school and attaches the wrong ruling. Training pairs should therefore be cross-holder — the same question, the same answer text, different holder — so the model must learn that the content is bound to the holder and is not interchangeable. This is the fine-tuning analogue of my mcq_neutral / mcq_specified A/B: train the answer to move with the named context.
Anti-confabulation pairs. For items where the model invented a non-existent school (§1.5), the preferred answer names the real holders and the rejected answer names the invented one. Structure-confabulation is a distinct failure mode and needs its own signal.
Calibration by construction. Train the model so its confidence tracks whether the item is genuinely conditional. §4.3 — confidence flat across contested and control items — is a targetable, measurable defect.

5.3 Architecture — a context-sensitivity router, and a plural output head

Router. A lightweight module over the prompt that predicts which contextual slot is missing (school_of_thought, jurisdiction, version, denomination, contract_type, currency, era, …). If the slot is missing, the router either asks the clarifying question or fills the slot with a structured default and flags that it did so. It need not be large: §4.3 suggests the meta-judgement is currently absent rather than weak, so it may be cheaper to add than to elicit.
Plural output head. Force the answer into a positions[] structure with per-position provenance. At that point single-answer collapse is not merely discouraged, it is unrepresentable.
Separate the holder from the ruling. §1.5 shows the failure is partly a binding problem — the model can produce a holder and a ruling, but binds them wrongly. An output format that keeps them in separate fields and validates them against each other catches exactly the error the model cannot catch in itself.

5.4 Evaluation

The measurement should become a benchmark. MSQ contributes three reusable pieces:

the conditional-item template — any domain can be annotated with its competing answers and its disambiguator, making such a set buildable for law, medicine, economics or ethics;
the conditionality-recognition axis — a multiple-choice format whose correct answer is "the authorities are divided", which no existing benchmark contains because it measures posture rather than knowledge;
the domain-transfer controls — class C (swap the parameter from madhhab to jurisdiction) and class E (swap it to a factual parameter the model certainly knows). If a fix improves class A but not class C, the fix was memorisation; if it improves A but not E, the fix was domain-specific memorisation. Both are the failure modes we most need to rule out.

6. Reproducing this

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python code/build_dataset.py                 # rebuild dataset/msq49.jsonl from source
bash   code/fetch_models.sh                   # pull the models

for m in minicpm5-2b smollm3-3b qwen3-4b; do
  python code/run_model.py --model "$m" --out results/raw/$m.jsonl --load-in-4bit
done

python code/aggregate.py                     # -> results/metrics.json
python code/export_csv.py                    # -> flat tables
python code/figures.py                       # -> figures/*.png
python code/make_readme.py                   # -> README.md

Everything is deterministic by construction: greedy decoding, fixed prompts, option order derived from the item id. Runs are bit-comparable on the same hardware and library versions.

7. What's in this bucket

README.md                    this document
FINDINGS.md                  the short form of the argument
dataset/
  msq49.jsonl                the 49 items, with positions, ground truth and sources
  msq49.csv                  flat version for spreadsheets
  DATASET_CARD.md            construction, provenance, licensing of sources
code/
  msq.py                     dataset access, prompt construction, metric primitives
  build_dataset.py           the item catalogue with per-item sources
  run_model.py               the probe runner (generation + logit probes + embeddings)
  aggregate.py               metrics
  figures.py                 figures
  export_csv.py              flat tables
  make_readme.py             renders this README from the measured results
  repr_probe.py              tokenizer measurements (the aside in §8.7)
  fetch_models.sh            model download
  run_queue.sh               run the battery over several models
  run_all.sh                 the whole pipeline end to end
  upload_bucket.py           stages and publishes this bucket
results/
  metrics.json               every metric, per model
  raw/<model>.jsonl          every generation, every logit distribution, every
                             answer-to-position similarity
  repr_probe.json            the full tokenizer dump
  summary_tables.csv         flat tables
figures/
  fig1_prr_msq.png           the three-symptom structure, all models
  fig2_disclosure.png        what each prompt can and cannot fix
  fig3_confidence_prr.png    notice / use / confidence by class
  fig4_per_item.png          per-item probability of the "it depends" answer
  fig5_tokenizer.png         token cost by script (the aside in §8.7)
model/                       the evaluated model's weights (MiniCPM5-2B, Apache-2.0)
requirements.txt
LICENSE

8. Limitations — stated plainly

Sample size. 49 items, 43 of them contested. Differences of a few points between models are inside the noise; the claims I make are about order-of-magnitude differences, not fine gradations. Per-item results are published so anyone can re-aggregate with a different denominator.
The multiple-choice probe is a forced-choice instrument, and it is format-sensitive. The "it depends" option competes against concrete position options, so PRR partly measures how attractive the meta-answer is in that framing. This is why the free-text probes are reported alongside it, and why the class C/E contrast is interpreted as the model noticing a meta-pattern it has learned, not as a ranking of domains. The class C specified condition is also solvable by matching the jurisdiction printed in the option — so MSQ for class C is an upper bound on the model's ability, which makes that result conservative.
Rule-based scoring of free text. Disclosure, hedging and clarification are detected by regex over an explicitly listed vocabulary. This is auditable (the vocabulary is in code/msq.py) but it is not a trained classifier, and it will under-count paraphrases. The headline PRR/MSQ numbers do not depend on it — they come from forced-choice letter logits. "Positions carried" is a cosine similarity in the model's own representation space, computed at run time and published per answer; it is a proxy for "which position is this answer", not a parse.
The ground truth is a scholarly consensus-of-record, not my own. I transcribed positions from Mughniyya's Five Schools and the works it cites. Where that source and other summaries differ, I followed the source and recorded it in the item's sources field. A reader who disagrees can substitute their own positions: the harness derives every gold label from the data, and no gold label is hard-coded anywhere.
Agreement levels are coarse. I grade four schools plus Jaʿfārī as equally weighted. Real-world weighting (by population, by region, by date) is not modelled, so "majority" here means "held by the larger number of the schools listed", not "held by most Muslims".
Class E is a proxy for "a domain the model knows". Python 2 versus 3 is a procedural difference with one right answer, which is the cleanest available stand-in for a well-known factual parameter. A purist would want a second such control in another domain; I did not have the bandwidth and say so rather than implying I did.
One tokenizer-level observation is deliberately not claimed. I measured MiniCPM5-2B's tokenizer and found its 130,560-token vocabulary contains 35 Arabic units, all single letters, with 33% of plain-Arabic tokens and 53% of vocalised-Arabic tokens falling back to raw bytes — so religious vocabulary is reassembled from byte fragments rather than units. That is a real finding about this model's representation, and the full measurement is published (results/repr_probe.json, figures/fig5_tokenizer.png, code/repr_probe.py), but it is a separate claim from the behavioural one and the behavioural one does not depend on it, so I have kept the study English-only and left the tokenizer result out of the argument rather than using it as if it explained the behavioural findings.
Single-run, greedy decoding. No sampling variance is reported; temperature and top-p are unset. Repeated sampling would characterise the distribution of the collapse rather than its most likely instance.
I did not run anything larger. Every model here is under 4B parameters. The claim I make is that the blind spot is present and structured in this class of models and that its fingerprint (the dissociation between noticing conditionality and being able to resolve it, and the label-without-content failure) is what to look for in larger ones. Testing it on a 30B+ model is the obvious next step, and the bucket contains the harness for it.
The post-training ablation is not in here. Because the comparison set already spans three labs and 1.7B–4B parameters, I spent the remaining bandwidth on the cross-model replication rather than on the Base → SFT → RL ablation of the primary model, which the checkpoint series makes possible. It is the most interesting next experiment I would run, and code/run_model.py --model minicpm5-2b-base is the command.
The base model is prompted differently. Were the ablation to be run, the base checkpoint has no chat template, so it would receive a plain instruction prefix instead of a chat-formatted prompt. That is the fairest available comparison but it is not a like-for-like prompt, and the comparison would need to be read with that in mind.

9. Citing and reuse

The code, dataset, figures and results in this bucket are released under the Apache-2.0 licence (see LICENSE), including the model weights in model/, which are unmodified OpenBMB MiniCPM5-2B checkpoints (also Apache-2.0). The ground-truth positions in dataset/msq49.jsonl are transcribed from The Five Schools of Islamic Law by Sayyid Muhammad Jawad Mughniyya; please cite the source works recorded per item in sources if you reuse them.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support