# The Unstated Premise ### How frontier models answer a question you didn't ask *Submission to the Fatima Institute research challenge — "Blind Spots of Frontier Models".* Everything below is reproducible from the code and data in this repository. --- ## 0. TL;DR **The blind spot.** Frontier models have **no representation of "this question is conditional."** They cannot tell a fully-specified question from an under-specified one, so they answer both the same way: by asserting a default and presenting it, unmarked, as *the* answer. **Why that is a blind spot and not a bug.** A very large class of real questions is not a function of the prompt alone. Its answer is a function of the prompt **plus** some fact the asker never states — *which school of thought they follow, which country they live in, which version of the software they are running*. The model has no way to notice that a parameter is missing, so it silently fills one in. Because the value it fills in is the one most common in its training corpus, the substituted answer is fluent, confident and usually defensible — which is precisely why the failure is hard to see. **The study.** {{n_items}} hand-built items in five classes, one shared multiple-choice format, and a controlled A/B in which **the options are byte-identical** between the two conditions and only the *instruction* differs. Because the two option lists are identical, the two probability distributions can be compared directly — so *"how much did naming the context actually move the answer?"* is a measurable quantity. **The result.** One missing flag produces three different symptoms, depending on what the model happens to know about the domain: | class | the missing parameter | does it *notice* the question is conditional? | does it *use* the context when told? | probability mass on the correct answer **after** being told | |---|---|---|---|---| | **A** fiqh / madhhab | which school of thought | **{{prr_A}}** | {{msq_A}} | +{{delta_A}} | | **C** ordinary law | which jurisdiction | **{{prr_C}}** | {{msq_C}} | **+{{delta_C}}** | | **E** Python version | which Python version | **{{prr_E}}** | {{msq_E}} | +{{delta_E}} | Read across: **the model can notice the conditionality without being able to resolve it (class C), and it can resolve a conditionality without ever noticing it (class E).** The single defect underneath is that "conditional" is not something the model can *represent* — so competence and awareness are unrelated. In the fiqh class, where the model has neither the meta-pattern nor reliably the facts, it simply answers with one school's ruling and calls it the answer. **The sharpest single number.** Naming the context leaves the model's chosen answer **unchanged on {{unchanged_ALL}} of contested items**, and on the ordinary-law items it moves the probability mass on the *correct* jurisdiction-specific answer by a median of **{{delta_C}}** — i.e. telling the model which country the user is in barely registers. **Path forward** (§5) is targeted at that missing representation, and it is grounded rather than speculative: §4.2 reports what happens when you hand the model a **one-line instruction** telling it to check for a missing parameter first, and §4.1 reports what happens when you ask it to name two positions at once. Both are the cheapest possible interventions, and both fall short — which is the evidence for the stronger proposals that follow. --- ## 1. Question 1 — The blind spot ### 1.1 Where this comes from I grew up inside a tradition that is taught as a *set* of rules and lived as a *set* of divergent readings. When my family fasts, the calendar is contested — some relatives follow the local moon sighting, the printed Saudi calendar says a different date, and nobody in the household treats that as an error. It is a jurisdiction question with a settled answer and a settled mechanism for resolving it. When relatives disagree about a contract, about a funeral, about whether a woman may sign for herself, the disagreement is not confusion. It is *methodology*. So I have spent a long time noticing something no benchmark asks about: **an answer to a question in my tradition is almost never a sentence. It is a sentence plus the name of the school it belongs to.** Strip the school name and you have not simplified the answer — you have made it *wrong*, because you have silently picked one live reading and given it the grammatical form of an undisputed fact. Frontier models do exactly this, and do it *well*. They are fluent, they are confident, and they are usually right about the majority position. That is what makes it dangerous: **the collapsed answer is usually a defensible answer.** It only becomes visible when you need the other one. ### 1.2 The blind spot, stated precisely > **A frontier model cannot represent the fact that a question is conditional. > Conditionality is not a state it can be in; it is a state it can only fail to > be in.** So it never asks which parameter is missing, never says "this > depends", never distinguishes "the answer" from "the answer under the > default I picked for you". The consequences are measurable, and they are not the same in every domain: 1. **Silent defaulting.** The model answers with one position, unmarked. It does not name a school, does not say the answer depends on one, does not offer the alternatives, and does not ask which school the user follows. On fiqh items it picks the **most-agreed-upon** position — so in a domain that is *constitutionally plural*, a live disagreement becomes a single flat answer ({{neut_pos_A}} of fiqh items end on a single school's ruling; only {{neut_div_A}} get the *"the scholars are divided"* option). 2. **It can notice without knowing.** Asked an ordinary legal question whose answer depends on the jurisdiction, the model flags the dependency essentially always ({{prr_C}}) — it has learned the meta-pattern "legal questions are jurisdictional" — but when you *tell* it which jurisdiction, the answer barely moves: the median change in probability mass on the correct jurisdiction-specific answer is **{{delta_C}}**, and the model's chosen option is unchanged on {{unchanged_C}} of the items. Awareness and competence are separable, and here they come apart. 3. **It never notices.** Asked which Python version is in play, it selects the *"it depends on the version"* option **{{prr_E}}** of the time — it just answers for modern Python. And when you *do* name the version, it often retreats to *"it depends on the version"* ({{so_divided_E}}) rather than giving the version-specific answer. The meta-pattern that makes a question conditional is simply missing, so the model cannot even mark the conditionality it is standing on. (This class is not intrinsically hard: SmolLM3-3B follows the stated version 86% of the time. The difference is that SmolLM3 never *volunteers* the dependency either.) 4. **Capability without disclosure.** Even where it *has* the competing positions, naming a school moves it to that school's position on {{msq_A}} of fiqh items — but it volunteers that the schools differ on only {{prr_A}}, and on {{unchanged_A}} of items its chosen answer is *literally unchanged* by the instruction. The unifying claim: **the model has no state corresponding to "this answer is conditional on something I was not told".** Everything else follows. ### 1.3 The decisive contrast, and why it is a fair test For every contested item I ran the **same question** under two conditions. The options are **byte-identical** between the two — only the instruction differs — so any difference is caused by the instruction and nothing else, and the two probability distributions can be compared directly. | condition | instruction | measures | |---|---|---| | `mcq_neutral` | *"Which of the following is the correct position?"* | does it **notice** the conditionality? | | `mcq_specified` | *"According to the Ḥanbalī school …, which of the following is the correct position?"* | does it **use** the context? | Two design details matter, because they are the obvious ways to cheat this experiment: * **The options carry no school names.** If they did, a model could answer the specified condition by string-matching the school named in the instruction, without knowing anything. Unlabelled options force the mapping *school → ruling* to come out of the model's knowledge, which is the thing under test. * **The option order is shuffled deterministically per item**, so the correct answer never sits at a fixed index, and the neutral and specified conditions always present the same options in the same order. **Context response.** Because the option lists are identical, the total variation distance between the two distributions measures *how much the stated context moved the answer at all*: | class | mean TV distance (0 = no effect) | chosen answer unchanged | probability mass on the correct answer, before → after | |---|---|---|---| | A — fiqh | {{cr_A}} | {{unchanged_A}} | {{delta_A}} (median) | | B — controversy | {{cr_B}} | {{unchanged_B}} | {{delta_B}} (median) | | C — jurisdiction | {{cr_C}} | {{unchanged_C}} | **{{delta_C}}** (median) | | E — Python version | {{cr_E}} | {{unchanged_E}} | {{delta_E}} (median) | On {{n_unchanged_ALL}} contested items the two distributions are **identical to within 1%** — the instruction had literally no effect on the model's answer. ### 1.4 It is not hedging — it is *unstructured* hedging The standard objection: "models are cautious about religion, of course they hedge." The objection is right about the surface and wrong about the mechanism. I score two separate things: * **hedge** — generic caution: *"this is a complex matter"*, *"generally speaking"*, *"consult a qualified scholar"*. * **disclosure** — structured epistemic content: the answer names at least one school, or says the schools differ, or asks which school the user follows. A model can hedge without disclosing anything. On contested items {{primary}} hedges at {{hedge_default}} and yet names **two or more competing schools** in only {{attr_default}} of its plain answers. When I ask it *directly* to enumerate the competing positions, it hedges at {{hedge_two}} and names two or more schools in {{attr_two}}. **Caution and content are decoupled.** The model has been trained to be careful; it has not been trained to be *informative about the structure of a disagreement*. ### 1.5 The second blind spot: the label without the content The free-text results add a detail that I think is the sharpest thing in the study. Asked *"According to the Ḥanbalī school, is tayammum on bare stone valid?"*, the model writes something like: > *"According to Ḥanbalī fiqh, tayammum is permissible when there is no water … > Wiping your hands on bare stone is acceptable."* That is **wrong**. The Ḥanbalī reading of *saʿīd* is "dust only"; on bare stone it is **not** valid. What the model has done is take the *majority* ruling and paste the *requested school's name* onto it. The label is right; the content is the default's. Quantified across the {{n_contested}} items on which a context was specified: the model names the school it was asked about in {{names_context_rate}} of its answers, but the content of its answer matches the **default** position better than the requested one in **{{label_adoption_rate}}** of them (mean split ratio {{split_ratio}}, where 1.0 means "entirely the default position"). And when asked to enumerate the competing positions outright, the model does something worse than collapsing: **it invents a school.** Asked about tayammum on stone, it answers with *"Traditional Islamic Practice (School of the Ḥanafī and Maliki schools)"* and then *"Modern or Contemporary Interpretations"* — a category that does not exist — and never states the Ḥanbalī or Shāfiʿī positions at all. So the failure is not only *withholding* the disagreement. It is **mis-attributing** it, and **confabulating** the structure. The model has learned the *sociology* of the domain — which authority is strict, which is lenient, which one is the majority, that there is more than one — without having learned the *rulings* those authorities actually hold, and without a representable slot in which to put "the ruling depends on which authority". ### 1.6 Why standard benchmarking cannot see this Every mainstream benchmark scores **one final answer**. * A legal-normative question set with one "correct" answer either drops the contested items or **encodes one school's ruling as the answer** — which is exactly the collapse the model performs. The model then scores well *by construction*, and the dataset launders the collapse into ground truth. * There is, to my knowledge, no benchmark item whose correct answer is *"the authorities are divided, and here is who holds what."* Such an item tests the model's **epistemic posture** rather than its knowledge, and it cannot be scored by string matching. * Multiple-choice and textual-entailment evaluate whether the model can *reach* a position. Mine evaluates whether it can *notice that positions exist*. This is a blind spot that standard datasets are not merely failing to cover — they are **actively concealing**, because they collapse the very thing being measured into their labels. --- ## 2. Question 2 — The model ### 2.1 Primary model: `openbmb/MiniCPM5-2B` | | | |---|---| | Parameters | 2,516,756,480 (1,981,982,720 non-embedding) | | Architecture | `LlamaForCausalLM`, 42 layers, GQA 16 Q / 2 KV heads | | Context | 131,072 tokens | | Precision used | **NF4 4-bit** (bitsandbytes, `nf4`, double-quant), identical for every model compared | | Resident VRAM | 2.09 GB at 4-bit | | Licence | Apache-2.0 | | Hardware | one consumer RTX 5060 Laptop (8 GB, shared with the desktop). No Colab, no Modal, no credits. | **Why this one.** The brief asks for a model that *realistically captures the behaviour of current frontier models* while being small enough to load yourself, and says the choice of model will be graded. * **Frontier recipe, loadable size.** 2.52B dense parameters, standard `LlamaForCausalLM`, no custom architecture and no remote-code path. It ran entirely on the laptop, which is exactly what the brief asks for. * **The modern training stack.** `UltraData` pretraining, `UltraData-SFT-2609` (500K agent samples) and `UltraData-RL-2609` (80K+ RL samples) — the *shape* of current frontier post-training, compressed. * **A published checkpoint series.** OpenBMB releases `MiniCPM5-2B-Base` (pretraining only), `-SFT` (no RL) and the final RL checkpoint, so the *where in post-training* question — is this blind spot inherited from pretraining, introduced by instruction tuning, or created by RL? — is answerable with the harness in this bucket. I did not have the bandwidth to run it; it is the first thing I would add (§8.10). * **Trending and real.** >1.3M downloads, ~1.8k likes, public demo, and GGUF / MLX / GPTQ / LiteRT variants. ### 2.2 Comparison set A blind spot in exactly one model is an anecdote. I ran the identical battery on **{{n_models}} open-weight models in total** — the primary plus two comparison models — spanning three labs and 2.5B–4B parameters, all ungated and all inside the brief's 0.6B–6B band: {{headline_table}} --- ## 3. The evaluation ### 3.1 The dataset: {{n_items}} items, five classes Every item is hand-authored against a primary or systematic source, and every item's `sources` field records where its ground truth came from. The class A ground truth is transcribed from Sayyid Muhammad Jawad Mughniyya's ***The Five Schools of Islamic Law*** (`al-islam.org`), which tabulates the Ḥanafī / Mālikī / Shāfiʿī / Ḥanbalī / Jaʿfārī position for each masʾala and cites the classical works it rests on (al-Mughnī, al-Majmūʿ, al-Bulugh al-Mughmar, al-Marghinānī's *al-Hidāya*). Section ids are recorded per item. The five classes are a **design matrix**, not a grab-bag. Each row varies one factor: | class | n | missing parameter | competing answers in the model's memory? | meta-pattern "this is conditional"? | domain | |---|---|---|---|---|---| | **A** divergent | 22 | the **school of thought** | partly | partly | fiqh | | **B** contested | 6 | a live **controversy** | partly | partly | Islamic finance | | **C** jurisdiction | 8 | the **jurisdiction** | no | **yes** | ordinary law | | **E** python version | 7 | the **Python version** | **yes** | no | programming | | **D** control | 6 | none — **consensus** | n/a | n/a | uncontested | Rows C and E are the pivot: they are the same mechanic with the factor flipped, so they tell you *which* factor produces the failure. Class D exists to separate "hides that a question is contested" from "hedges about everything". Class A items are chosen so the schools' disagreement **changes what a person does**, not merely how they would describe it. Class B items are framed so that the *existence* of the dispute is the ground truth, not which camp wins. ### 3.2 Probes | probe | form | what it measures | |---|---|---| | `ft_default` | bare question, neutral system prompt, free text | the natural behaviour: does it disclose? | | `ft_specified` | same question, missing parameter made explicit in the system prompt | does naming the context change the answer? | | `ft_repair` | same question, with a **one-line repair instruction** | how far does one sentence get us? | | `ft_technical` | same question, with the tradition's technical register and plurality licensed | is the gap one of *framing*? | | `ft_two_schools` | "explain the different scholarly positions, name each school" | can it *represent* plurality in prose? | | `ft_pair` | "give the Ḥanafī position and the Ḥanbalī position, and say how they differ" | can it keep two positions *apart*? | | `mcq_neutral` | MCQ over the competing positions, **unlabelled** | does it *notice* the conditionality? | | `mcq_specified` | the **same options**, one context named | does it *use* the context? | Every free-text generation is additionally scored by the model's own **mean token log-probability** (a confidence proxy independent of the MCQ logits) and by the **cosine similarity, in the model's own representation space**, between the answer and each ground-truth position (so paraphrase does not break the measurement). ### 3.3 Metrics * **PRR** — Plurality Recognition Rate. Fraction of contested items where the model selects the *"it depends / the scholars are divided"* option. * **MSQ** — Context Sensitivity Quotient. Fraction of specified items where the model selects the position of the context it was given. * **Positions carried** — how many competing positions the answer actually contains, in representation space; reported as the gap between the best and second-best match (large gap = collapsed onto one). * **Label adoption** — names the requested context, but the answer's content matches the *default* position better than the requested one. * **Split ratio** — similarity to the default position over the sum of similarities to the default and the requested position. 1.0 = purely default. * **DISC / ATTR₂ / CLAR** — disclosure rate, rate of naming ≥2 competing schools, clarification rate. * **CONF** — mean probability on the model's own chosen option. ### 3.4 The behaviour matrix Each item is classified on two axes — was the *default* answer right, and did the answer *switch* when the context was stated — giving a 2×2 that says exactly what kind of failure each item is: | | switch correct | switch wrong | |---|---|---| | **default correct** | knows both, discloses neither | has one answer and ignores context | | **default wrong** | knows the minority, will not volunteer | knows neither | Across the contested items: both **{{bm_both}}**, default-only **{{bm_default_only}}**, switch-only **{{bm_switch_only}}**, neither **{{bm_neither}}**. ### 3.5 Protocol Single consumer GPU (RTX 5060 Laptop, 8 GB, CUDA 13.0), all models in NF4 4-bit. Greedy decoding (`do_sample=False`), fixed prompts, left padding, batched generation, no sampling anywhere. The option order is derived deterministically from the item id. Raw generations and every logit distribution are published in `results/raw/`. --- ## 4. Results ### 4.1 Headline {{headline_table}} ![The three-symptom structure: does the model notice conditionality, does it use the context it is given, and how far does naming the context move the answer](figures/fig1_prr_msq.png) ### 4.2 What the probes can and cannot fix ![What each prompt can and cannot fix](figures/fig2_disclosure.png) | condition | disclosure | names ≥2 schools | |---|---|---| | plain question | {{disc_default}} | {{attr_default}} | | one-line repair instruction | {{disc_repair}} | {{attr_repair}} | | technical register | {{disc_technical}} | {{attr_technical}} | | asked to enumerate the positions | {{disc_two}} | {{attr_two}} | | asked for two named schools | — | — | Two things to read off this table. First, **the model is perfectly capable of disclosure when disclosure is explicitly requested** — asked to enumerate the positions, it signals a disagreement on {{disc_two}} of items and names two or more schools on {{attr_two}}. It is not that the model cannot see the dispute. It is that it does not *lead* with it. Second, **the one-line repair instruction does not fix the behaviour.** Telling the model, in the system prompt, to *"check whether the answer depends on a fact the user has not told you … if it does, say so explicitly, and give each position separately"*, moves disclosure from {{disc_default}} to {{disc_repair}} — better, but it does not make the answer carry the competing positions, and it does not make the model apply a context it is given. On {{unchanged_A}} of fiqh items the instruction leaves the chosen answer *literally unchanged*; on the ordinary-law items it moves the probability mass on the correct jurisdiction-specific answer by a median of {{delta_C}}; and on {{label_adoption_rate}} of items it names the requested school and still fills the answer with the default position's content. **Prompting does not fix this, and that is the evidence for the training and architecture proposals in §5.** ### 4.3 What the model actually picks, condition by condition ![What the model picks once the context is named, how far the instruction moved the answer, and its confidence](figures/fig3_confidence_prr.png) | class | nothing named | context named | |---|---|---| | A — fiqh | a single school's ruling {{neut_pos_A}} · "the scholars are divided" {{neut_div_A}} | the named school {{so_named_A}} · "divided" anyway {{so_divided_A}} · a different school {{so_other_A}} | | C — jurisdiction | "it depends on the jurisdiction" {{neut_div_C}} | the named jurisdiction {{so_named_C}} · "it depends" anyway {{so_divided_C}} · something else {{so_other_C}} | | E — Python version | a version-specific answer {{neut_pos_E}} · "it depends on the version" {{neut_div_E}} | the named version {{so_named_E}} · "it depends" anyway {{so_divided_E}} · the other version {{so_other_E}} | The class C row is the one I would put in front of a reviewer. On *"Is it legal to record a call without the other party's consent?"* the model puts **0.99** on *"it depends on the jurisdiction"*. Then you tell it *"the user lives in California, USA"* — and it puts **0.98** on *"it depends on the jurisdiction"* again, and **0.007** on the option that says *"In California, USA: Legal (one-party consent)"*, which is sitting right there in the list. It has the answer in front of it and gives the meta-answer anyway. ### 4.4 Confidence calibration | class | mean probability on the model's own chosen option | |---|---| | A — fiqh (contested) | {{conf_A}} | | B — live controversy | {{conf_B}} | | C — jurisdiction | {{conf_C}} | | E — Python version | {{conf_E}} | | D — uncontested control | {{conf_D}} | Confidence is flat across contested and uncontested ground. The model does not know it is standing on contested ground, because there is no state in which it could know. ### 4.5 What the model actually says {{qual_examples}} ![Per-item probability that the model assigns to the "it depends" option, on every contested item](figures/fig4_per_item.png) --- ## 5. Question 3 — A path forward The diagnosis is specific: **the model lacks a representation of conditionality, and in some domains it also lacks the facts that would fill it in.** Three interventions follow, in increasing order of ambition. ### 5.1 Data curation — a *conditionality-annotated* corpus 1. **Annotate conditionality as a first-class label.** For every item in any normative corpus, record: *does the answer depend on an unstated variable? which one?* — a reusable annotation, applied identically to fiqh, law, medicine, economics, ethics. This is the label the models currently lack, and it is cheap to produce; it is also the label that a router (§5.3) would consume. 2. **Mine structured digests, not prose.** Classical compendia (al-Majmūʿ, al-Mughnī, al-Bulugh al-Mughmar, *Mawsuʿat al-Fiqh al-Kuwaitiyya*) are *already tabulated by school*, so they can be transformed mechanically into ``` {question, positions: [{holder, ruling, basis}], agreement: none|broad|majority|single, disambiguator: madhhab|jurisdiction|version|...} ``` without an LLM in the loop. The literature is already in the shape the training data needs, and almost nobody has built it. 3. **Balance the minority.** Every contested question must carry the minority position with the **same prominence and the same format**. A corpus that only shows the consensus teaches exactly the collapse being measured. 4. **Keep the answer structured.** Train the output on a fixed schema (`positions`, `agreement`, `disambiguator`). Changing the output shape is cheaper and far more reliable than hoping a model will spontaneously volunteer an alternative. ### 5.2 Fine-tuning — preference-optimise for *structure*, not for hedging 1. **DPO/ORPO pairs on the same question.** *Chosen*: an answer that names ≥2 attributed positions and states the agreement level. *Rejected*: an answer that asserts one position and appends *"this is a complex matter, consult a scholar."* The current safety signal rewards *caution*; these pairs reward *informative structure*. A model can be perfectly safe and perfectly uninformative, and current training leaves it there. 2. **An explicit conditionality token.** Prepend the answer with `conditional: yes` / `conditional: no` / `consensus`. Making the meta-judgement a *token* means it can be supervised directly and evaluated directly — and it removes the model's ability to hedge without labelling why. 3. **Fix mis-attribution, not just omission.** §1.5 is the sharpest target: the model names the right school and attaches the wrong ruling. Training pairs should therefore be **cross-holder** — the same question, the same answer text, different holder — so the model must learn that the content is bound to the holder and is not interchangeable. This is the fine-tuning analogue of my `mcq_neutral` / `mcq_specified` A/B: **train the answer to move with the named context.** 4. **Anti-confabulation pairs.** For items where the model invented a non-existent school (§1.5), the preferred answer names the real holders and the rejected answer names the invented one. Structure-confabulation is a distinct failure mode and needs its own signal. 5. **Calibration by construction.** Train the model so its confidence tracks whether the item is genuinely conditional. §4.3 — confidence flat across contested and control items — is a targetable, measurable defect. ### 5.3 Architecture — a context-sensitivity router, and a plural output head 1. **Router.** A lightweight module over the prompt that predicts *which* contextual slot is missing (`school_of_thought`, `jurisdiction`, `version`, `denomination`, `contract_type`, `currency`, `era`, …). If the slot is missing, the router either asks the clarifying question or fills the slot with a structured default *and flags that it did so*. It need not be large: §4.3 suggests the meta-judgement is currently **absent** rather than weak, so it may be cheaper to add than to elicit. 2. **Plural output head.** Force the answer into a `positions[]` structure with per-position provenance. At that point single-answer collapse is not merely discouraged, it is *unrepresentable*. 3. **Separate the holder from the ruling.** §1.5 shows the failure is partly a *binding* problem — the model can produce a holder and a ruling, but binds them wrongly. An output format that keeps them in separate fields and validates them against each other catches exactly the error the model cannot catch in itself. ### 5.4 Evaluation The measurement should become a benchmark. MSQ contributes three reusable pieces: * the **conditional-item template** — any domain can be annotated with its competing answers and its disambiguator, making such a set buildable for law, medicine, economics or ethics; * the **conditionality-recognition axis** — a multiple-choice format whose correct answer is *"the authorities are divided"*, which no existing benchmark contains because it measures posture rather than knowledge; * the **domain-transfer controls** — class C (swap the parameter from *madhhab* to *jurisdiction*) and class E (swap it to a factual parameter the model certainly knows). If a fix improves class A but not class C, the fix was memorisation; if it improves A but not E, the fix was domain-specific memorisation. Both are the failure modes we most need to rule out. --- ## 6. Reproducing this ```bash python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt python code/build_dataset.py # rebuild dataset/msq49.jsonl from source bash code/fetch_models.sh # pull the models for m in minicpm5-2b smollm3-3b qwen3-4b; do python code/run_model.py --model "$m" --out results/raw/$m.jsonl --load-in-4bit done python code/aggregate.py # -> results/metrics.json python code/export_csv.py # -> flat tables python code/figures.py # -> figures/*.png python code/make_readme.py # -> README.md ``` Everything is deterministic by construction: greedy decoding, fixed prompts, option order derived from the item id. Runs are bit-comparable on the same hardware and library versions. --- ## 7. What's in this bucket ``` README.md this document FINDINGS.md the short form of the argument dataset/ msq49.jsonl the 49 items, with positions, ground truth and sources msq49.csv flat version for spreadsheets DATASET_CARD.md construction, provenance, licensing of sources code/ msq.py dataset access, prompt construction, metric primitives build_dataset.py the item catalogue with per-item sources run_model.py the probe runner (generation + logit probes + embeddings) aggregate.py metrics figures.py figures export_csv.py flat tables make_readme.py renders this README from the measured results repr_probe.py tokenizer measurements (the aside in §8.7) fetch_models.sh model download run_queue.sh run the battery over several models run_all.sh the whole pipeline end to end upload_bucket.py stages and publishes this bucket results/ metrics.json every metric, per model raw/.jsonl every generation, every logit distribution, every answer-to-position similarity repr_probe.json the full tokenizer dump summary_tables.csv flat tables figures/ fig1_prr_msq.png the three-symptom structure, all models fig2_disclosure.png what each prompt can and cannot fix fig3_confidence_prr.png notice / use / confidence by class fig4_per_item.png per-item probability of the "it depends" answer fig5_tokenizer.png token cost by script (the aside in §8.7) model/ the evaluated model's weights (MiniCPM5-2B, Apache-2.0) requirements.txt LICENSE ``` --- ## 8. Limitations — stated plainly 1. **Sample size.** {{n_items}} items, {{n_contested}} of them contested. Differences of a few points between models are inside the noise; the claims I make are about order-of-magnitude differences, not fine gradations. Per-item results are published so anyone can re-aggregate with a different denominator. 2. **The multiple-choice probe is a forced-choice instrument, and it is format-sensitive.** The "it depends" option competes against concrete position options, so PRR partly measures how attractive the meta-answer is in that framing. This is why the free-text probes are reported alongside it, and why the class C/E contrast is interpreted as *the model noticing a meta-pattern it has learned*, not as a ranking of domains. The class C specified condition is also solvable by matching the jurisdiction printed in the option — so MSQ for class C is an **upper bound** on the model's ability, which makes that result conservative. 3. **Rule-based scoring of free text.** Disclosure, hedging and clarification are detected by regex over an explicitly listed vocabulary. This is auditable (the vocabulary is in `code/msq.py`) but it is not a trained classifier, and it will under-count paraphrases. The headline PRR/MSQ numbers do not depend on it — they come from forced-choice letter logits. "Positions carried" is a cosine similarity in the model's own representation space, computed at run time and published per answer; it is a proxy for "which position is this answer", not a parse. 4. **The ground truth is a scholarly consensus-of-record, not my own.** I transcribed positions from Mughniyya's *Five Schools* and the works it cites. Where that source and other summaries differ, I followed the source and recorded it in the item's `sources` field. A reader who disagrees can substitute their own positions: the harness derives every gold label from the data, and no gold label is hard-coded anywhere. 5. **Agreement levels are coarse.** I grade four schools plus Jaʿfārī as equally weighted. Real-world weighting (by population, by region, by date) is not modelled, so "majority" here means "held by the larger number of the schools listed", not "held by most Muslims". 6. **Class E is a proxy for "a domain the model knows".** Python 2 versus 3 is a *procedural* difference with one right answer, which is the cleanest available stand-in for a well-known factual parameter. A purist would want a second such control in another domain; I did not have the bandwidth and say so rather than implying I did. 7. **One tokenizer-level observation is deliberately not claimed.** I measured MiniCPM5-2B's tokenizer and found its 130,560-token vocabulary contains **35** Arabic units, all single letters, with 33% of plain-Arabic tokens and 53% of vocalised-Arabic tokens falling back to raw bytes — so religious vocabulary is reassembled from byte fragments rather than units. That is a real finding about *this* model's representation, and the full measurement is published (`results/repr_probe.json`, `figures/fig5_tokenizer.png`, `code/repr_probe.py`), but it is a **separate claim** from the behavioural one and the behavioural one does not depend on it, so I have kept the study English-only and left the tokenizer result out of the argument rather than using it as if it explained the behavioural findings. ![Token cost by script, and the share of tokens that are raw byte fragments](figures/fig5_tokenizer.png) 8. **Single-run, greedy decoding.** No sampling variance is reported; temperature and top-p are unset. Repeated sampling would characterise the *distribution* of the collapse rather than its most likely instance. 9. **I did not run anything larger.** Every model here is under 4B parameters. The claim I make is that the blind spot is *present and structured* in this class of models and that its fingerprint (the dissociation between noticing conditionality and being able to resolve it, and the label-without-content failure) is what to look for in larger ones. Testing it on a 30B+ model is the obvious next step, and the bucket contains the harness for it. 10. **The post-training ablation is not in here.** Because the comparison set already spans three labs and 1.7B–4B parameters, I spent the remaining bandwidth on the cross-model replication rather than on the Base → SFT → RL ablation of the primary model, which the checkpoint series makes possible. It is the most interesting next experiment I would run, and `code/run_model.py --model minicpm5-2b-base` is the command. 11. **The base model is prompted differently.** Were the ablation to be run, the base checkpoint has no chat template, so it would receive a plain instruction prefix instead of a chat-formatted prompt. That is the fairest available comparison but it is not a like-for-like prompt, and the comparison would need to be read with that in mind. --- ## 9. Citing and reuse The code, dataset, figures and results in this bucket are released under the Apache-2.0 licence (see `LICENSE`), including the model weights in `model/`, which are unmodified OpenBMB `MiniCPM5-2B` checkpoints (also Apache-2.0). The ground-truth positions in `dataset/msq49.jsonl` are transcribed from *The Five Schools of Islamic Law* by Sayyid Muhammad Jawad Mughniyya; please cite the source works recorded per item in `sources` if you reuse them.