Buckets:

aaaEpalea
/

Evidence-Composition-Fidelity-Fatima-Fellowship

Files
xet
aaaEpalea/Evidence-Composition-Fidelity-Fatima-Fellowship / README.md
aaaEpalea
29 days ago
preview code
download
raw
39.4 kB
---
license: mit
task_categories:
  - text-generation
tags:
  - llm-evaluation
  - calibration
  - uncertainty
  - evidence-composition
pretty_name: Evidence Composition Fidelity - Blind Spot Diagnosis
---
# Evidence Composition Fidelity: A Blindspot in How LLMs Compose Evidence

## 1. The Blindspot

Part of my job at the Nigeria Revenue Service (the Service), the apex tax agency in Nigeria,
aside from heading my unit, is to build and deploy AI systems and agents used in the work we do
in our unit and in the Service at large. However, a recurring problem I have come across in my
work and unable to concretely solve for so far is that the systems/agents we have built find it
hard to correctly compose evidence. For context, we normally work with streams of evidence,
which includes filings, reports, correspondence, etc from companies. We pass these streams of
evidence to the AI and then ask it to reason over that stream, not just answer a fixed factual
question. In that setting, the main pain points/blind spots we keep coming across over and over
again, regardless of which LLM model or agent framework we use are:

- Order robustness is taken lightly: we noticed that more recent data always tends to override
  older data by default, even when the older data is not wrong or specifically superseded. The
  data has just been said recently, or last, in the conversation compared to the other data in
  the same conversation. The AI at most times happily lets the last thing it read/saw dominate
  its answer without caring whether or not that is the right thing to weight most heavily in the
  conversation.
- Dependence discounting is basically absent: in our use cases, a company might report the same
  figure in three or more different places (e.g. in a filing, a cover letter, a supporting
  schedule, an email correspondence, etc) and the AI system most often times than not treats the
  many different reports of the same figure as many confirmations (from independent sources)
  when it is really one number, repeated in different places by the same entity (in our case
  same company). Real independent corroboration and mere repetition get treated the same way by
  most AI models/agent frameworks.
- Temporal override is unreliable: in some cases, a company files a correction. For example,
  this year's numbers supersedes the numbers that were reported last year or the initial estimate
  is later revised. An agent should update its working belief and take the correction to account
  by correctly overriding the old numbers with the new numbers. However, this is sadly not the
  case. As we have noticed many times, corrections often get lost, especially once everything
  else is discussed in-between; the model can (and usually does) drift back toward the earlier,
  wrong number that has been overridden.
- Numerical fidelity breaks down under pressure: given the sector we work in (taxation and
  finance), an exact figure like ₦4,189,204.20 must, at all times stay exactly ₦4,189,204.20,
  especially for tax purposes, if we are to get the right reports and risk analyses out. But
  models often round it, "clean it up", or drift toward a rounder or more typical-looking number.
  In some worse cases, when a value is missing or redacted in the company's document, a model
  will sometimes just invent a number that is plausible in its place instead of flagging it or
  mentioning it as missing in its answer, which is close to the worst possible failure mode in a
  compliance context.

The above blindspot has made the systems we have built non-trustworthy and undeployable, at
least not without human oversight, given that compliance failure could lead to wrongful
taxation, which in turn could lock the Service in unending litigations, which in turn would lead
to loss of revenue to the federal government.

Sadly, none of the above blindspot evaluations show up in standard LLM benchmarks, which mostly
test whether a model knows a fact it was trained on or given, i.e. a fixed, single-shot lookup.
The blindspot I am describing is different. It is about how a model composes a live stream of
evidence into a correct belief, as that evidence arrives, especially when that evidence may
update and sometimes contradict itself or existing evidence, which is exactly what happens in a
real tax administration workflow (and perhaps, I'd argue, many other fields including physical
sciences, finance, sports, legal review, medical use cases, journalism, etc). I call this
evidence composition fidelity, and this evaluation is built to test it directly.

## 2. Evaluating the gap

Model: Qwen/Qwen2.5-1.5B-Instruct (1.5B parameters, within the 0.6–6B range), loaded and run in
Google Colab on a free T4 GPU. I chose this model because it is a small instruction-tuned model
from a family of models whose larger versions are in active use and frontier research. I believe
that if a composition failure shows up even at this scale in a model family that is well-regarded,
it will be reasonable to bet that the same underlying tendency is present, maybe harder to
elicit, in the larger models people actually deploy. My claim is about the shape of the failure.
I am not saying that a 1.5B model's exact numbers transfer unchanged to a 70B version.

Design: I built a synthetic dataset (generate_dataset.py or the 4th cell of the work notebook,
seeded for reproducibility) around four batteries, one per blind spot discussed above:
- Order robustness battery: we generate this battery using a fictional entity (e.g.
  Corvantex-OR7) which gets one confirming and one disconfirming piece of evidence of
  designed-comparable strength. I ask for a confidence score after hearing them in one order,
  then run an independent copy of the same conversation with the order reversed, and then I
  compare the confidence scores.
- Dependence discounting battery: the same fictional entity as above gets either one source,
  three independent confirming sources, or three dependent sources, i.e., the text of the
  sources makes it clear that the second and the third are just repeating the first.
- Temporal override battery: in this battery, a claim is reported true in 2019, then a report
  in 2023 overturns that of 2019. The report in 2023 is unambiguous with the overturning. For
  this, I ask for confidence "as of 2019" and "right now", sometimes I insert an irrelevant
  sentence between the correction and the question to test whether the correction survives being
  crowded by something unrelated.
- Numerical fidelity battery: in this case, real, exact values across three fields (Chemistry,
  Physics and Finance), in four categories are generated. The Categories are as follows:
  1. an ordinary control value;
  2. a real-but-unusual value near the famous constant (evaluating the question does the model
     drift toward the constant?);
  3. a value with many decimal places (evaluating the question does it get truncated?);
  4. and a value replaced with an explicit missing-data placeholder (evaluating the question does
     the model invent a number or does it say the number is missing?).
My choice of fictional entity names for the first three batteries (I also call them the epistemic
batteries) above is because I believe this makes the evaluation more truthful as the model will
have no pretrained prior about such fictional entities. Therefore, any change in the model's
answer has to come from how it processed the evidence I gave it in that specific conversation,
not from some prior belief it already holds due to its trained parameters. In contrast, I use
real, well-known constants and values in the numeric battery since the effect I am trying to
evaluate there (i.e. drift towards a "typical" number) only exists if such a typical value
actually exists in the model's training data.
The structure of the evaluation is such that every battery was run under a 2x2 design, which are:
- 2 prompt styles (a plain instruction [weak], vs. an explicit "don't round, don't blend, don't
  invent" prompt rider [strict]) X
- 2 context textures (the model sees one piece of evidence in isolation [isolated], vs. embedded
  in a fuller stream of evidence [sequential]).
This was the most important methodological decision I made in this whole study because it lets
me tell the difference between "a shallow prompting problem that a better and stricter system
prompt fixes" and "a deeper compositional failure that survives even after being explicitly told
not to happen explicitly". I ran 15 items/pairs per condition for the three epistemic batteries
and 8 numeric variants across 3 domains per condition. This makes a total of 816 model
generations in total. I then computed 95% bootstrap confidence intervals on every headline
result value, so that I could tell which differences were actually or likely real and not just
sampling noise at this sample size.
### What I found
1. Order robustness gets worse, not better, under the strict instruction, when evidence is
   isolated: Across all the four conditions the model shows a bias for recent data. It leans
   towards whichever evidence piece it heard last. But the largest swing of any condition (64.9
   points of confidence, 95% CI: 51.3–78.3) shows up under the strict prompt with isolated
   evidence, not the weak/sequential default. The interval doesn't overlap the best performing
   condition, which is the weak+sequential (24.0, CI 6.0–44.0) at all. The fact that we tell the
   model explicitly to reason only from the evidence given to it did not fix this. If anything,
   it correlated with the worst outcome I measured. The fact that the naive fix (a better and
   stricter instruction) does not touch this failure mode is the single most surprising and, I
   believe, most important result in this battery.
2. Temporal corrections are fragile, and this fragility is specific to strict prompting: under
   the strict prompt, especially the case whereby we add one content-free, irrelevant sentence
   between an unambiguous correction and the question, we notice that the model's accuracy at
   getting the direction right drops measurably (e.g. strict+sequential: 0.73 direct → 0.33 with
   distractor, CIs barely overlapping). Under the weak prompt, the effect is noisier and in one
   condition, it even reverses. This maps directly onto the NRS scenario that motivated this
   study, that is, a filed correction should not be dislodged by whatever unrelated thing happens
   to come up next in a case file, and this small model shows exactly that vulnerability.
3. The model invents specific values for data explicitly flagged as missing, at a rate around
   one-third to two-fifths, in every condition tested: when a value is replaced with an explicit
   placeholder like [ERROR_DATA_UNAVAILABLE], the model still, roughly a third of the time,
   states a specific number instead of reporting that the data is missing (rates from 0.333 to
   0.417 across all four conditions, all confidence intervals clearly above zero). This is the
   most direct synthetic analogue of the exact behavior I described from my own work, which is
   when a model quietly fills a plausible-looking figure where the real one should have been
   reported as or flagged absent.
4. Dependence discounting revealed two different failures hiding behind one number. Under the
   strict prompt with isolated evidence, checking gain_independent and gain_dependent separately
   shows the negative discounting gap is not from over-trusting copies. Confidence from three
   copies of one source stays roughly flat (gain_dependent, 95% CI −10.0 to 0.0). The negative
   gap comes from gain_independent itself being reliably negative (95% CI −30.3 to −3.3): seeing
   three genuinely independent confirming sources under a strict instruction actually lowers the
   model's confidence below its one-source baseline. Under the weak prompt, both gain_independent
   and gain_dependent are reliably positive in every condition tested (all four confidence
   intervals exclude zero), so the model is not simply failing to react to evidence there, as I
   first hypothesized. It shows a real, if small and only partially reliable, edge toward
   weighting independent sources more than copies. The strict+sequential condition remains
   inconclusive on which component drives its gap, since both intervals cross zero at this
   sample size.
5. Numerical drift and truncation were rarer and less conclusive at this sample size: Cat2
   canonical-value drift and Cat3 decimal truncation each appeared in only one of four
   conditions, on a small base of 24 items, with confidence intervals crossing zero. This battery
   is worth a much larger numeric sample before I can draw a firm conclusion in it, unlike the
   above three findings.
### Important Note on Methodology
The model actually frequently deviated from the exact requested output format (e.g. writing
"Confidence: 75" instead of the literally requested "CONFIDENCE: 75"), especially under the weak
prompt. However, I made parsing the model output tolerant of these deviations, since a strict
parser would have silently and non-randomly dropped weak prompt responses, thus biasing the
whole comparison, which was one of our main goals. For the numeric battery specifically, the
model frequently wrote a full narrative report and never reached the required footer restating
its values; when that happened, I gave it one follow-up turn with its own reply pre-filled as
REPORTED_VALUE[CATn]:, thus forcing the format while leaving the value entirely up to the model
to decide (generate). Every row is flagged as parsed on first try or rescued, and both rates are
reported in full in results/summary_metrics.json. This is important so nothing is hidden in the
numbers reported in this repo.
## 3. Proposed path forward (speculative)
The failures discussed above come from asking one ordinary LLM to play every role at once.
Specifically, the model played the roles of reading evidence, judging its reliability, tracking
when it arrived and whether it's a duplicate, and composing a final belief informally, inside a
single context window, with no explicit, persistent representation of "what I currently believe"
that exists outside that one conversation. My proposed direction, which I'd want to further test
include, but not limited to the following:
1. Decompose composition into an explicit pipeline: Instead of one model call doing everything,
   I propose to test different models handling separate steps. For example, I can dedicate a
   model to be an extractor that tags each piece of evidence with whether it supports or
   contradicts a claim, whether it's independent or a copy of something already seen, and when
   it arrived; and another model that is in charge of a distinct composition step that combines
   those tags into an updated belief. My guess is that even a small, fixed formula for combining
   well-tagged evidence would outperform one model doing extraction and composition together and
   that a learned composition step, trained to be robust to noisy extraction, would do better
   still.
2. Give the model (or system) a persistent, external belief state: currently, I am making the
   model re-derive its position from the full raw conversation every time but I am proposing a
   separate external belief state which the model retrieves from and acts on by updating. In a
   tax-agency setting this looks like a running case record (current belief, with a provenance
   trail) that gets updated incrementally as new filings arrive which is usually what is done.
   As opposed to re-reading the entire case file from scratch on every question, which only
   happens in very rare cases. My hypothesis is that fidelity should degrade more slowly as the
   case file grows if belief lives in a compact, explicitly-updated record instead of being
   re-derived from raw context each time.
3. Train specifically on redundancy and correction, not just single-document QA: this is a
   concrete, practical fine-tuning idea. I propose to build a data curation strategy around
   explicitly labeled examples of "these two documents say the same thing, don't double-count it"
   and "this document supersedes that one, don't drift back." Current instruction-tuning data
   likely has very little of this, since most QA and instruction data is built around single,
   static documents rather than corrected or duplicated streams, which would explain why this
   gap is invisible to standard benchmarks in the first place.
4. On numerical fidelity specifically, I'd want to test whether simply training a model to
   refuse to state a number it wasn't explicitly given (a stronger, more calibrated "I don't
   know" for the missing-data case) closes most of the imputation gap, since it doesn't obviously
   require touching the composition pipeline at all. This might be a much cheaper, narrower fix
   than the architectural changes above.
5. Build a much larger, open benchmark from this pilot's design, and publish it. This pilot's
   four batteries (order, dependence, temporal, numerical fidelity) are a narrow, single-model
   test of a bigger idea: a controlled benchmark where each way evidence composition can go
   wrong is varied on its own, with known ground truth, so results are not confounded by testing
   several things at once. A natural next step is extending this design along axes this pilot
   does not yet cover, for example how a model handles a genuinely novel entity against one it
   has already seen, how much fidelity survives when the model has to compress or summarize its
   own history under a fixed memory budget, and evidence that arrives as more than text, for
   example a scanned report alongside a written correction. I would also want to run this across
   several models, not just one, and publish the resulting dataset and evaluation code as an open
   benchmark other researchers can build on and compare against.
None of the proposed paths forward above is proven here. They are all speculative. However, this
evaluation shows that the failures exist and persist under an explicit strict instruction; it
does not test whether any of the fixes above would actually work. That's the next, larger piece
of work.
## Metrics glossary
Plain-language definitions of every term used above and in the tables, in the order they first
appear.
- Confidence score - a 0-100 number the model states for how strongly it believes a claim,
  extracted from its response text.
- Prompt style (weak / strict) - weak is a plain, generic instruction ("write a summary of this
  data"); strict adds an explicit zero-tolerance rider not to round, blend, or invent. Crossing
  both against context texture is what lets a result be read as "fixed by a better instruction"
  vs. "survives an explicit instruction not to happen."
- Context texture (isolated / sequential) - isolated means the model sees only the one piece of
  evidence relevant to the current question; sequential means that evidence is embedded in a
  fuller, sometimes cluttered stream of other material.
- abs_diff (order robustness) - the absolute difference in stated confidence between hearing the
  same confirming and disconfirming evidence in one order vs. the reverse. 0 = perfectly
  order-invariant; higher = more sensitive to order alone.
- signed_diff - the same difference, keeping the sign. Consistently negative means the model
  leans toward whichever evidence it heard last (recency bias); consistently positive means it
  leans toward whichever it heard first (primacy bias).
- Cold baseline / zero-evidence prior - the model's stated confidence in a claim with no
  supporting evidence at all. Used both as a sanity check (a fictional entity should score near
  50, i.e. a coin flip) and as the normative target for the order-robustness Brier score.
- Brier score - (stated probability - target probability)^2, averaged over items. 0 is a perfect
  score; higher is worse. It scores a stated confidence against a specific target, rather than
  only comparing two conditions to each other.
- Order-robustness Brier target - each entity's own measured cold-baseline confidence, since one
  confirming and one disconfirming piece of evidence of comparable strength should, if reasoned
  about correctly, leave belief close to where it started.
- gain_independent / gain_dependent / discounting_gap (dependence discounting) - gain_independent
  is the increase in confidence from 3 independent confirming sources over 1 source;
  gain_dependent is the same for 3 sources that are explicitly copies of one another.
  discounting_gap = gain_independent - gain_dependent. A large positive gap means the model
  correctly treats copies as weaker evidence than independent corroboration; a gap near or below
  zero means it does not.
- Dependence Brier - the squared gain_dependent value (as a probability), since a Bayes-ideal
  updater should gain exactly zero confidence from redundant copies of one source.
- correct_direction (temporal override) - whether the model correctly rates its "as of the
  original date" confidence higher than its "right now" confidence, after an explicit,
  unambiguous correction. distractor_condition (direct / with_distractor) marks whether an
  irrelevant, content-free sentence was inserted between the correction and the question.
- Temporal Brier - squared deviation from two fixed target values (then = 80, now = 15), chosen
  to reflect how convincing the designed evidence is meant to be. These are a stated design
  assumption, not derived ground truth.
- Cat1 / Cat2 / Cat3 / Cat4 (numerical fidelity) - four ways an exact numeric value can be
  corrupted when restated: Cat1 (control) is an ordinary value (any error is the baseline noise
  floor); Cat2 (latent pull) is a real but slightly unusual value near a well-known constant
  (does the restated value drift toward the constant?); Cat3 (awkward decimal) is a value with
  many decimal places (does the tail get truncated?); Cat4 (obfuscated) is a value replaced with
  an explicit missing-data placeholder (does the model invent a specific number instead of
  reporting it as missing?). Cat1-3 can only be measured under sequential, since those categories
  require the model to have actually been shown the value.
- 95% bootstrap confidence interval (CI) - the range that a metric's value would plausibly fall
  in if the same-sized sample of items were redrawn at random, estimated by resampling the
  existing results (with replacement) 2,000 times and taking the middle 95% of the resulting
  means. An interval that excludes 0, or two conditions whose intervals don't overlap, is
  evidence the difference is unlikely to be sampling noise at this sample size; an interval that
  straddles 0, or heavily overlapping intervals, means the point estimate should be treated as
  suggestive only.
- parsed / exact_format / rescued - bookkeeping on how each response was scored. parsed: whether
  a usable confidence/value could be extracted at all. exact_format: whether the response used
  the literally requested format (e.g. CONFIDENCE: 75) rather than a variant the parser was made
  tolerant of (e.g. Confidence: 75). rescued (numeric only): whether the model failed to state
  its restated values in its first response and needed one follow-up turn, with its reply
  pre-filled to force the required format while leaving the value itself up to the model.
## Appendix A: Full results heatmap
![Evidence-fidelity failure rate by condition](eval_summary_heatmap.png)
Every cell is a failure rate or normalized error (higher = worse), except the dependence-gap row,
where lower is worse (see the key below). Blank cells under isolated for Cat1-Cat3 are expected,
not missing data: that condition only ever shows the model the obfuscated Cat4 row in isolation,
so Cat1-3 fidelity can only be measured in sequential, where the model sees the full data table.
Row key (exact values, from results/summary_metrics.json):

| Row | weak+isolated | weak+sequential | strict+isolated | strict+sequential | What it measures |
|---|---|---|---|---|---|
| Order robustness (abs conf. diff, /100) | 0.423 | 0.240 | 0.649 | 0.323 | Mean absolute swing in stated confidence when the same two pieces of evidence are heard in reverse order. 0 = perfectly order-invariant. |
| Order Brier (vs. own cold prior) | 0.255 | 0.157 | 0.331 | 0.391 | Squared deviation of each ordering's confidence from that entity's own zero-evidence baseline confidence. |
| Dependence gap (lower = worse, /100) | 0.005 | 0.020 | -0.130 | -0.167 | (confidence gain from 3 independent sources) - (confidence gain from 3 copies of 1 source). |
| Dependence Brier (excess belief from copies) | 0.021 | 0.003 | 0.017 | 0.067 | Squared confidence gain from 3 copies of a single source, which should be exactly 0 for a Bayes-ideal updater. |
| Temporal error rate (direct) | 0.333 | 0.467 | 0.200 | 0.267 | 1 - (fraction correctly rating pre-correction confidence higher than post-correction confidence). |
| Temporal error rate (with distractor) | 0.533 | 0.333 | 0.467 | 0.667 | Same as above, but with one irrelevant sentence inserted between the correction and the question. |
| Cat1 baseline error rate | - | 0.000 | - | 0.000 | Fraction of ordinary control values reported incorrectly at all (the noise floor). |
| Cat2 drift toward canonical value | - | 0.000 | - | 0.125 | Fraction of near-a-famous-constant values that drifted toward the constant instead of the actual given value. |
| Cat3 decimal truncation rate | - | 0.083 | - | 0.000 | Fraction of many-decimal values whose tail was cut off. |
| Cat4 imputation rate | 0.333 | 0.333 | 0.333 | 0.417 | Fraction of explicitly-missing values for which the model invented a specific number instead of reporting it as missing. |

## Appendix B: Full bootstrap confidence intervals

95% bootstrap CIs (2,000 resamples) on every metric above, from results/bootstrap_ci.json and
results/dependence_gains_breakdown.csv. n is the number of independent items/pairs the interval
is built from (15 for the three epistemic batteries, 24 for numeric).
Order robustness
| Condition | n | mean abs_diff | 95% CI |
|---|---|---|---|
| strict + isolated | 15 | 64.867 | 51.333 - 78.333 |
| strict + sequential | 15 | 32.333 | 21.667 - 43.333 |
| weak + isolated | 15 | 42.333 | 23.325 - 62.667 |
| weak + sequential | 15 | 24.000 | 6.000 - 44.000 |

| Condition | n | mean Brier | 95% CI |
|---|---|---|---|
| strict + isolated | 15 | 0.331 | 0.229 - 0.435 |
| strict + sequential | 15 | 0.391 | 0.227 - 0.551 |
| weak + isolated | 15 | 0.255 | 0.184 - 0.328 |
| weak + sequential | 15 | 0.157 | 0.082 - 0.259 |

Dependence discounting

| Condition | n | mean gap | 95% CI |
|---|---|---|---|
| strict + isolated | 15 | -13.000 | -24.000 - -3.333 |
| strict + sequential | 15 | -16.667 | -41.333 - 5.008 |
| weak + isolated | 15 | 0.467 | 0.000 - 1.202 |
| weak + sequential | 15 | 2.000 | 0.000 - 5.000 |

| Condition | n | mean Brier | 95% CI |
|---|---|---|---|
| strict + isolated | 15 | 0.017 | 0.000 - 0.050 |
| strict + sequential | 15 | 0.066 | 0.007 - 0.138 |
| weak + isolated | 15 | 0.021 | 0.005 - 0.039 |
| weak + sequential | 15 | 0.003 | 0.001 - 0.005 |

Dependence discounting - gains broken down separately (follow-up check for finding #4)

| Condition | n | gain_independent | 95% CI | gain_dependent | 95% CI |
|---|---|---|---|---|---|
| strict + isolated | 15 | -16.333 | -30.342 - -3.333 | -3.333 | -10.000 - 0.000 |
| strict + sequential | 15 | -19.000 | -45.667 - 6.667 | -2.333 | -15.008 - 11.675 |
| weak + isolated | 15 | 8.000 | 2.658 - 14.008 | 7.533 | 2.000 - 14.000 |
| weak + sequential | 15 | 4.667 | 1.667 - 8.000 | 2.667 | 0.667 - 5.333 |

Temporal override

| Condition | n | correct-direction rate | 95% CI |
|---|---|---|---|
| strict + isolated + direct | 15 | 0.800 | 0.600 - 1.000 |
| strict + isolated + with_distractor | 15 | 0.533 | 0.267 - 0.800 |
| strict + sequential + direct | 15 | 0.733 | 0.467 - 0.933 |
| strict + sequential + with_distractor | 15 | 0.333 | 0.133 - 0.600 |
| weak + isolated + direct | 15 | 0.667 | 0.400 - 0.867 |
| weak + isolated + with_distractor | 15 | 0.467 | 0.200 - 0.733 |
| weak + sequential + direct | 15 | 0.533 | 0.267 - 0.800 |
| weak + sequential + with_distractor | 15 | 0.667 | 0.400 - 0.933 |

| Condition | n | mean Brier | 95% CI |
|---|---|---|---|
| strict + isolated + direct | 15 | 0.112 | 0.070 - 0.172 |
| strict + isolated + with_distractor | 15 | 0.241 | 0.170 - 0.308 |
| strict + sequential + direct | 15 | 0.172 | 0.104 - 0.239 |
| strict + sequential + with_distractor | 15 | 0.240 | 0.156 - 0.333 |
| weak + isolated + direct | 15 | 0.189 | 0.126 - 0.249 |
| weak + isolated + with_distractor | 15 | 0.203 | 0.136 - 0.266 |
| weak + sequential + direct | 15 | 0.255 | 0.206 - 0.296 |
| weak + sequential + with_distractor | 15 | 0.156 | 0.104 - 0.212 |

Numerical fidelity (Cat1-3 only measurable under sequential; see Appendix A)

| Metric | Condition | n | mean | 95% CI |
|---|---|---|---|---|
| Cat1 error | strict + sequential | 24 | 0.000 | 0.000 - 0.000 |
| Cat1 error | weak + sequential | 24 | 0.000 | 0.000 - 0.000 |
| Cat2 drift toward canonical | strict + sequential | 24 | 0.125 | 0.000 - 0.292 |
| Cat2 drift toward canonical | weak + sequential | 24 | 0.000 | 0.000 - 0.000 |
| Cat3 truncated | strict + sequential | 24 | 0.000 | 0.000 - 0.000 |
| Cat3 truncated | weak + sequential | 24 | 0.083 | 0.000 - 0.208 |
| Cat4 imputed | strict + isolated | 24 | 0.333 | 0.167 - 0.500 |
| Cat4 imputed | strict + sequential | 24 | 0.417 | 0.208 - 0.625 |
| Cat4 imputed | weak + isolated | 24 | 0.333 | 0.167 - 0.542 |
| Cat4 imputed | weak + sequential | 24 | 0.333 | 0.167 - 0.542 |

## Appendix C: Cold-baseline results

The model's stated confidence with zero supporting evidence, by source battery and condition
(results/summary_metrics.json -> cold_baseline_mean_confidence_by_condition). A well-behaved,
uncontaminated fictional-entity design would land close to 50 (a coin flip) in every cell; the
spread below shows this did not fully hold, which adds noise to the order-robustness Brier score
specifically, since that score is normed against each entity's own measured cold-baseline value.

| Source battery | weak+isolated | weak+sequential | strict+isolated | strict+sequential |
|---|---|---|---|---|
| order_robustness | 58.667 | 54.000 | 62.000 | 53.333 |
| dependence_discounting | 46.333 | 50.000 | 55.667 | 46.667 |
| temporal_override | 55.667 | 63.333 | 40.533 | 50.000 |
Mean shift from adding an irrelevant distractor alone (isolated -> sequential, zero real evidence
either way; nonzero = anchoring on noise not content): strict = -2.733, weak = 2.222.
## Appendix D: Full parse / format-compliance / rescue rates
From results/summary_metrics.json -> parse_rate_by_battery_condition. parsed = fraction of
responses a usable value could be extracted from at all; exact_format = fraction following the
literally requested format on the first try; rescued (numeric only) = fraction needing the
prefilled follow-up turn described in the Important Note on Methodology above.
| Battery | Condition | parsed | exact_format | rescued |
|---|---|---|---|---|
| cold | strict + isolated | 1.00 | 1.00 | - |
| cold | strict + sequential | 1.00 | 1.00 | - |
| cold | weak + isolated | 1.00 | 0.91 | - |
| cold | weak + sequential | 1.00 | 0.33 | - |
| dependence | strict + isolated | 1.00 | 1.00 | - |
| dependence | strict + sequential | 1.00 | 1.00 | - |
| dependence | weak + isolated | 1.00 | 0.38 | - |
| dependence | weak + sequential | 1.00 | 0.11 | - |
| numeric | strict + isolated | 1.00 | 0.71 | 0.25 |
| numeric | strict + sequential | 1.00 | 0.58 | 0.38 |
| numeric | weak + isolated | 1.00 | 0.29 | 0.54 |
| numeric | weak + sequential | 1.00 | 0.25 | 0.75 |
| order | strict + isolated | 1.00 | 1.00 | - |
| order | strict + sequential | 1.00 | 1.00 | - |
| order | weak + isolated | 1.00 | 0.53 | - |
| order | weak + sequential | 1.00 | 0.27 | - |
| temporal | strict + isolated | 1.00 | 1.00 | - |
| temporal | strict + sequential | 1.00 | 1.00 | - |
| temporal | weak + isolated | 0.98 | 0.98 | - |
| temporal | weak + sequential | 1.00 | 1.00 | - |

## Appendix E: Numeric fidelity by domain

The numeric battery spans three domains (8 items each per condition). Breaking Cat1-4 down by
domain, rather than only by prompt style and context texture, shows two of the four failure
categories are not spread evenly across domains:

- Cat2 (drift toward a canonical value) and Cat3 (decimal truncation) occurred exclusively in
  the chemistry domain in this run - zero instances in physics or finance. This sharpens finding
  #5 above: these two failure types are not just rare and condition-specific, they are so far
  domain-specific as well, on this small a sample.
- Cat4 (imputation of a missing value) occurred in every domain, but at different rates: physics
  shows the highest imputation rate in three of the four conditions tested, finance the lowest
  and most stable.

| Metric | Domain | Condition | n | Failure rate |
|---|---|---|---|---|
| Cat1 error | chemistry / finance / physics | strict + sequential | 8 each | 0.000 |
| Cat1 error | chemistry / finance / physics | weak + sequential | 8 each | 0.000 |
| Cat2 drift toward canonical | chemistry | strict + sequential | 8 | 0.375 |
| Cat2 drift toward canonical | finance | strict + sequential | 8 | 0.000 |
| Cat2 drift toward canonical | physics | strict + sequential | 8 | 0.000 |
| Cat2 drift toward canonical | chemistry / finance / physics | weak + sequential | 8 each | 0.000 |
| Cat3 truncated | chemistry | weak + sequential | 8 | 0.250 |
| Cat3 truncated | finance / physics | weak + sequential | 8 each | 0.000 |
| Cat3 truncated | chemistry / finance / physics | strict + sequential | 8 each | 0.000 |
| Cat4 imputed | chemistry | strict + isolated | 8 | 0.250 |
| Cat4 imputed | chemistry | strict + sequential | 8 | 0.500 |
| Cat4 imputed | chemistry | weak + isolated | 8 | 0.625 |
| Cat4 imputed | chemistry | weak + sequential | 8 | 0.125 |
| Cat4 imputed | finance | strict + isolated | 8 | 0.125 |
| Cat4 imputed | finance | strict + sequential | 8 | 0.125 |
| Cat4 imputed | finance | weak + isolated | 8 | 0.125 |
| Cat4 imputed | finance | weak + sequential | 8 | 0.250 |
| Cat4 imputed | physics | strict + isolated | 8 | 0.625 |
| Cat4 imputed | physics | strict + sequential | 8 | 0.625 |
| Cat4 imputed | physics | weak + isolated | 8 | 0.250 |
| Cat4 imputed | physics | weak + sequential | 8 | 0.625 |

At n=8 per domain-condition cell, these are directional observations, not statistically
established differences - a larger numeric sample per domain would be needed to confirm the
chemistry-specificity of Cat2/Cat3 or the physics/finance gap in Cat4 imputation rates.

## References

The following are cited as the specific existing literatures this diagnosis's four failure modes
sit closest to:

- Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022). Fantastically Ordered
  Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. Proceedings of
  the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 8086-8098.
  - order sensitivity in in-context learning; the closest existing-literature anchor for the
  order-robustness battery here.
- Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024).
  Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for
  Computational Linguistics, 12, 157-173. (arXiv:2307.03172)
  - documents primacy/recency-shaped position sensitivity in long-context use, directly relevant
  to the sequential-context conditions and the recency finding reported above.
- Kadavath, S., Conerly, T., Askell, A., et al. (2022). Language Models (Mostly) Know What They
  Know. arXiv:2207.05221.
  - calibration and self-evaluation of stated confidence in language models; the general
  calibration/uncertainty literature this study's Brier-scoring approach sits within.
- Graves, A., Wayne, G., Reynolds, M., et al. (2016). Hybrid Computing Using a Neural Network
  with Dynamic External Memory. Nature, 538(7626), 471-476. (Differentiable Neural Computer)
  - together with the earlier Neural Turing Machine (Graves, Wayne & Danihelka, 2014,
  arXiv:1410.5401), the external-memory-architecture lineage that this study's proposed
  persistent-substrate direction (Section 3) sits closest to.
- Kirkpatrick, J., Pascanu, R., Rabinowitz, N., et al. (2017). Overcoming Catastrophic
  Forgetting in Neural Networks. PNAS, 114(13), 3521-3526. (arXiv:1612.00796)
  - the continual-learning (Elastic Weight Consolidation) baseline family relevant to any
  persistent-belief-state proposal that must avoid overwriting prior knowledge when updated.
- Alchourron, C. E., Gardenfors, P., & Makinson, D. (1985). On the Logic of Theory Change:
  Partial Meet Contraction and Revision Functions. Journal of Symbolic Logic, 50(2), 510-530.
  - the formal (AGM) belief-revision literature that the temporal-override battery's notion of
  "correctly overturning a prior belief" is an informal, empirical instance of.

## Repository contents

| File | Description |
|---|---|
| oel_blindspot_eval.ipynb | The notebook that generated the dataset and produced every result above ||
| data/full_dataset.json | The full 816-item dataset (order/dependence/temporal/numeric/cold-baseline) |
| results/*.csv | Per-item scored results for each battery |
| results/summary_metrics.json | All summary statistics, including parse/rescue rates |
| results/bootstrap_ci.json | 95% bootstrap confidence intervals for every headline metric |
| results/dependence_gains_breakdown.csv | gain_independent vs. gain_dependent, separated (finding #4 follow-up) |
| results/numerical_fidelity_raw_responses.jsonl | Raw model responses for the numeric battery (first-pass and, where rescued, the rescue turn) |
| results/worst_condition.json | The auto-selected worst epistemic condition and the rule used to select it |
| eval_summary_heatmap.png | The combined 2x2 x failure-mode heatmap |
Reproducibility: MODEL_ID = Qwen/Qwen2.5-1.5B-Instruct, SEED = 0, N_PER_CONDITION = 15,
N_NUMERIC_VARIANTS = 8, TEMPORAL_TARGET_THEN = 80, TEMPORAL_TARGET_NOW = 15 (a stated design
assumption, not derived ground truth).

## Limitations

- Single model, single seed. These results characterize one 1.5B instruction-tuned model at one
  point in time. A cross-model comparison is planned but not run here.
- Sample size. 15 items/pairs per epistemic condition and 24 per numeric condition are enough to
  detect the headline effects above as distinguishable from noise (per the bootstrap intervals),
  but several secondary observations (Cat2, Cat3, the strict+sequential dependence result) remain
  underpowered.
- Hand-authored packets. All evidence packets use a fixed, hand-chosen evidence-strength design
  and fictional-entity naming scheme; results may not transfer to naturally occurring text.
- This is a diagnosis of monolithic, in-context composition only. No decomposed architecture is
  built or compared against in this submission - see Section 3.
Xet Storage Details

Size:: 39.4 kB
Xet hash:: fd73622eeeccadc554fd897eb8e167fa7058b803494914bbd2a76c51d82fa96b
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.