Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
pbhappliedsystems 
posted an update Oct 7
Post
144
quant_eval v7.22: what quantization does to agent behavior

Six paired runs, four models, Q4_K_M / Q5_K_M / Q8_0 vs the full-weight F16 lane on identical fixtures: 19,200 per-case results across 8 agent capabilities.

- Mistral-Nemo Q4_K_M: 5 of 8 families degraded (mixed_brief_json −28 pts, toolcall_only −21.5)
- Q5_K_M and Q8_0: no significant change in any family. The cliff is between Q4 and Q5.
- Ministral-3-14B vs BF16: Q4_K_M lost 1 family; Q5_K_M and Q8_0 lost none.

Paper: https://doi.org/10.5281/zenodo.22948597
Data (CC BY 4.0): https://doi.org/10.5281/zenodo.22009419
Arena: pbhappliedsystems/quant-eval-agent-arena

Which model should we measure next?

The stateful_followup cliff is one instruction, and it is not a Q4 thing.

I pulled D1, D3, D5 and D6 from your Zenodo deposit. All four match the sha256 in build_manifest.json.

Mistral-Nemo Q4_K_M loses 34 stateful_followup cases and gains 0. All 34 are boundary tier, and all 34 are the same miss. The turn-2 prompt says "Reverse items, preserve context, ... Do not add keys", and Q4 returns turn 2 without the context object. F16 never does: 0 of 144 boundary cases across its three lanes.

Qwen2.5-7B runs the other way. There F16 drops context on 34 of 48 boundary cases, Q4_K_M on 12. That is 22 of the 24 cases behind Q4's 24-to-0 win on that family.

Same tier, same key, opposite precisions. So on this fixture the family mostly measures one thing: keeping an object you were told to preserve while editing its neighbours. And a degradation-only count shows Mistral's half and hides Qwen's.

Have you run the boundary prompt without "Do not add keys"?

·

Thank you—your criticism was correct, and we tested it directly.
We ran a parallel ablation across Mistral-Nemo and Qwen2.5-7B, comparing each F16 baseline against Q4_K_M, Q5_K_M, and Q8_0. Each of 48 identical boundary scenarios was tested with the original Do not add keys wording, with that sentence removed, and with an explicit retention contract.
The original wording reproduced the asymmetry you identified. Mistral-Nemo Q4 passed 14/48 control cases while F16 passed 48/48. Qwen2.5-7B moved in the opposite direction: F16 passed 15/48 and Q4 passed 32/48. But every tested model and precision passed 48/48 when the phrase was removed, and 48/48 under explicit existing-key retention.
So the old result was not a general Q4 cliff. It was an instruction interaction that affected models and precisions differently.
We did not simply remove the constraint. We replaced that construction with a four-tier stateful family covering foundation, operational, stress, and boundary transformations. Retention and key admission are now stated separately, with an explicit allowed-key set. Calibration and locked evaluation splits are disjoint.
In the new 200-case production evaluation:

  • Mistral-Nemo F16 vs Q4_K_M: 150/200 vs 80/200, with 70 F16-only passes and no Q4-only passes.
  • Mistral-Nemo F16 vs Q5_K_M: 150/200 vs 150/200.
  • Mistral-Nemo F16 vs Q8_0: 150/200 vs 150/200.
  • Qwen2.5-7B F16 vs Q8_0: 145/200 vs 147/200, with 9 F16-only and 11 Q8-only passes; statistically inconclusive.
    One limitation remains visible: Mistral scored 0/50 in the new boundary tier in both lanes, so that tier is currently non-discriminating for this model. The Mistral Q4 degradation now comes from the other constructions rather than the original one-instruction boundary artifact.

Your comment materially improved the evaluation. Thank you for reproducing the deposited data and identifying the confound.

Your observation directly motivated this ablation and the subsequent revision of the stateful_followup evaluation. If I publish the updated methodology or results, may I acknowledge you by name and reference your Hugging Face comment? If so, please let me know your preferred name, affiliation, and profile or citation link.
If you would like to receive the results and available publication/ bundles, please contact me through pbhappliedsystems.com/contact and reference these run names:

  • Mistral_Nemo_Instruct_2407_20261009_033750
  • Mistral_Nemo_Instruct_2407_20261009_071015
  • Mistral_Nemo_Instruct_2407_20261009_091019
  • Qwen2.5_7B_Instruct_20261009_191201
  • Qwen2.5_7B_Instruct_20261009_215528
  • Qwen2.5_7B_Instruct_20261009_221516

The Mistral rerun is the part I trust most. It reproduced to the case.

The deposit has Mistral-Nemo Q4_K_M failing turn 2 on 34 of 48 boundary cases. 48 minus 34 is your 14/48.

Qwen2.5-7B moved a little. The deposit gives F16 14/48 and Q4_K_M 36/48 on that tier. Your rerun has 15 and 32. Is decoding greedy there? A 4-case wobble on one lane is worth knowing before anyone reads a win off a single run.

On the new 200, Mistral F16 at 150 with boundary at 0/50 means F16 passes every one of the other 150 cases. So 70 to 0 against Q4 is about as one-sided as a paired result gets, two-sided sign test p around 1e-21.

Qwen F16 vs Q8 at 9 and 11 discordant gives exact McNemar p = 0.82. Inconclusive is the right word.

The check I'd run next is the one that caught the first artifact. Do the 70 share one failure signature, or do they spread across foundation, operational and stress?

On the acknowledgement, thank you, that is kind. Let me come back to you on the details.

·

Dipankar — thank you again. Your criticism was substantiated. The earlier stateful_followup construction placed too much evidentiary weight on one boundary prompt and could conceal opposing operation-level effects behind a single family aggregate. I did not treat that as merely a reporting problem; I used it to redesign and revalidate the family before returning it to production.

Updated stateful evaluation design

The updated production evaluation now uses a locked, disjoint 200-case stateful split crossing:

  • Four primary operations:
    • reverse_primary_tags
    • rotate_queue_left
    • expire_decoy_record
    • raise_primary_priority
  • Four construction bands:
    • foundation
    • operational
    • stress
    • boundary

All 16 cells contain 12 or 13 unique paired cases. The 200 evaluation cases are semantically unique and have zero content overlap with the calibration split. The bands are explicitly described as construction strata, not as an empirically ordered difficulty scale.

Before promotion, I ran the proposed boundary ablation, decode-stability experiments, state-transition isolation experiments, and crossed-construction calibration on Mistral-Nemo and Qwen2.5-7B against Q4_K_M, Q5_K_M, and Q8_0. The revised design then received six full 200-case-per-family publication runs.

Decoding and the reported Qwen wobble

The current runs are not greedy.

For Qwen2.5-7B, both F16 and quantized lanes now use the publisher-declared decoding configuration:

  • temperature=0.7
  • top_p=0.8
  • top_k=20
  • repetition_penalty=1.05
  • Fixed seed 42 for stateful follow-up

The old deposit and these revised runs should not be pooled: the evaluation contract, stateful construction, and Qwen decoding contract changed.

There is strong repeatability evidence within the new contract. Qwen’s F16 baseline was independently executed in the Q4, Q5, and Q8 publication comparisons. Across all three runs:

  • Case-level pass/fail movement: 0/1,600
  • Raw-output movement: 0/1,600

Mistral-Nemo’s F16 baseline also reproduced exactly across its three precision comparisons:

  • Case-level pass/fail movement: 0/1,600
  • Raw-output movement: 0/1,600

That directly addresses the earlier concern about interpreting a small Qwen movement from one non-greedy run.

Mistral-Nemo F16 versus Q4_K_M

The redesigned stateful result is:

  • F16: 127/200
  • Q4_K_M: 22/200
  • F16-only passes: 105
  • Q4-only passes: 0
  • Difference: −0.525
  • Exact two-sided McNemar: p ≈ 4.93 × 10⁻³²

Most importantly, the 105 discordant cases do not share the original boundary-only signature. They are distributed across every construction band:

  • Boundary: 28
  • Foundation: 21
  • Operational: 30
  • Stress: 26

The largest band contains only 28.6% of the discordance.

The effect also separates by operation:

  • expire_decoy_record: −0.70
  • raise_primary_priority: −0.86
  • rotate_queue_left: −0.54
  • reverse_primary_tags: 0.00

The last result is itself important: both Mistral lanes scored 0/50 on reverse_primary_tags. The revised harness therefore exposes a baseline model weakness instead of automatically attributing every failure to quantization.

The Q4 loss is genuine under this contract, but it is not the former single boundary-instruction artifact.

Mistral Q5_K_M and Q8_0 did not reproduce the Q4 cliff:

  • Q5 stateful: F16 127/200, Q5 129/200, difference +0.010
  • Q8 stateful: F16 127/200, Q8 126/200, difference −0.005

Both are classified as preserved within margin.

Qwen2.5-7B F16 versus Q4_K_M

The redesigned family-level result is:

  • F16: 152/200
  • Q4_K_M: 144/200
  • F16-only passes: 27
  • Q4-only passes: 19
  • Difference: −0.040
  • Exact two-sided McNemar: p ≈ 0.302

The appropriate conclusion is inconclusive—not a Q4 win or a demonstrated Q4 loss.

The operation-level decomposition explains why the aggregate must not be interpreted alone:

  • reverse_primary_tags: −0.42
  • expire_decoy_record: +0.14
  • rotate_queue_left: +0.12
  • raise_primary_priority: 0.00

This is precisely the pattern the earlier evaluation could hide: the same quantized model degrades materially on one transition while improving on others.

By construction band:

  • Boundary: −0.021
  • Foundation: −0.173
  • Operational: +0.038
  • Stress: 0.000

Qwen’s other precisions were stable at the family level:

  • Q5_K_M: F16 152/200, Q5 160/200, difference +0.040, preserved within margin
  • Q8_0: F16 152/200, Q8 155/200, difference +0.015, preserved within margin

What changed in quant_eval

Since your comment, the following has been added or corrected:

  1. Publisher-declared Qwen decoding parameters are enforced for both local and Modal runners.
  2. Decoding controls and their effective backend values are recorded in the manifest.
  3. Seeded non-greedy repeatability is distinguished from guaranteed deterministic decoding.
  4. Decode-stability experiments were added before production promotion.
  5. Self-fed versus oracle-fed state-transition isolation was added.
  6. Initialization correctness, conditional-transition correctness, and end-to-end correctness are reported separately.
  7. The narrow stateful construction was replaced with the crossed four-operation × four-band design.
  8. Calibration and locked evaluation splits are disjoint.
  9. Every stateful publication row records operation_id, construction_band, and transition_load.
  10. Those fields are defined in the versioned public data dictionary.
  11. Reports disclose results by operation and construction band.
  12. Failure signatures include failed gates and normalized JSON-difference paths.
  13. Family-level effect concentration is reported so a narrow sub-capability cannot silently become a broad family claim.
  14. Exact paired McNemar inference remains the headline test; individual crossed cells are descriptive strata.
  15. Holm adjustment is applied across the reported family-level tests.
  16. Machine-readable claim lineage maps conclusions to rollup fields and statistical rules.
  17. Publication bundles are independently verified for checksums, schemas, telemetry joins, token reconciliation, statistics, and claim-lineage pointers.
  18. The revised production contract was validated with six full publication runs covering both models and Q4_K_M, Q5_K_M, and Q8_0.

Your observation therefore changed both the fixture design and the evidentiary limits of the resulting claims. The current harness no longer treats the original boundary behavior as a sufficient characterization of stateful-follow-up capability.

Publication runs and supporting evidence

The six publication run IDs are:

  • Mistral_Nemo_Instruct_2407_20261018_235327
  • Mistral_Nemo_Instruct_2407_20261019_040329
  • Mistral_Nemo_Instruct_2407_20261019_122449
  • Qwen2.5_7B_Instruct_20261019_153512
  • Qwen2.5_7B_Instruct_20261019_181307
  • Qwen2.5_7B_Instruct_20261019_201739

If you would like the complete publication bundle for any or all of these runs—including the 3,200 per-case rows, locked fixtures, telemetry, manifests, provenance, rollups, failure-signature analysis, data dictionary, claim lineage, and checksums—please identify the run IDs in a request through pbhappliedsystems.com/contact.

I would be interested to know whether the revised design leaves any part of your original concern unresolved, or whether anything else in the approach warrants closer examination.

And thank you: this was exactly the kind of external scrutiny the project needed.

Your numbers close at the cell level, which is the part I checked.

Mistral: the operation deltas times 50 are 35, 43, 27 and 0. That is your 105.
Qwen: -21, +7, +6 and 0 net to the -8.

The Mistral line I'd lead with is not -0.525. Both lanes score 0/50 on reverse_primary_tags, so F16 has 127 passes to lose and Q4 keeps 22 of them. 17%.

Qwen is where I think the aggregate hides the story. reverse_primary_tags nets -21 of 50. So at least 21 of the family's 27 F16-only passes sit in one operation, and the other three net +13 for Q4.

The part I don't think the 0/1,600 settles is the draw. Seed 42 reproducing seed 42 shows the pipeline is deterministic. At temperature 0.7 each case is still one sample, so it can't say how much of a paired discordance is sampling.

The cheap control: Qwen F16 at seed 43 against F16 at seed 42. If that alone disagrees on 40-odd cases, the 46 discordant pairs against Q4 are mostly the draw. If it disagrees on a handful, reverse_primary_tags is a real Q4 loss the family test averages away.

Which way does it come out?

·

Thank you—your distinction between same-seed reproducibility and cross-seed sampling stability was correct.

I ran the proposed same-artifact control on Qwen2.5-7B F16: seed 42 versus seed 43, with the artifact, 200 fixtures, backend, runtime, context, and all other decoding parameters held fixed at temperature 0.7.

The result was:

  • Seed 42: 156/200 passed
  • Seed 43: 163/200 passed
  • Discordant outcomes: 17/200
  • Seed-42-only passes: 5
  • Seed-43-only passes: 12
  • Exact McNemar p = 0.143
  • Raw outputs differed on 118/200 cases, but only 17 differences changed the scored outcome

By operation:

  • reverse_primary_tags: 49/50 versus 50/50; only 1 discordant case
  • raise_primary_priority: 50/50 versus 50/50; 0 discordant
  • expire_decoy_record: 36/50 versus 38/50; 4 discordant
  • rotate_queue_left: 21/50 versus 25/50; 12 discordant

So it comes out much closer to your “handful” case than the “40-odd” alternative—particularly for reverse_primary_tags, where seed variation moved only one scored outcome. Most cross-seed movement occurred in rotate_queue_left.

That said, I want to preserve an important evidence boundary. This control used quant_eval’s redesigned current stateful contract and calibration split. It does not retroactively decompose the 46 discordances in the original deposited comparison. An exact historical replay is no longer possible because the original hash-bound F16 artifact is unavailable and its exact upstream revision and conversion toolchain were not recovered. I therefore will not present the 17 as a correction factor for the historical result.

What the experiment does establish is that, under the current crossed production design, reverse_primary_tags is stable across seeds 42 and 43. The operation-specific signal is no longer hidden inside the family aggregate. quant_eval now reports operation, construction-band, and operation-by-band evidence directly, while explicitly separating raw-output movement from scored-outcome movement.

If you would like the complete machine-readable results and reports, please submit a request through https://pbhappliedsystems.com/contact and name the relevant run folders:

  • Qwen2.5_7B_Instruct_20261023_003226 — F16 seed-42 versus F16 seed-43 control
  • Mistral_Nemo_Instruct_2407_20261018_235327
  • Mistral_Nemo_Instruct_2407_20261019_040329
  • Mistral_Nemo_Instruct_2407_20261019_122449
  • Qwen2.5_7B_Instruct_20261019_153512
  • Qwen2.5_7B_Instruct_20261019_181307
  • Qwen2.5_7B_Instruct_20261019_201739

Your criticism materially improved the instrument. It led to the stateful redesign, operation-level reporting, disjoint locked evaluation split, transition-isolation testing, construction-diversity testing, repeated same-seed controls, and now the explicit cross-seed control.

If you would be willing, I would greatly appreciate your endorsement so that I can submit my whitepaper, The quant_eval Public Corpus: Behavioral Measurement of Quantization Degradation, to arXiv. I would be happy to send you the manuscript and endorsement request for your consideration.

rotate_queue_left is the row to build the next report around.

12 of 50 flipped between seeds 42 and 43. The other three operations flipped 5 of 150 between them.
At its pass rate, about 46 percent, two independent draws would disagree on roughly 25 of 50. You saw 12.
So the fixture carries most of the outcome, and the seed carries the rest.

That gives each operation its own noise floor.
Seeds alone moved rotate_queue_left from 21 to 25. A quant arm that moves it by 4 is inside that.
The same 4 on reverse_primary_tags, which flipped once, would be a real signal.

Agreed on not back-applying the 17 to the old 46. Different contract, different fixtures.

On the arXiv endorsement: thank you for asking. Let me come back to you on that.

What did your repeated same-seed controls show on rotate_queue_left? If that is near 0, the seed is the noise there, and that operation needs several seeds per arm.

·

Thanks—your distinction between same-seed reproducibility and sampling variation was correct, so I ran a dedicated repeated-seed F16-versus-Q4_K_M experiment.

First, the repeated same-seed controls on the production fixtures showed zero movement for rotate_queue_left: across three independent Qwen2.5-7B F16 production lanes using seed 42, all 50 case-level outcomes were identical. That indicates ordinary execution nondeterminism was effectively zero in those controls.

I then evaluated the current 200-case crossed stateful construction at five generation seeds—42, 43, 44, 45, and 46—with the same seeds applied independently to F16 and Q4_K_M. Each operation had 50 base cases at every seed.

The operation-level results were:

Operation F16 pass range Q4 pass range Q4−F16 delta range
reverse_primary_tags 50–50/50 35–37/50 −0.30 to −0.26
rotate_queue_left 14–16/50 26–29/50 +0.20 to +0.30
expire_decoy_record 37–37/50 43–44/50 +0.12 to +0.14
raise_primary_priority 50–50/50 50–50/50 0.00

Your expectation about rotate_queue_left being the seed-sensitive row was confirmed.

For F16, 11 of its 50 rotate_queue_left cases changed outcome across the five seeds; the largest disagreement between any seed pair was 7/50. For Q4_K_M, 26/50 cases were seed-sensitive, and the largest pairwise disagreement was 21/50.

However, the precision effect remained directionally stable: Q4_K_M exceeded F16 on rotate_queue_left at all five seeds, by 10–15 passes. The result therefore contains substantial case-level sampling movement, especially in Q4, but the observed Q4 advantage is not an artifact of selecting seed 42.

The other operations were more stable:

  • reverse_primary_tags: F16 remained 50/50 at every seed, while Q4 remained between 35/50 and 37/50. Q4 lost 13–15 cases at every seed and never had a Q4-only win in the paired comparisons.
  • expire_decoy_record: F16 remained 37/50, while Q4 remained between 43/50 and 44/50. Q4 gained 6–7 cases at every seed, with no F16-only wins.
  • raise_primary_priority: both remained 50/50 across all five seeds.

So the broader finding is that the stateful aggregate conceals four different behaviors: a stable Q4 degradation on reverse_primary_tags, a seed-sensitive but directionally stable Q4 improvement on rotate_queue_left, a smaller stable Q4 improvement on expire_decoy_record, and a ceiling tie on raise_primary_priority.

I agree that these effects should not be reduced to a single family-level delta or treated by mechanically subtracting a generic “noise floor.” The evaluation now reports operation-specific seed movement, pass-rate ranges, effect directions, and within-seed paired results separately.

The 1,000 rows per artifact are also treated as repeated measurements of 200 base cases—not 1,000 independent observations. The 20 operation-by-seed exact McNemar values are explicitly labeled unadjusted exploratory diagnostics; no pooled 1,000-row p-value or collection of 20 independent confirmatory claims is permitted.

The completed experimental evidence is:

Qwen2.5_7B_Instruct_20261024_052753

If you would like the full results, including the case-level CSV, manifest, fixtures, telemetry, and repeated-seed reports, you can request that run by name through https://pbhappliedsystems.com/contact.

The row I can't reconcile is Q4 at seed 42.

Your 10-19 production lane ran Qwen Q4_K_M at seed 42 and scored 144/200.
Summing your five-seed ranges, Q4 is at least 35+26+43+50 = 154 at every seed. Seed 42 included.
F16 replays: 50+15+37+50 = 152, same as production.

So F16 reproduces and Q4 moves by 10 or more, outside its own five-seed spread of 6.
Per operation, from your 10-21 deltas on that F16: reverse_primary_tags was 29, now 35 to 37. rotate_queue_left was 21, now 26 to 29. expire_decoy_record held at 44.

That also flips the family read. Production said -0.040, inconclusive. These five seeds put Q4 above F16 at every one.

Something other than the seed changed in the Q4 lane between 10-19 and 10-24. The GGUF file, the llama.cpp build, or how the seed reaches the sampler.

One more line worth putting in the report.
Q4's worst seed pair on rotate_queue_left disagreed on 21 of 50. At a 55 percent pass rate, two independent draws would disagree on about 25.
F16 tops out at 7, against about 21 for independent draws at its 30 percent.
So on that operation the fixture decides F16, and the seed mostly decides Q4. Quantization is adding sampling entropy there, not just moving the mean.

Does the 10-24 Q4 lane carry the same artifact hash as the 10-19 production one?

·

You are correct—the seed-42 row does not reconcile, and I checked the complete records rather than relying on the aggregate report.

The October 19 and October 24 Q4_K_M lanes use the same GGUF artifact: SHA-256 dcbe3199008880ff6cbdf058c5f054eb4fbf3cdc6a3b7bf6fbf249eea5137eae. They also use the same llama-server binary, F16 artifact, underlying 200 cases, prompts, expected outputs, context size, temperature, top-p, top-k, repetition penalty, and recorded seed.

Q4_K_M nevertheless moved from 144/200 in the production run to 155/200 at seed 42 in the repeated-seed experiment. At case level, 27 Q4 outcomes changed: reverse_primary_tags gained a net six cases, rotate_queue_left gained five, and the other two operations had no net change.

F16 reproduced 152/200 in aggregate, but it was not perfectly case-identical across these two contexts: two scored outcomes exchanged direction and 34 raw outputs changed. I therefore overstated the evidence when I described ordinary same-seed execution movement as effectively zero across the production and experimental contexts.

The known contextual difference is generation position. In the production run, stateful generation began after roughly 513 earlier model calls; in the targeted experiment it began immediately after preflight. Consequently, the five-seed experiment establishes directional consistency within that targeted context, but it does not isolate seed from execution position, server state, or same-seed runtime movement.

Your sampling-entropy interpretation is plausible, especially for rotate_queue_left, but I agree that it is not yet demonstrated. I am adding a fresh-process, repeated seed-42 control with counterbalanced call history so that same-seed movement, between-seed movement, and execution-position effects are reported separately. I will report those findings before making a stronger attribution.

Same hash, same binary, same seed, and F16 still changed 34 raw outputs. That points at the logits, not the draw.

Seeds 42 and 43 changed 118 of 200 raw outputs. Your two seed-42 F16 contexts changed 34.
If call position were advancing the sampler's RNG, those two numbers would look alike. So the seed reaches the sampler, and something upstream of it moved.

llama-server documents a candidate. cache_prompt defaults to true, and its README says logits are not bit-for-bit identical across batch sizes, so it "can cause nondeterministic results".
Which prefix is already cached depends on history. -np defaults to auto, and a request goes to the slot whose cached prompt matches it best (--slot-prompt-similarity 0.10).
After 513 calls from other families the slots hold different prompts than after preflight.

That would also explain the earlier 0/1,600. Three production lanes with the same call history get the same caches. That was same-history reproducibility, not same-seed.

The Q4 side is less symmetric than jitter should be.
27 flips at net +11 means 19 gains and 8 losses. Two-sided sign test p = 0.052.
Borderline, but float noise has no reason to favour the later context.

A cheap discriminator before the counterbalanced run: F16 and Q4 with cache_prompt off and -np 1, once in production order, once straight after preflight.
If raw movement goes to 0, it was the cache. If Q4 still moves, it is something else.

Where do the 27 Q4 flips sit in the stateful block? Bunched near its start, or spread evenly?

·

Thank you—you identified the right mechanism to investigate. The completed controls show that the October 19 versus October 24 discrepancy was caused by llama-server’s prompt-cache state interacting with execution history, not by a different artifact, binary, fixture set, or failed seed propagation.

First, the direct artifact answer remains yes:

  • Qwen2.5-7B Q4_K_M SHA-256 in both runs: dcbe3199008880ff6cbdf058c5f054eb4fbf3cdc6a3b7bf6fbf249eea5137eae
  • Same F16 artifact
  • Same llama-server binary
  • Same 200 base cases and prompts
  • Same context and decoding parameters
  • Same recorded seed, 42

The initial five-seed experiment was internally valid within its targeted, stateful-first execution context, but it was not directly comparable to the production-order run. Your cache hypothesis explains why.

1. Five-seed experiment

The five-seed experiment evaluated seeds 42–46 on the same 200 crossed stateful cases.

Operation F16 range Q4_K_M range Q4−F16 range
reverse_primary_tags 50–50/50 35–37/50 −0.30 to −0.26
rotate_queue_left 14–16/50 26–29/50 +0.20 to +0.30
expire_decoy_record 37–37/50 43–44/50 +0.12 to +0.14
raise_primary_priority 50–50/50 50–50/50 0.00

This established operation-specific seed sensitivity within that context. It did not explain why seed-42 Q4 scored 155/200 there but 144/200 in production order.

2. Counterbalanced execution-context experiment

I then ran three independent seed-42 realizations in each of two contexts:

  • Cold/stateful-first: stateful began at call 2.
  • Production order: JSON and JSON multistep preceded stateful, which began at approximately call 514 for F16 and 506 for Q4.

All three realizations were byte-identical within each arm.

Context F16 Q4_K_M
Cold/stateful-first 152/200 155/200
Production order 152/200 144/200

The production-order arm reproduced the October 19 production trajectory. The cold arm reproduced the later seed-42 trajectory.

Across contexts:

  • F16: 34/200 raw outputs changed, with one cold-only and one production-only pass.
  • Q4_K_M: 104/200 raw outputs changed, with 19 cold-only and eight production-only passes.
  • Q4’s exact two-sided paired p-value was 0.052239, matching your calculation.

The Q4 outcome changes were not bunched near the start of the stateful block:

Position Raw outputs changed Cold-only passes Production-only passes
1–50 24 3 3
51–100 29 4 1
101–150 24 5 3
151–200 27 7 1

The 27 pass/fail changes were therefore distributed 6, 5, 8, 8 across the four 50-case position bins. Raw changes extended from position 2 through position 194.

By operation:

Operation Cold Q4 Production Q4 Cold-only Production-only Raw changed
reverse_primary_tags 35 29 10 4 29
rotate_queue_left 26 21 8 3 36
expire_decoy_record 44 44 1 1 17
raise_primary_priority 50 50 0 0 22

That answered the execution-position question, but not yet the lower-level cause.

3. Backend-mechanism experiment

Your cache_prompt and slot-routing hypothesis prompted a six-arm experiment with three fresh-process realizations per arm—18 runs total:

  1. Cold, cache enabled
  2. Production order, cache enabled
  3. Production order followed by server restart
  4. Cold, prompt cache disabled
  5. Production order, prompt cache disabled
  6. Request-count-matched neutral warmup

The controls recorded:

  • Explicit prompt-cache policy
  • --parallel slot count
  • Server epoch
  • Sanitized slot snapshots
  • Preceding scored-call and token volume
  • Neutral-warmup volume
  • Top-20 pre-sampling log-softmax diagnostics

I did not capture the full vocabulary logits, so I am not claiming full-logit identity. The captured pre-sampling surface was sufficient to locate the first observable divergence.

Every arm was byte-stable across its three realizations.

Arm F16 Q4_K_M
Cold, cache enabled 152/200 155/200
Production order, cache enabled 152/200 144/200
Production order, then server restart 152/200 155/200
Cold, cache disabled 151/200 156/200
Production order, cache disabled 151/200 156/200
Neutral warmup, cache enabled 152/200 155/200

The discriminating comparisons were exact:

Cache enabled: cold versus production order

  • F16: 34 raw outputs changed.
  • Q4_K_M: 104 raw outputs changed.
  • The captured first-step pre-sampling probability surface changed in all 400 turn-level records for each runner.
  • The first captured divergence occurred at generation step 1 in all 400 records.
  • The selected first token changed in 25 F16 records and 68 Q4 records.

Cache disabled: cold versus production order

For both F16 and Q4_K_M:

  • Zero raw-output changes
  • Zero pass/fail changes
  • Zero captured pre-sampling probability-surface changes
  • Zero selected-first-token changes

Production order followed by server restart

The restarted server reproduced the cold trajectory exactly for both artifacts:

  • Zero raw-output changes
  • Zero pass/fail changes
  • Zero captured probability-surface changes

Neutral warmup

The neutral arm matched the production arm’s request counts—512 preceding calls for F16 and 504 for Q4—and used slightly greater token volume. It nevertheless reproduced the cold trajectory exactly.

That rules out request count or approximate token volume alone. The effect required content-specific cached prompt history.

So your proposed mechanism was correct: with prompt caching enabled, prior prompt content changed the pre-sampling probability surface. Repeating the same seed and same call history reproduced the same trajectory, but that was same-history reproducibility—not context-independent same-seed reproducibility.

4. Production correction and validation

quant_eval’s local llama-server production contract now enforces:

--no-cache-prompt
--parallel 1

The runner, telemetry, manifest, reporting, comparability contract, data dictionary, and publication validation all fail closed if the required state-isolation evidence is missing or inconsistent.

A 50-case-per-family calibration completed successfully with the corrected backend. The subsequent locked publication evaluation ran:

  • 200 cases per canonical family
  • 200 separately reported exogenous fuzz-replay cases
  • 3,600 scored CSV rows
  • 5,758 reconciled scored generation records

The publication bundle passed independent checksum, schema, telemetry-join, token-reconciliation, paired-statistics, effect-concentration, and claim-lineage validation with no blockers.

For stateful follow-up under the corrected production contract:

  • F16: 151/200
  • Q4_K_M: 156/200
  • F16-only passes: 18
  • Q4-only passes: 23
  • Delta: +0.025
  • Conclusion: inconclusive

That result belongs to the corrected v7.22.44 execution contract. I am not back-applying it to the older cache-enabled runs.

5. Separate Modal control

I also tested the hosted llama-cpp-python path separately because it does not use the local llama-server slot architecture.

For Qwen2.5-14B F16 and Q4_K_M, the backend called llm.reset() before every request. All 800 scored turn calls returned machine-readable proof that request state had been reset.

The artifacts, A100-40GB tier, llama-cpp-python 0.3.20 runtime, 8,192-token context, decoding parameters, seed, and 200 base cases matched the historical reference.

Lane Historical Request-reset Outcome changes Raw changes
F16 163/200 162/200 1 16
Q4_K_M 162/200 163/200 1 96

Both changed outcomes were on the same rotate_queue_left base case, in opposite directions.

This shows that request-state reset did not materially alter the capability conclusion, but it also did not eliminate raw-output movement on the hosted GPU path. I therefore did not promote llm.reset() into ordinary Modal production inference. The remaining hosted raw variability is reported separately and is not labeled a prompt-cache effect or evidence that quantization adds entropy.

Bottom line

Your reconciliation criticism was correct, and my earlier same-seed interpretation was too strong.

The completed evidence supports these narrower conclusions:

  1. The October 19 versus October 24 Q4 discrepancy was not caused by an artifact, binary, fixture, decoding, or seed mismatch.
  2. It was caused by content-specific llama-server prompt-cache history changing the pre-sampling probability surface.
  3. The 27 Q4 outcome flips were distributed throughout the stateful block, not concentrated near its start.
  4. Disabling prompt caching eliminated all cold-versus-production-order movement in the controlled local experiment.
  5. Restarting the server also restored the cold trajectory.
  6. Request count alone did not reproduce the effect.
  7. Q4 was more behaviorally sensitive to the cached-history change than F16, but that does not by itself establish that quantization adds sampling entropy.
  8. Current local production runs use the corrected cache-disabled, single-slot contract and fail publication closed if that contract is not verified.

The consolidated evidence is recorded under:

  • Qwen2.5_7B_Instruct_20261102_stateful_context_study
  • Qwen2.5_7B_Instruct_20261105_backend_mechanism_study
  • Qwen2.5_7B_Instruct_20261106_013427
  • Qwen2.5_7B_Instruct_20261106_025009
  • Qwen2.5_14B_Instruct_1M_20261106_052249

If you would like the complete case-level results, telemetry, manifests, fixtures, probability diagnostics, and machine-readable decision matrices, those runs can be requested by name through https://pbhappliedsystems.com/contact.

I will be establishing a public changelog for quant_eval on Hugging Face and pbhappliedsystems.com to document changes and their supporting evidence. As previously mentioned, I would like to attribute the improvements prompted by your comments appropriately. The changelog will proceed regardless; if you are comfortable being credited, please let me know your preferred name or profile link for attribution.

Cache off, zero changes. Restart, zero changes. Same request count with neutral content, zero changes. That is a clean kill.

One row in your six-arm table reads differently to me.
Cold with the cache on is 152 / 155. Cold with it off is 151 / 156.
So the cold arm was not cache-free either. Inside the stateful block, later calls can reuse prefixes from earlier ones.
Production order is one case of a general effect: any reused prefix can move the logits.

My guess at the why: a reused prefix's KV was computed in a different batch split than a fresh pass would use, and the kernels are not bit-identical across splits.
Neutral warmup shares no prefix with the stateful prompts, so nothing is reused, which fits it reproducing cold exactly.

Q4 moves more in both of your setups: 104 raw changes vs 34 locally, 96 vs 16 on Modal with reset.
Your top-20 log-softmax capture can say why.
If Q4's step-1 gap between the top two tokens is smaller than F16's on the same prompts, Q4 is not noisier, it is closer to ties, and any small perturbation flips more of them.

On credit: thank you for asking, let me come back to you on that one.

What does the top-1 margin look like on the 68 Q4 records whose first token changed?

·

Dipankar, your reading of the cache-enabled cold arm is correct. “Cold” meant that stateful evaluation began immediately after preflight in a fresh server process; it did not mean that every request within the stateful block was evaluated without prefix-cache reuse. Later stateful calls could reuse prefixes established by earlier calls in that same block.

That distinction fits the controls:

  • Cache enabled, production order versus immediate-stateful execution: outputs moved.
  • Cache disabled with -np 1: production order versus immediate-stateful execution produced zero changes.
  • Server restart before stateful evaluation: zero changes.
  • Neutral warmup with the same request count but no shared stateful prefixes: zero changes.

The evidence therefore implicates reusable prompt-cache state, not request count alone or a simple advancement of the sampler RNG. Your proposed KV/batch-split explanation is plausible, but the experiment did not instrument the kernels or KV construction directly, so I am keeping that as a mechanism hypothesis rather than a measured conclusion.

I completed the top-1-margin analysis for the 68 Q4 records whose selected first token changed between the cache-enabled immediate-stateful and production-order arms.

The margin is defined as:

top-1 log probability − top-2 log probability

It is computed from the captured top-20 pre-sampling log-softmax surface.

Population Immediate-stateful mean / median Production-order mean / median
Q4 records with a changed selected first token 1.070 / 1.045 0.879 / 0.886
Q4 records without a changed selected first token 2.548 / 2.622 2.522 / 2.647
F16 on the same 68 prompts 0.830 / 0.848 0.816 / 0.831

The changed Q4 records do occupy a lower-margin part of the Q4 population. However, they are not closer to top-two ties than F16 on the same prompts:

  • Q4 had the narrower margin in 17/68 records in the immediate-stateful context.
  • Q4 had the narrower margin in 29/68 records in production order.
  • The top-1/argmax token changed in 0/68 records.
  • The sampled token was the argmax in 50/68 immediate-stateful records, but only 18/68 production-order records.
  • All 68 selected-first-token changes occurred on turn 1; none occurred on turn 2.
  • F16 changed its sampled first token on 11 of those same 68 prompts, also with zero argmax changes.

So the specific explanation that Q4 moved more because quantization placed its top two tokens closer together is not supported by these records. Q4’s changed cases have smaller margins than Q4’s unchanged cases, but their margins are generally wider than the corresponding F16 margins.

The cleaner interpretation is that cache-associated numerical perturbations changed the probability surface enough to alter the sampled token while preserving the highest-probability token. The same seed therefore does not guarantee the same sampled token when reused-prefix computation changes the probabilities presented to the sampler.

This also means the larger Q4 movement—104 raw-output changes locally and 96 in the Modal reset comparison, versus 34 and 16 for F16—cannot be explained by the first-token top-1/top-2 margin alone. It may involve differences elsewhere in the sampled distribution or amplification across later decoding steps. The current evidence does not justify calling Q4 intrinsically noisier or claiming that quantization created more first-token top-two ties.

Two evidence boundaries are important:

  1. These are top-20 pre-sampling log probabilities, not complete full-vocabulary logits.
  2. The 68 records were selected because their sampled first token changed, so this is a descriptive post-hoc analysis rather than an independent confirmatory test.

I have added the margin analysis and those boundaries to the machine-readable backend-mechanism report. The completed evidence package is:

Qwen2.5_7B_Instruct_20261105_backend_mechanism_study

Thank you for identifying the within-block cache distinction. It materially improves the interpretation: the relevant condition is not merely “production history versus cold start,” but whether any reusable prompt prefix changes how the cached KV state is constructed and reused.

Q4 is not closer to ties. I had that wrong, and your 17 of 68 settles it.

The line I keep looking at is 50 vs 18.
The argmax is the same in both arms on all 68, yet the sampler picked it 50 times in one and 18 in the other.
So at least 32 records moved off the argmax in production order, and at most 18 moved onto it.
That has a sign. So did your 19 cold-only passes vs 8 production-only.
Two counts, same direction: production order loses argmax picks and loses passes.

Jitter should not have a sign. A reused prefix that flattens the top of the distribution would.
Your selection caveat has a clean way around it, since you captured both arms on all 200 prompts.
Is the top-1 logprob lower in production order across all 200, for Q4 and for F16?