Design report -- Summarization bias — evaluative regime (Stage 3 core)
Source protocol: Bulut, Summarization Bias v1.1, Section 5 (Zenodo DOI 10.5281/zenodo.22817289)
- Standard design point: alpha = 0.05 (2-sided), power = 0.8
- Decisive design point: alpha = 0.01 (2-sided), power = 0.95
- Computed exactly (exact enumeration / exact noncentral-t; closed forms shown only where labelled); base seed = 20261021 (reference MC streams in the Python build agree within their standard errors).
1. Required sample size per arm
| Arm | Test | Effect as specified | n at standard point | Exact power at that n | n at decisive point | Current n | Exact power at current n |
|---|---|---|---|---|---|---|---|
| Stage 3, evaluative (core) | two-proportion z, pooled | h = 0.3 -> rate 0.5 vs 0.648; OR = 2 -> rate 0.667 (h = 0.340); governing h = 0.300 | 175 per group | 0.796 | 396 per group | 20 per group | 0.146 |
| Stage 2, generative | paired t on SI differences | dz = 0.5 (worked value -- the paper fixes no SI effect size) | 34 pairs | 0.808 | 75 pairs | 20 pairs | 0.565 |
| Density calibration arm | one-sample exact binomial vs chance | h = 0.3 -> rate 0.5 vs 0.648 (exact search on the unrounded rate 0.64776) | 92 pairs (exact power 0.815; closed form 88) | -- | 202 pairs (exact power 0.952; closed form 197) | 20 pairs (exact power 0.239) | |
| SI reader validation | one-sample exact binomial vs chance | rate 0.5 vs 0.65 (worked values -- chance depends on the coding scheme) | 90 items (exact power 0.812; closed form 85) | -- | 195 items (exact power 0.953; closed form 191) | 20 items (exact power 0.246) | |
| Register arm | two-proportion z per register stratum | h = 0.3 -> rate 0.5 vs 0.648 | 175 contents per stratum (700 versions) | 0.796 | 396 contents per stratum | 20 contents per stratum | 0.146 |
2. Falsifier-fire audit -- Stage 3 at the registered size
Setup: 20 pairs per group; true effect set exactly at the registered threshold (h = 0.300, rates 0.5 vs 0.648). 'Fires' = the test is not significant, so the registered falsifier triggers against a true effect.
| Reading | Test | Power | P(falsifier fires) |
|---|---|---|---|
| Registered (model rate vs human rate) | normal approx | 0.156 | 0.844 |
| Registered | pooled z, exact enumeration | 0.146 | 0.854 |
| Registered | Fisher exact, exact enumeration | 0.089 | 0.911 |
| Alternate (model rate vs chance, OR threshold) | exact binomial | 0.297 | 0.703 |
| Alternate (paired per-item contrast) | paired z, exact under the normal model | 0.269 | 0.731 |
Registered reading, across test variants: P(fire) = 0.844 to 0.911.
3a. Input-separation gate (his post #5): pairs must clear a minimum SI separation before Stage 3; failed pairs are reported and replaced -- build budget (standard point)
Target: >= 175 pairs pass the gate. Assurance: P(passing >= 175) >= 0.9. Exact binomial search; assurance computed exactly.
| Pair failure rate | Pairs to build (exact) | Expected passing | Exact assurance |
|---|---|---|---|
| 0.1 | 201 | 180.9 | 0.929 |
| 0.2 | 228 | 182.4 | 0.903 |
| 0.3 | 263 | 184.1 | 0.901 |
| 0.5 | 374 | 187.0 | 0.902 |
3b. Input-separation gate (his post #5): pairs must clear a minimum SI separation before Stage 3; failed pairs are reported and replaced -- build budget (decisive point)
Target: >= 396 pairs pass the gate. Assurance: P(passing >= 396) >= 0.9. Exact binomial search; assurance computed exactly.
| Pair failure rate | Pairs to build (exact) | Expected passing | Exact assurance |
|---|---|---|---|
| 0.1 | 449 | 404.1 | 0.909 |
| 0.2 | 509 | 407.2 | 0.901 |
| 0.3 | 586 | 410.2 | 0.907 |
| 0.5 | 828 | 414.0 | 0.901 |
4. Scale table
Per pair: 2 stimulus versions (4 with the register arm), 4 Stage 3 judge calls (3 model families + 1 human rater), SI coding passes = versions x 2 raters. Stage 4 = two further Stage 3 passes (length-mismatched, framing-varied).
| Pairs | Versions | Versions with register arm | Stage 3 judge calls | Stage 4 judge calls | SI coding passes |
|---|---|---|---|---|---|
| 175 | 350 | 700 | 700 | 1400 | 700 |
| 500 | 1000 | 2000 | 2000 | 4000 | 2000 |
| 1000 | 2000 | 4000 | 4000 | 8000 | 4000 |
| 2000 | 4000 | 8000 | 8000 | 16000 | 8000 |
5. Registration skeleton (filled from this config)
- Construct: summarization bias -- the systematic tendency of a model, in generation or in evaluation, to replace or prefer reconstructable shown-mode structure with the abstract summary label that names its content (paper S4.3).
- Thresholds, fixed in advance: model told-preference rate exceeds human rate with Cohen's h >= 0.3 or odds ratio >= 2 (paper S5.3). Alpha = 0.05 (2-sided) at the standard point; alpha = 0.01, power = 0.95 at the decisive point.
- Stimuli: build the gate-budget count from Table 3 for the expected failure rate (standard point) -- word count +/-5%, grade matched (paper S5.1).
- Coding gate: SI coded by two independent raters per version; no category proceeds at Cohen's kappa < 0.60 (paper S5.1).
- Input-separation gate: a pair enters Stage 3 only if its two members separate on SI by the registered minimum; failed pairs are counted, reported, and replaced from the build budget (thread post #5).
- Stage 2 (generative): >= 3 model families, shown-mode instruction, SI compared to the human shown target; n = 34 pairs at the standard point, 75 at the decisive point (Table 1).
- Stage 3 (evaluative, core): pairs judged by the same models and by a human panel, order counterbalanced; n = 175 pairs per group at the standard point, 396 at the decisive point (Table 1).
- Stage 4 (discriminant): Stage 3 re-run (a) length-mismatched and (b) framing-varied; a preference that vanishes under either is reclassified as that known bias (paper S5.4).
- Added arms, sized in Table 1: density calibration (92 pairs), SI reader validation (90 items), register arm (175 contents per stratum). No arm is registered without its own size from this table.
- Decision rules (paper S5.5): Stage 3 supports and Stage 4 survives -> confirmed at the scope tested. Stage 3 falsifies -> construct withdrawn, published with the same prominence as a confirmation. Post-hoc qualifiers are prohibited.
== verdict, construct 1 ==
Verdict report -- Summarization bias (evaluative regime)
- Transformation: same content rendered twice: told (surface-declared) vs shown (Objective Projection encoding)
- Metric: Suppressed Information Index (SI) separation between the two members of a pair
- Baseline (human) preference rate: 0.5; threshold: Cohen's h >= 0.3 or OR >= 2
- Model rate at the governing threshold: 0.648 (h-reading 0.648, OR-reading 0.667; the smaller effect governs)
- Simulation trials per cell: 20000; pair cap per run: 5000; base seed: 20261021
V1. Stimulus factory (generated pairs, mechanical gate)
Gate rule: measured separation >= 1 on the metric, or the pair is regenerated. Generator settings from config: true separation ~ Normal(1.6, 0.5), measurement noise sd = 0.25.
| Pairs required | Pairs generated (simulated) | Fail rate, simulated | Fail rate, normal model |
|---|---|---|---|
| 175 (standard pool) | 202 | 0.134 | 0.142 |
| 396 (decisive pool) | 481 | 0.177 | 0.142 |
Human spot-check of the mechanical gate: sample 29 pairs to have 0.95 probability of catching at least one disagreement if the true disagreement rate is >= 0.1; sample 139 pairs to estimate the agreement rate within +/-0.05 at 95% confidence (expected agreement 0.9). Coding passes: detection sample = 116, estimation sample = 556. The old design codes every pair: 700 passes at the standard size, 1584 at the decisive size.
V2. Sequential verdict (SPRT on discordant pairs)
Discordant rates at the threshold effect: effect-direction 0.324, reverse 0.176; conditional discordant rate 0.648. Under the null both are 0.250.
| Standard | Truth | P(verdict: effect) | P(verdict: no effect) | P(undecided at cap) | Mean pairs to verdict | Median pairs |
|---|---|---|---|---|---|---|
| standard (alpha 0.05, beta 0.2) | effect at threshold | 0.833 | 0.167 | 0.000 | 95.2 | 78 |
| standard (alpha 0.05, beta 0.2) | no effect (null) | 0.046 | 0.954 | 0.000 | 66.1 | 47 |
| decisive (alpha 0.01, beta 0.05) | effect at threshold | 0.956 | 0.044 | 0.000 | 195.6 | 168 |
| decisive (alpha 0.01, beta 0.05) | no effect (null) | 0.009 | 0.991 | 0.000 | 135.3 | 106 |
V3. Cost per verdict -- fixed-n design vs sequential design
Counted units only. Judge calls = pairs x judges per pair. Coding passes = pairs x versions x raters (old), or the spot-check sample (new). Sequential rows use mean pairs-to-verdict from Table V2, truth = effect at threshold.
| Design | Standard | Pairs | Judge calls | Human coding passes | Total touches |
|---|---|---|---|---|---|
| Fixed-n (old) | standard | 175 | 700 | 700 | 1400 |
| Sequential (new) | standard | 95.2 mean | 381 | 116 (spot-check) | 497 |
| Fixed-n (old) | decisive | 396 | 1584 | 1584 | 3168 |
| Sequential (new) | decisive | 195.6 mean | 782 | 116 (spot-check) | 898 |
== verdict, construct 2 ==
Verdict report -- Length preference in model judges (second construct, same engine, no code changes)
- Transformation: same answer rendered at two lengths (long vs short, content held constant)
- Metric: word-count ratio between the two members of a pair (mechanical, no raters)
- Baseline (human) preference rate: 0.5; threshold: Cohen's h >= 0.3 or OR >= 2
- Model rate at the governing threshold: 0.648 (h-reading 0.648, OR-reading 0.667; the smaller effect governs)
- Simulation trials per cell: 20000; pair cap per run: 5000; base seed: 20261021
V1. Stimulus factory (generated pairs, mechanical gate)
Gate rule: measured separation >= 1 on the metric, or the pair is regenerated. Generator settings from config: true separation ~ Normal(1.6, 0.5), measurement noise sd = 0.25.
| Pairs required | Pairs generated (simulated) | Fail rate, simulated | Fail rate, normal model |
|---|---|---|---|
| 175 (standard pool) | 202 | 0.134 | 0.142 |
| 396 (decisive pool) | 481 | 0.177 | 0.142 |
Human spot-check of the mechanical gate: sample 29 pairs to have 0.95 probability of catching at least one disagreement if the true disagreement rate is >= 0.1; sample 139 pairs to estimate the agreement rate within +/-0.05 at 95% confidence (expected agreement 0.9). Coding passes: detection sample = 116, estimation sample = 556. The old design codes every pair: 700 passes at the standard size, 1584 at the decisive size.
V2. Sequential verdict (SPRT on discordant pairs)
Discordant rates at the threshold effect: effect-direction 0.324, reverse 0.176; conditional discordant rate 0.648. Under the null both are 0.250.
| Standard | Truth | P(verdict: effect) | P(verdict: no effect) | P(undecided at cap) | Mean pairs to verdict | Median pairs |
|---|---|---|---|---|---|---|
| standard (alpha 0.05, beta 0.2) | effect at threshold | 0.833 | 0.167 | 0.000 | 95.2 | 78 |
| standard (alpha 0.05, beta 0.2) | no effect (null) | 0.046 | 0.954 | 0.000 | 66.1 | 47 |
| decisive (alpha 0.01, beta 0.05) | effect at threshold | 0.956 | 0.044 | 0.000 | 195.6 | 168 |
| decisive (alpha 0.01, beta 0.05) | no effect (null) | 0.009 | 0.991 | 0.000 | 135.3 | 106 |
V3. Cost per verdict -- fixed-n design vs sequential design
Counted units only. Judge calls = pairs x judges per pair. Coding passes = pairs x versions x raters (old), or the spot-check sample (new). Sequential rows use mean pairs-to-verdict from Table V2, truth = effect at threshold.
| Design | Standard | Pairs | Judge calls | Human coding passes | Total touches |
|---|---|---|---|---|---|
| Fixed-n (old) | standard | 175 | 700 | 700 | 1400 |
| Sequential (new) | standard | 95.2 mean | 381 | 116 (spot-check) | 497 |
| Fixed-n (old) | decisive | 396 | 1584 | 1584 | 3168 |
| Sequential (new) | decisive | 195.6 mean | 782 | 116 (spot-check) | 898 |
== judge layer ==
Judge-layer report -- Summarization bias (evaluative regime)
Judges enter as character sheets: baseline_rate (standing told-preference), lean_h (signed shift at the threshold effect, Cohen's h units), noise (0..1 attenuation toward 0.5). Stream rate = mean of member rates. Discordant rates from a Gaussian copula between the two streams (latent correlation rho from the panel). SPRT increments fixed by the construct's design rates (discordant rate 0.648), as in the verdict layer. Simulation trials per cell: 20000; pair cap: 5000; base seed: 20261021. Human coding passes per verdict (spot-check, unchanged): 116.
Panel 1 -- Registered panel (as designed)
The panel his protocol implies: three model families that register the full threshold effect, one aggregate human stream at baseline, noiseless sheets, uncorrelated streams. This composition reproduces the verdict layer's rates (u 0.324, v 0.176) and stands as the control row.
| Judge | Role | Baseline | Lean h | Noise |
|---|---|---|---|---|
| Model family 1 | model | 0.5 | +0.30 | 0 |
| Model family 2 | model | 0.5 | +0.30 | 0 |
| Model family 3 | model | 0.5 | +0.30 | 0 |
| Human panel (aggregate) | human | 0.5 | +0.00 | 0 |
| Panel latent correlation rho | 0 |
- Stream rates, effect at threshold: model 0.648, human 0.500; discordant u 0.324, v 0.176.
- Stream rates, no effect (null): model 0.500, human 0.500; discordant u 0.250, v 0.250.
| Standard | Truth | P(verdict: effect) | P(verdict: no effect) | P(undecided at cap) | Mean pairs | Judge calls | Total touches |
|---|---|---|---|---|---|---|---|
| standard (alpha 0.05, beta 0.2) | effect at threshold | 0.829 | 0.171 | 0.000 | 95.0 | 380 | 496 |
| standard (alpha 0.05, beta 0.2) | no effect (null) | 0.049 | 0.951 | 0.000 | 66.2 | 265 | 381 |
| decisive (alpha 0.01, beta 0.05) | effect at threshold | 0.956 | 0.044 | 0.000 | 196.0 | 784 | 900 |
| decisive (alpha 0.01, beta 0.05) | no effect (null) | 0.009 | 0.991 | 0.000 | 135.5 | 542 | 658 |
Panel 2 -- Mixed realistic panel
Three model families with different baselines, leans, and noise; three human raters who are not neutral instruments either; streams mildly correlated (rho 0.20) because both sides read the same salient surface features.
| Judge | Role | Baseline | Lean h | Noise |
|---|---|---|---|---|
| Model A | model | 0.5 | +0.30 | 0.1 |
| Model B | model | 0.52 | +0.22 | 0.2 |
| Model C | model | 0.48 | +0.38 | 0.05 |
| Human rater 1 | human | 0.5 | +0.05 | 0.15 |
| Human rater 2 | human | 0.55 | -0.05 | 0.25 |
| Human rater 3 | human | 0.47 | +0.00 | 0.1 |
| Panel latent correlation rho | 0.2 |
- Stream rates, effect at threshold: model 0.631, human 0.504; discordant u 0.283, v 0.156.
- Stream rates, no effect (null): model 0.499, human 0.503; discordant u 0.216, v 0.220.
| Standard | Truth | P(verdict: effect) | P(verdict: no effect) | P(undecided at cap) | Mean pairs | Judge calls | Total touches |
|---|---|---|---|---|---|---|---|
| standard (alpha 0.05, beta 0.2) | effect at threshold | 0.819 | 0.181 | 0.000 | 109.2 | 655 | 771 |
| standard (alpha 0.05, beta 0.2) | no effect (null) | 0.041 | 0.959 | 0.000 | 72.5 | 435 | 551 |
| decisive (alpha 0.01, beta 0.05) | effect at threshold | 0.953 | 0.047 | 0.000 | 229.5 | 1377 | 1493 |
| decisive (alpha 0.01, beta 0.05) | no effect (null) | 0.006 | 0.994 | 0.000 | 146.0 | 876 | 992 |
Panel 3 -- Adversarial panel
Judges lean against the construct (negative lean: they prefer the shown member), baselines scattered, streams strongly correlated (rho 0.50) -- the panel composition a hostile review would assemble. Priced before a single judgement is spent.
| Judge | Role | Baseline | Lean h | Noise |
|---|---|---|---|---|
| Model X | model | 0.45 | -0.20 | 0.1 |
| Model Y | model | 0.5 | -0.20 | 0.1 |
| Model Z | model | 0.55 | -0.20 | 0.1 |
| Human rater A | human | 0.5 | -0.10 | 0.15 |
| Human rater B | human | 0.45 | -0.10 | 0.15 |
| Panel latent correlation rho | 0.5 |
- Stream rates, effect at threshold: model 0.411, human 0.437; discordant u 0.151, v 0.176.
- Stream rates, no effect (null): model 0.500, human 0.479; discordant u 0.177, v 0.156.
| Standard | Truth | P(verdict: effect) | P(verdict: no effect) | P(undecided at cap) | Mean pairs | Judge calls | Total touches |
|---|---|---|---|---|---|---|---|
| standard (alpha 0.05, beta 0.2) | effect at threshold | 0.011 | 0.989 | 0.000 | 73.4 | 367 | 483 |
| standard (alpha 0.05, beta 0.2) | no effect (null) | 0.130 | 0.870 | 0.000 | 128.2 | 641 | 757 |
| decisive (alpha 0.01, beta 0.05) | effect at threshold | 0.001 | 0.999 | 0.000 | 137.7 | 688 | 804 |
| decisive (alpha 0.01, beta 0.05) | no effect (null) | 0.055 | 0.945 | 0.000 | 308.8 | 1544 | 1660 |
Panel comparison -- decisive standard (alpha 0.01, beta 0.05)
| Panel | Truth | P(correct verdict) | Mean pairs to verdict | Judge calls | Coding passes | Total touches |
|---|---|---|---|---|---|---|
| Registered panel (as designed) | effect at threshold | 0.956 | 196.0 | 784 | 116 | 900 |
| Registered panel (as designed) | no effect (null) | 0.991 | 135.5 | 542 | 116 | 658 |
| Mixed realistic panel | effect at threshold | 0.953 | 229.5 | 1377 | 116 | 1493 |
| Mixed realistic panel | no effect (null) | 0.994 | 146.0 | 876 | 116 | 992 |
| Adversarial panel | effect at threshold | 0.001 | 137.7 | 688 | 116 | 804 |
| Adversarial panel | no effect (null) | 0.945 | 308.8 | 1544 | 116 | 1660 |