| # Design report -- Summarization bias — evaluative regime (Stage 3 core) |
|
|
| Source protocol: Bulut, Summarization Bias v1.1, Section 5 (Zenodo DOI 10.5281/zenodo.22817289) |
|
|
| - Standard design point: alpha = 0.05 (2-sided), power = 0.8 |
| - Decisive design point: alpha = 0.01 (2-sided), power = 0.95 |
| - Computed exactly (exact enumeration / exact noncentral-t; closed forms shown only where labelled); base seed = 20261021 (reference MC streams in the Python build agree within their standard errors). |
|
|
| ## 1. Required sample size per arm |
|
|
| | Arm | Test | Effect as specified | n at standard point | Exact power at that n | n at decisive point | Current n | Exact power at current n | |
| |---|---|---|---|---|---|---|---| |
| | Stage 3, evaluative (core) | two-proportion z, pooled | h = 0.3 -> rate 0.5 vs 0.648; OR = 2 -> rate 0.667 (h = 0.340); governing h = 0.300 | 175 per group | 0.796 | 396 per group | 20 per group | 0.146 | |
| | Stage 2, generative | paired t on SI differences | dz = 0.5 (worked value -- the paper fixes no SI effect size) | 34 pairs | 0.808 | 75 pairs | 20 pairs | 0.565 | |
| | Density calibration arm | one-sample exact binomial vs chance | h = 0.3 -> rate 0.5 vs 0.648 (exact search on the unrounded rate 0.64776) | 92 pairs (exact power 0.815; closed form 88) | -- | 202 pairs (exact power 0.952; closed form 197) | 20 pairs (exact power 0.239) | |
| | SI reader validation | one-sample exact binomial vs chance | rate 0.5 vs 0.65 (worked values -- chance depends on the coding scheme) | 90 items (exact power 0.812; closed form 85) | -- | 195 items (exact power 0.953; closed form 191) | 20 items (exact power 0.246) | |
| | Register arm | two-proportion z per register stratum | h = 0.3 -> rate 0.5 vs 0.648 | 175 contents per stratum (700 versions) | 0.796 | 396 contents per stratum | 20 contents per stratum | 0.146 | |
|
|
| ## 2. Falsifier-fire audit -- Stage 3 at the registered size |
|
|
| Setup: 20 pairs per group; true effect set exactly at the registered threshold (h = 0.300, rates 0.5 vs 0.648). 'Fires' = the test is not significant, so the registered falsifier triggers against a true effect. |
|
|
| | Reading | Test | Power | P(falsifier fires) | |
| |---|---|---|---| |
| | Registered (model rate vs human rate) | normal approx | 0.156 | 0.844 | |
| | Registered | pooled z, exact enumeration | 0.146 | 0.854 | |
| | Registered | Fisher exact, exact enumeration | 0.089 | 0.911 | |
| | Alternate (model rate vs chance, OR threshold) | exact binomial | 0.297 | 0.703 | |
| | Alternate (paired per-item contrast) | paired z, exact under the normal model | 0.269 | 0.731 | |
|
|
| Registered reading, across test variants: P(fire) = 0.844 to 0.911. |
|
|
| ## 3a. Input-separation gate (his post #5): pairs must clear a minimum SI separation before Stage 3; failed pairs are reported and replaced -- build budget (standard point) |
|
|
| Target: >= 175 pairs pass the gate. Assurance: P(passing >= 175) >= 0.9. Exact binomial search; assurance computed exactly. |
|
|
| | Pair failure rate | Pairs to build (exact) | Expected passing | Exact assurance | |
| |---|---|---|---| |
| | 0.1 | 201 | 180.9 | 0.929 | |
| | 0.2 | 228 | 182.4 | 0.903 | |
| | 0.3 | 263 | 184.1 | 0.901 | |
| | 0.5 | 374 | 187.0 | 0.902 | |
|
|
| ## 3b. Input-separation gate (his post #5): pairs must clear a minimum SI separation before Stage 3; failed pairs are reported and replaced -- build budget (decisive point) |
|
|
| Target: >= 396 pairs pass the gate. Assurance: P(passing >= 396) >= 0.9. Exact binomial search; assurance computed exactly. |
|
|
| | Pair failure rate | Pairs to build (exact) | Expected passing | Exact assurance | |
| |---|---|---|---| |
| | 0.1 | 449 | 404.1 | 0.909 | |
| | 0.2 | 509 | 407.2 | 0.901 | |
| | 0.3 | 586 | 410.2 | 0.907 | |
| | 0.5 | 828 | 414.0 | 0.901 | |
|
|
| ## 4. Scale table |
|
|
| Per pair: 2 stimulus versions (4 with the register arm), 4 Stage 3 judge calls (3 model families + 1 human rater), SI coding passes = versions x 2 raters. Stage 4 = two further Stage 3 passes (length-mismatched, framing-varied). |
|
|
| | Pairs | Versions | Versions with register arm | Stage 3 judge calls | Stage 4 judge calls | SI coding passes | |
| |---|---|---|---|---|---| |
| | 175 | 350 | 700 | 700 | 1400 | 700 | |
| | 500 | 1000 | 2000 | 2000 | 4000 | 2000 | |
| | 1000 | 2000 | 4000 | 4000 | 8000 | 4000 | |
| | 2000 | 4000 | 8000 | 8000 | 16000 | 8000 | |
|
|
| ## 5. Registration skeleton (filled from this config) |
|
|
| 1. Construct: summarization bias -- the systematic tendency of a model, in generation or in evaluation, to replace or prefer reconstructable shown-mode structure with the abstract summary label that names its content (paper S4.3). |
| 2. Thresholds, fixed in advance: model told-preference rate exceeds human rate with Cohen's h >= 0.3 or odds ratio >= 2 (paper S5.3). Alpha = 0.05 (2-sided) at the standard point; alpha = 0.01, power = 0.95 at the decisive point. |
| 3. Stimuli: build the gate-budget count from Table 3 for the expected failure rate (standard point) -- word count +/-5%, grade matched (paper S5.1). |
| 4. Coding gate: SI coded by two independent raters per version; no category proceeds at Cohen's kappa < 0.60 (paper S5.1). |
| 5. Input-separation gate: a pair enters Stage 3 only if its two members separate on SI by the registered minimum; failed pairs are counted, reported, and replaced from the build budget (thread post #5). |
| 6. Stage 2 (generative): >= 3 model families, shown-mode instruction, SI compared to the human shown target; n = 34 pairs at the standard point, 75 at the decisive point (Table 1). |
| 7. Stage 3 (evaluative, core): pairs judged by the same models and by a human panel, order counterbalanced; n = 175 pairs per group at the standard point, 396 at the decisive point (Table 1). |
| 8. Stage 4 (discriminant): Stage 3 re-run (a) length-mismatched and (b) framing-varied; a preference that vanishes under either is reclassified as that known bias (paper S5.4). |
| 9. Added arms, sized in Table 1: density calibration (92 pairs), SI reader validation (90 items), register arm (175 contents per stratum). No arm is registered without its own size from this table. |
| 10. Decision rules (paper S5.5): Stage 3 supports and Stage 4 survives -> confirmed at the scope tested. Stage 3 falsifies -> construct withdrawn, published with the same prominence as a confirmation. Post-hoc qualifiers are prohibited. |
|
|
| == verdict, construct 1 == |
| # Verdict report -- Summarization bias (evaluative regime) |
|
|
| - Transformation: same content rendered twice: told (surface-declared) vs shown (Objective Projection encoding) |
| - Metric: Suppressed Information Index (SI) separation between the two members of a pair |
| - Baseline (human) preference rate: 0.5; threshold: Cohen's h >= 0.3 or OR >= 2 |
| - Model rate at the governing threshold: 0.648 (h-reading 0.648, OR-reading 0.667; the smaller effect governs) |
| - Simulation trials per cell: 20000; pair cap per run: 5000; base seed: 20261021 |
|
|
| ## V1. Stimulus factory (generated pairs, mechanical gate) |
|
|
| Gate rule: measured separation >= 1 on the metric, or the pair is regenerated. Generator settings from config: true separation ~ Normal(1.6, 0.5), measurement noise sd = 0.25. |
|
|
| | Pairs required | Pairs generated (simulated) | Fail rate, simulated | Fail rate, normal model | |
| |---|---|---|---| |
| | 175 (standard pool) | 202 | 0.134 | 0.142 | |
| | 396 (decisive pool) | 481 | 0.177 | 0.142 | |
|
|
| Human spot-check of the mechanical gate: sample 29 pairs to have 0.95 probability of catching at least one disagreement if the true disagreement rate is >= 0.1; sample 139 pairs to estimate the agreement rate within +/-0.05 at 95% confidence (expected agreement 0.9). Coding passes: detection sample = 116, estimation sample = 556. The old design codes every pair: 700 passes at the standard size, 1584 at the decisive size. |
|
|
| ## V2. Sequential verdict (SPRT on discordant pairs) |
|
|
| Discordant rates at the threshold effect: effect-direction 0.324, reverse 0.176; conditional discordant rate 0.648. Under the null both are 0.250. |
|
|
| | Standard | Truth | P(verdict: effect) | P(verdict: no effect) | P(undecided at cap) | Mean pairs to verdict | Median pairs | |
| |---|---|---|---|---|---|---| |
| | standard (alpha 0.05, beta 0.2) | effect at threshold | 0.833 | 0.167 | 0.000 | 95.2 | 78 | |
| | standard (alpha 0.05, beta 0.2) | no effect (null) | 0.046 | 0.954 | 0.000 | 66.1 | 47 | |
| | decisive (alpha 0.01, beta 0.05) | effect at threshold | 0.956 | 0.044 | 0.000 | 195.6 | 168 | |
| | decisive (alpha 0.01, beta 0.05) | no effect (null) | 0.009 | 0.991 | 0.000 | 135.3 | 106 | |
|
|
| ## V3. Cost per verdict -- fixed-n design vs sequential design |
|
|
| Counted units only. Judge calls = pairs x judges per pair. Coding passes = pairs x versions x raters (old), or the spot-check sample (new). Sequential rows use mean pairs-to-verdict from Table V2, truth = effect at threshold. |
|
|
| | Design | Standard | Pairs | Judge calls | Human coding passes | Total touches | |
| |---|---|---|---|---|---| |
| | Fixed-n (old) | standard | 175 | 700 | 700 | 1400 | |
| | Sequential (new) | standard | 95.2 mean | 381 | 116 (spot-check) | 497 | |
| | Fixed-n (old) | decisive | 396 | 1584 | 1584 | 3168 | |
| | Sequential (new) | decisive | 195.6 mean | 782 | 116 (spot-check) | 898 | |
|
|
| == verdict, construct 2 == |
| # Verdict report -- Length preference in model judges (second construct, same engine, no code changes) |
|
|
| - Transformation: same answer rendered at two lengths (long vs short, content held constant) |
| - Metric: word-count ratio between the two members of a pair (mechanical, no raters) |
| - Baseline (human) preference rate: 0.5; threshold: Cohen's h >= 0.3 or OR >= 2 |
| - Model rate at the governing threshold: 0.648 (h-reading 0.648, OR-reading 0.667; the smaller effect governs) |
| - Simulation trials per cell: 20000; pair cap per run: 5000; base seed: 20261021 |
|
|
| ## V1. Stimulus factory (generated pairs, mechanical gate) |
|
|
| Gate rule: measured separation >= 1 on the metric, or the pair is regenerated. Generator settings from config: true separation ~ Normal(1.6, 0.5), measurement noise sd = 0.25. |
|
|
| | Pairs required | Pairs generated (simulated) | Fail rate, simulated | Fail rate, normal model | |
| |---|---|---|---| |
| | 175 (standard pool) | 202 | 0.134 | 0.142 | |
| | 396 (decisive pool) | 481 | 0.177 | 0.142 | |
|
|
| Human spot-check of the mechanical gate: sample 29 pairs to have 0.95 probability of catching at least one disagreement if the true disagreement rate is >= 0.1; sample 139 pairs to estimate the agreement rate within +/-0.05 at 95% confidence (expected agreement 0.9). Coding passes: detection sample = 116, estimation sample = 556. The old design codes every pair: 700 passes at the standard size, 1584 at the decisive size. |
|
|
| ## V2. Sequential verdict (SPRT on discordant pairs) |
|
|
| Discordant rates at the threshold effect: effect-direction 0.324, reverse 0.176; conditional discordant rate 0.648. Under the null both are 0.250. |
|
|
| | Standard | Truth | P(verdict: effect) | P(verdict: no effect) | P(undecided at cap) | Mean pairs to verdict | Median pairs | |
| |---|---|---|---|---|---|---| |
| | standard (alpha 0.05, beta 0.2) | effect at threshold | 0.833 | 0.167 | 0.000 | 95.2 | 78 | |
| | standard (alpha 0.05, beta 0.2) | no effect (null) | 0.046 | 0.954 | 0.000 | 66.1 | 47 | |
| | decisive (alpha 0.01, beta 0.05) | effect at threshold | 0.956 | 0.044 | 0.000 | 195.6 | 168 | |
| | decisive (alpha 0.01, beta 0.05) | no effect (null) | 0.009 | 0.991 | 0.000 | 135.3 | 106 | |
|
|
| ## V3. Cost per verdict -- fixed-n design vs sequential design |
|
|
| Counted units only. Judge calls = pairs x judges per pair. Coding passes = pairs x versions x raters (old), or the spot-check sample (new). Sequential rows use mean pairs-to-verdict from Table V2, truth = effect at threshold. |
|
|
| | Design | Standard | Pairs | Judge calls | Human coding passes | Total touches | |
| |---|---|---|---|---|---| |
| | Fixed-n (old) | standard | 175 | 700 | 700 | 1400 | |
| | Sequential (new) | standard | 95.2 mean | 381 | 116 (spot-check) | 497 | |
| | Fixed-n (old) | decisive | 396 | 1584 | 1584 | 3168 | |
| | Sequential (new) | decisive | 195.6 mean | 782 | 116 (spot-check) | 898 | |
|
|
| == judge layer == |
| # Judge-layer report -- Summarization bias (evaluative regime) |
|
|
| Judges enter as character sheets: baseline_rate (standing told-preference), lean_h (signed shift at the threshold effect, Cohen's h units), noise (0..1 attenuation toward 0.5). Stream rate = mean of member rates. Discordant rates from a Gaussian copula between the two streams (latent correlation rho from the panel). SPRT increments fixed by the construct's design rates (discordant rate 0.648), as in the verdict layer. |
| Simulation trials per cell: 20000; pair cap: 5000; base seed: 20261021. Human coding passes per verdict (spot-check, unchanged): 116. |
|
|
| ## Panel 1 -- Registered panel (as designed) |
|
|
| The panel his protocol implies: three model families that register the full threshold effect, one aggregate human stream at baseline, noiseless sheets, uncorrelated streams. This composition reproduces the verdict layer's rates (u 0.324, v 0.176) and stands as the control row. |
|
|
| | Judge | Role | Baseline | Lean h | Noise | |
| |---|---|---|---|---| |
| | Model family 1 | model | 0.5 | +0.30 | 0 | |
| | Model family 2 | model | 0.5 | +0.30 | 0 | |
| | Model family 3 | model | 0.5 | +0.30 | 0 | |
| | Human panel (aggregate) | human | 0.5 | +0.00 | 0 | |
| | Panel latent correlation rho | | | | 0 | |
|
|
| - Stream rates, effect at threshold: model 0.648, human 0.500; discordant u 0.324, v 0.176. |
| - Stream rates, no effect (null): model 0.500, human 0.500; discordant u 0.250, v 0.250. |
|
|
| | Standard | Truth | P(verdict: effect) | P(verdict: no effect) | P(undecided at cap) | Mean pairs | Judge calls | Total touches | |
| |---|---|---|---|---|---|---|---| |
| | standard (alpha 0.05, beta 0.2) | effect at threshold | 0.829 | 0.171 | 0.000 | 95.0 | 380 | 496 | |
| | standard (alpha 0.05, beta 0.2) | no effect (null) | 0.049 | 0.951 | 0.000 | 66.2 | 265 | 381 | |
| | decisive (alpha 0.01, beta 0.05) | effect at threshold | 0.956 | 0.044 | 0.000 | 196.0 | 784 | 900 | |
| | decisive (alpha 0.01, beta 0.05) | no effect (null) | 0.009 | 0.991 | 0.000 | 135.5 | 542 | 658 | |
|
|
| ## Panel 2 -- Mixed realistic panel |
|
|
| Three model families with different baselines, leans, and noise; three human raters who are not neutral instruments either; streams mildly correlated (rho 0.20) because both sides read the same salient surface features. |
|
|
| | Judge | Role | Baseline | Lean h | Noise | |
| |---|---|---|---|---| |
| | Model A | model | 0.5 | +0.30 | 0.1 | |
| | Model B | model | 0.52 | +0.22 | 0.2 | |
| | Model C | model | 0.48 | +0.38 | 0.05 | |
| | Human rater 1 | human | 0.5 | +0.05 | 0.15 | |
| | Human rater 2 | human | 0.55 | -0.05 | 0.25 | |
| | Human rater 3 | human | 0.47 | +0.00 | 0.1 | |
| | Panel latent correlation rho | | | | 0.2 | |
|
|
| - Stream rates, effect at threshold: model 0.631, human 0.504; discordant u 0.283, v 0.156. |
| - Stream rates, no effect (null): model 0.499, human 0.503; discordant u 0.216, v 0.220. |
|
|
| | Standard | Truth | P(verdict: effect) | P(verdict: no effect) | P(undecided at cap) | Mean pairs | Judge calls | Total touches | |
| |---|---|---|---|---|---|---|---| |
| | standard (alpha 0.05, beta 0.2) | effect at threshold | 0.819 | 0.181 | 0.000 | 109.2 | 655 | 771 | |
| | standard (alpha 0.05, beta 0.2) | no effect (null) | 0.041 | 0.959 | 0.000 | 72.5 | 435 | 551 | |
| | decisive (alpha 0.01, beta 0.05) | effect at threshold | 0.953 | 0.047 | 0.000 | 229.5 | 1377 | 1493 | |
| | decisive (alpha 0.01, beta 0.05) | no effect (null) | 0.006 | 0.994 | 0.000 | 146.0 | 876 | 992 | |
|
|
| ## Panel 3 -- Adversarial panel |
|
|
| Judges lean against the construct (negative lean: they prefer the shown member), baselines scattered, streams strongly correlated (rho 0.50) -- the panel composition a hostile review would assemble. Priced before a single judgement is spent. |
|
|
| | Judge | Role | Baseline | Lean h | Noise | |
| |---|---|---|---|---| |
| | Model X | model | 0.45 | -0.20 | 0.1 | |
| | Model Y | model | 0.5 | -0.20 | 0.1 | |
| | Model Z | model | 0.55 | -0.20 | 0.1 | |
| | Human rater A | human | 0.5 | -0.10 | 0.15 | |
| | Human rater B | human | 0.45 | -0.10 | 0.15 | |
| | Panel latent correlation rho | | | | 0.5 | |
|
|
| - Stream rates, effect at threshold: model 0.411, human 0.437; discordant u 0.151, v 0.176. |
| - Stream rates, no effect (null): model 0.500, human 0.479; discordant u 0.177, v 0.156. |
|
|
| | Standard | Truth | P(verdict: effect) | P(verdict: no effect) | P(undecided at cap) | Mean pairs | Judge calls | Total touches | |
| |---|---|---|---|---|---|---|---| |
| | standard (alpha 0.05, beta 0.2) | effect at threshold | 0.011 | 0.989 | 0.000 | 73.4 | 367 | 483 | |
| | standard (alpha 0.05, beta 0.2) | no effect (null) | 0.130 | 0.870 | 0.000 | 128.2 | 641 | 757 | |
| | decisive (alpha 0.01, beta 0.05) | effect at threshold | 0.001 | 0.999 | 0.000 | 137.7 | 688 | 804 | |
| | decisive (alpha 0.01, beta 0.05) | no effect (null) | 0.055 | 0.945 | 0.000 | 308.8 | 1544 | 1660 | |
|
|
| ## Panel comparison -- decisive standard (alpha 0.01, beta 0.05) |
|
|
| | Panel | Truth | P(correct verdict) | Mean pairs to verdict | Judge calls | Coding passes | Total touches | |
| |---|---|---|---|---|---|---| |
| | Registered panel (as designed) | effect at threshold | 0.956 | 196.0 | 784 | 116 | 900 | |
| | Registered panel (as designed) | no effect (null) | 0.991 | 135.5 | 542 | 116 | 658 | |
| | Mixed realistic panel | effect at threshold | 0.953 | 229.5 | 1377 | 116 | 1493 | |
| | Mixed realistic panel | no effect (null) | 0.994 | 146.0 | 876 | 116 | 992 | |
| | Adversarial panel | effect at threshold | 0.001 | 137.7 | 688 | 116 | 804 | |
| | Adversarial panel | no effect (null) | 0.945 | 308.8 | 1544 | 116 | 1660 | |
|
|
|
|
