generation results

.gitattributes

@@ -45,3 +45,5 @@ report/pdnc-qwen3b/figures/fig1b-ladder.png filter=lfs diff=lfs merge=lfs -text
 report/pdnc-qwen3b/figures/fig3-paired-deltas.png filter=lfs diff=lfs merge=lfs -text
 results/litbank-qwen7b/results.json filter=lfs diff=lfs merge=lfs -text
 generation/gen-qwen3b-s30-c20.json filter=lfs diff=lfs merge=lfs -text
+report/gen-qwen3b-s30-c20/figures/gen1-canon-violation.png filter=lfs diff=lfs merge=lfs -text
+report/gen-qwen3b-s30-c20/figures/gen2-final.png filter=lfs diff=lfs merge=lfs -text

report/gen-qwen3b-s30-c20/generation.md

@@ -0,0 +1,99 @@
+# Generation: consistency at length
+
+30 stories × 9 memory conditions × 20 chapters, Qwen/Qwen2.5-3B-Instruct. Canon is stated in chapter 1 only; from chapter 2 each condition keeps whatever its memory carries. Violations are deterministic string tests against facts we planted, so no model or human judgement enters the measurement.
+
+## Means over stories
+
+| condition | n | on_premise | violation_rate | retention | restatement_rate | canon_capture | mean_first_violation | total_chars | prompt_tokens_last |
+|---|---|---|---|---|---|---|---|---|---|
+| base:none | 30 | 1.000 | 0.521 | 0.479 | 0.838 | 0.000 | 5.517 | 68,261 | 112 |
+| base:last-chapter | 30 | 1.000 | 0.383 | 0.617 | 0.854 | 0.000 | 2.623 | 77,028 | 1,017 |
+| base:rolling-summary | 30 | 1.000 | 0.512 | 0.487 | 0.858 | 0.000 | 3.858 | 71,582 | 519 |
+| base:full-context | 30 | 1.000 | 0.367 | 0.633 | 0.846 | 0.000 | 1.753 | 76,118 | 5,308 |
+| base:append-only-state | 30 | 1.000 | 0.613 | 0.388 | 0.887 | 0.550 | 4.151 | 70,313 | 329 |
+| base:dsg-state | 30 | 1.000 | 0.596 | 0.404 | 0.871 | 0.321 | 3.833 | 69,182 | 325 |
+| tuned:full-context | 30 | 0.765 | 0.054 | 0.946 | 0.092 | 0.000 | 13.833 | 64,431 | 5,582 |
+| tuned:dsg-state | 30 | 0.812 | 0.167 | 0.833 | 0.163 | 0.037 | 9.616 | 59,119 | 377 |
+| tuned:dsg-repair | 30 | 0.812 | 0.179 | 0.821 | 0.108 | 0.046 | 10.122 | 58,804 | 389 |
+
+## Paired comparisons (bootstrap over stories)
+
+| comparison | metric | mean Δ | 95% CI | excludes 0 | wins | n |
+|---|---|---|---|---|---|---|
+| base:rolling-summary vs base:none | violation_rate | -0.0083 | [-0.0750, +0.0583] | no | 11–11 | 30 |
+| base:rolling-summary vs base:none | restatement_rate | +0.0208 | [-0.0042, +0.0458] | no | 6–2 | 30 |
+| base:rolling-summary vs base:none | mean_first_violation | -1.6590 | [-2.9198, -0.4867] | **yes** | 9–19 | 30 |
+| base:rolling-summary vs base:none | total_chars | +3320.6000 | [+1781.1542, +4741.1792] | **yes** | 21–9 | 30 |
+| base:rolling-summary vs base:none | prompt_tokens_last | +406.7667 | [+385.4317, +424.8675] | **yes** | 30–0 | 30 |
+| base:full-context vs base:rolling-summary | violation_rate | -0.1458 | [-0.1958, -0.0958] | **yes** | 2–21 | 30 |
+| base:full-context vs base:rolling-summary | restatement_rate | -0.0125 | [-0.0375, +0.0083] | no | 2–4 | 30 |
+| base:full-context vs base:rolling-summary | mean_first_violation | -2.1044 | [-2.9495, -1.2611] | **yes** | 5–23 | 30 |
+| base:full-context vs base:rolling-summary | total_chars | +4535.7667 | [+3384.6583, +5692.6875] | **yes** | 26–4 | 30 |
+| base:full-context vs base:rolling-summary | prompt_tokens_last | +4788.4333 | [+4668.4992, +4894.6667] | **yes** | 30–0 | 30 |
+| base:dsg-state vs base:none | violation_rate | +0.0750 | [+0.0083, +0.1417] | **yes** | 15–8 | 30 |
+| base:dsg-state vs base:none | restatement_rate | +0.0333 | [+0.0083, +0.0583] | **yes** | 8–1 | 30 |
+| base:dsg-state vs base:none | mean_first_violation | -1.6834 | [-3.2605, -0.3609] | **yes** | 11–19 | 30 |
+| base:dsg-state vs base:none | total_chars | +920.7000 | [-139.2075, +2000.7142] | no | 19–11 | 30 |
+| base:dsg-state vs base:none | prompt_tokens_last | +212.2667 | [+200.2325, +226.3000] | **yes** | 30–0 | 30 |
+| base:dsg-state vs base:rolling-summary | violation_rate | +0.0833 | [+0.0167, +0.1500] | **yes** | 15–8 | 30 |
+| base:dsg-state vs base:rolling-summary | restatement_rate | +0.0125 | [-0.0125, +0.0375] | no | 7–4 | 30 |
+| base:dsg-state vs base:rolling-summary | mean_first_violation | -0.0244 | [-1.0238, +0.9517] | no | 16–14 | 30 |
+| base:dsg-state vs base:rolling-summary | total_chars | -2399.9000 | [-3634.5408, -1144.3883] | **yes** | 8–22 | 30 |
+| base:dsg-state vs base:rolling-summary | prompt_tokens_last | -194.5000 | [-212.8008, -173.5333] | **yes** | 0–30 | 30 |
+| base:dsg-state vs base:full-context | violation_rate | +0.2292 | [+0.1708, +0.2875] | **yes** | 25–0 | 30 |
+| base:dsg-state vs base:full-context | restatement_rate | +0.0250 | [-0.0042, +0.0542] | no | 8–3 | 30 |
+| base:dsg-state vs base:full-context | mean_first_violation | +2.0801 | [+1.3574, +2.7750] | **yes** | 26–3 | 30 |
+| base:dsg-state vs base:full-context | total_chars | -6935.6667 | [-8246.7008, -5637.2583] | **yes** | 1–29 | 30 |
+| base:dsg-state vs base:full-context | prompt_tokens_last | -4982.9333 | [-5092.4675, -4858.4958] | **yes** | 0–30 | 30 |
+| base:dsg-state vs base:append-only-state | violation_rate | -0.0167 | [-0.0542, +0.0208] | no | 5–8 | 30 |
+| base:dsg-state vs base:append-only-state | restatement_rate | -0.0167 | [-0.0333, -0.0042] | **yes** | 0–4 | 30 |
+| base:dsg-state vs base:append-only-state | mean_first_violation | -0.3173 | [-1.0473, +0.4100] | no | 11–12 | 30 |
+| base:dsg-state vs base:append-only-state | total_chars | -1131.0000 | [-2300.3942, +10.8017] | no | 12–18 | 30 |
+| base:dsg-state vs base:append-only-state | prompt_tokens_last | -4.5667 | [-18.7667, +11.0000] | no | 14–16 | 30 |
+| tuned:dsg-state vs base:dsg-state | violation_rate | -0.4292 | [-0.5126, -0.3458] | **yes** | 1–28 | 30 |
+| tuned:dsg-state vs base:dsg-state | restatement_rate | -0.7083 | [-0.7625, -0.6500] | **yes** | 0–30 | 30 |
+| tuned:dsg-state vs base:dsg-state | mean_first_violation | +5.7821 | [+2.8633, +8.8578] | **yes** | 19–10 | 30 |
+| tuned:dsg-state vs base:dsg-state | total_chars | -10063.0333 | [-11314.6358, -8847.3283] | **yes** | 0–30 | 30 |
+| tuned:dsg-state vs base:dsg-state | prompt_tokens_last | +52.2667 | [+20.8983, +83.4333] | **yes** | 21–9 | 30 |
+| tuned:full-context vs base:full-context | violation_rate | -0.3125 | [-0.3792, -0.2458] | **yes** | 1–28 | 30 |
+| tuned:full-context vs base:full-context | restatement_rate | -0.7542 | [-0.8083, -0.6958] | **yes** | 0–30 | 30 |
+| tuned:full-context vs base:full-context | mean_first_violation | +12.0800 | [+8.6250, +15.2822] | **yes** | 21–5 | 30 |
+| tuned:full-context vs base:full-context | total_chars | -11687.1333 | [-15433.9925, -8063.1475] | **yes** | 3–27 | 30 |
+| tuned:full-context vs base:full-context | prompt_tokens_last | +274.4333 | [+102.7308, +462.2383] | **yes** | 17–12 | 30 |
+| tuned:dsg-state vs tuned:full-context | violation_rate | +0.1125 | [+0.0583, +0.1708] | **yes** | 15–1 | 30 |
+| tuned:dsg-state vs tuned:full-context | restatement_rate | +0.0708 | [+0.0208, +0.1250] | **yes** | 15–5 | 30 |
+| tuned:dsg-state vs tuned:full-context | mean_first_violation | -4.2178 | [-7.6790, -0.7610] | **yes** | 8–12 | 30 |
+| tuned:dsg-state vs tuned:full-context | total_chars | -5311.5667 | [-8483.2700, -2172.0858] | **yes** | 7–23 | 30 |
+| tuned:dsg-state vs tuned:full-context | prompt_tokens_last | -5205.1000 | [-5340.3675, -5077.6975] | **yes** | 0–30 | 30 |
+| tuned:dsg-repair vs tuned:dsg-state | violation_rate | +0.0125 | [-0.0500, +0.0750] | no | 13–8 | 30 |
+| tuned:dsg-repair vs tuned:dsg-state | restatement_rate | -0.0542 | [-0.1000, -0.0125] | **yes** | 5–13 | 30 |
+| tuned:dsg-repair vs tuned:dsg-state | mean_first_violation | +0.5067 | [-3.8494, +4.5501] | no | 15–10 | 30 |
+| tuned:dsg-repair vs tuned:dsg-state | total_chars | -315.3333 | [-1312.5608, +701.6592] | no | 11–19 | 30 |
+| tuned:dsg-repair vs tuned:dsg-state | prompt_tokens_last | +11.9667 | [-15.4008, +40.2667] | no | 17–13 | 30 |
+| tuned:dsg-repair vs base:full-context | violation_rate | -0.1875 | [-0.2583, -0.1083] | **yes** | 3–22 | 30 |
+| tuned:dsg-repair vs base:full-context | restatement_rate | -0.7375 | [-0.7875, -0.6833] | **yes** | 0–30 | 30 |
+| tuned:dsg-repair vs base:full-context | mean_first_violation | +8.3689 | [+5.7915, +10.9789] | **yes** | 26–2 | 30 |
+| tuned:dsg-repair vs base:full-context | total_chars | -17314.0333 | [-18743.8392, -15911.0558] | **yes** | 0–30 | 30 |
+| tuned:dsg-repair vs base:full-context | prompt_tokens_last | -4918.7000 | [-5027.1008, -4797.4300] | **yes** | 0–30 | 30 |
+
+## Continuity errors on real generated text
+
+| condition | ch | fact | contradicting text |
+|---|---|---|---|
+| base:append-only-state | 20 | `s00f02` | …e member of the old town hall with silver hair and silver eyes, followed closely behind them, his silver-tipped cane gently tappin… |
+| base:none | 20 | `s01f03` | …un peeking through the clouds to cast a golden glow over the village. Silas, the village blacksmith, who had always been more a fi… |
+| base:full-context | 20 | `s01f00` | …pass, the brass token that guided you, has indeed pointed north, as it promised.” Maud’s grey eyes widened, her curiosity piqued. … |
+| base:append-only-state | 20 | `s01f07` | …e, his mind wandering to distant places and the wonders he yearned to explore. Silas, the bookbinder, his weathered hands busily m… |
+| base:rolling-summary | 20 | `s02f03` | …bride, the air is thick with the acrid smell of tar and burning pitch. Thaddeus, a clockmaker with amber eyes, and Barnaby, a youn… |
+| base:full-context | 20 | `s02f00` | …arried him towards the looming cathedral, its towering spire a ghostly sentry against the grey sky. Inside,… |
+| base:append-only-state | 20 | `s02f05` | …of the cathedral, his hair now a vibrant amber that seemed to mirror the fading embers of the city. He moved with a quiet grace, a… |
+| base:rolling-summary | 20 | `s03f03` | …, his hands trembling as he holds the brass flask that had belonged to his grandfather, a clockmaker. He had brought the flask to … |
+| base:full-context | 20 | `s03f00` | …y to preserve our legacy.” Barnaby, his grey eyes bright with excitement and curiosity, appeared at their feet, his small frame da… |
+| base:append-only-state | 20 | `s03f05` | …weighed him down with fear and the realization of his impending solitude. Thaddeus's deep blue eyes reflected the turmoil within h… |
+| base:none | 20 | `s04f02` | …renewed hope. As they emerged from the tunnel, the sun was beginning to rise, casting a golden glow over the canal. Ceridwen took … |
+| base:full-context | 20 | `s04f05` | …ely above a whisper. "They might have a solution, or at least some ideas." Ceridwen, her amber eyes fixed on the task at hand, nod… |
+
+## Figures
+
+![canon](figures/gen1-canon-violation.png)
+![final](figures/gen2-final.png)