GOVINDFROM commited on
Commit
362e668
·
verified ·
1 Parent(s): c056886

selection run: report

Browse files
report/gen-v5-selection/figures/gen1-canon-violation.pdf ADDED
Binary file (15.8 kB). View file
 
report/gen-v5-selection/figures/gen1-canon-violation.png ADDED
report/gen-v5-selection/figures/gen2-final.pdf ADDED
Binary file (17.4 kB). View file
 
report/gen-v5-selection/figures/gen2-final.png ADDED
report/gen-v5-selection/generation.md ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Generation: consistency at length
2
+
3
+ 30 stories × 4 memory conditions × 20 chapters, Qwen/Qwen2.5-3B-Instruct. Canon is stated in chapter 1 only; from chapter 2 each condition keeps whatever its memory carries. Violations are deterministic string tests against facts we planted, so no model or human judgement enters the measurement.
4
+
5
+ ## Means over stories
6
+
7
+ | condition | n | on_premise | violation_rate | retention | restatement_rate | canon_capture | mean_first_violation | total_chars | prompt_tokens_last |
8
+ |---|---|---|---|---|---|---|---|---|---|
9
+ | base:last-chapter | 30 | 1.000 | 0.354 | 0.646 | 0.833 | 0.000 | 2.689 | 75,330 | 1,005 |
10
+ | base:full-context | 30 | 1.000 | 0.400 | 0.600 | 0.842 | 0.000 | 2.650 | 75,779 | 5,360 |
11
+ | base:last-chapter@bo4-random | 30 | 1.000 | 0.296 | 0.704 | 0.808 | 0.000 | 3.514 | 76,207 | 1,012 |
12
+ | base:last-chapter@bo4-graph | 30 | 1.000 | 0.300 | 0.700 | 0.796 | 0.000 | 3.828 | 75,548 | 1,017 |
13
+
14
+ ## Paired comparisons (bootstrap over stories)
15
+
16
+ | comparison | metric | mean Δ | 95% CI | excludes 0 | wins | n |
17
+ |---|---|---|---|---|---|---|
18
+ | base:last-chapter@bo4-graph vs base:last-chapter | violation_rate | -0.0542 | [-0.1292, +0.0250] | no | 8–15 | 30 |
19
+ | base:last-chapter@bo4-graph vs base:last-chapter | restatement_rate | -0.0375 | [-0.1042, +0.0292] | no | 8–12 | 30 |
20
+ | base:last-chapter@bo4-graph vs base:last-chapter | mean_first_violation | +1.1383 | [-1.2317, +3.6961] | no | 13–11 | 30 |
21
+ | base:last-chapter@bo4-graph vs base:last-chapter | total_chars | +217.8000 | [-927.5825, +1380.0842] | no | 15–15 | 30 |
22
+ | base:last-chapter@bo4-graph vs base:last-chapter | prompt_tokens_last | +11.8667 | [-0.0333, +30.0667] | no | 5–2 | 30 |
23
+ | base:last-chapter@bo4-random vs base:last-chapter | violation_rate | -0.0583 | [-0.1417, +0.0250] | no | 11–17 | 30 |
24
+ | base:last-chapter@bo4-random vs base:last-chapter | restatement_rate | -0.0250 | [-0.1042, +0.0500] | no | 9–13 | 30 |
25
+ | base:last-chapter@bo4-random vs base:last-chapter | mean_first_violation | +0.8244 | [-1.1567, +2.8345] | no | 16–10 | 30 |
26
+ | base:last-chapter@bo4-random vs base:last-chapter | total_chars | +876.1667 | [-94.2483, +1809.7017] | no | 21–9 | 30 |
27
+ | base:last-chapter@bo4-random vs base:last-chapter | prompt_tokens_last | +6.1000 | [-11.5000, +25.7675] | no | 7–4 | 30 |
28
+ | base:last-chapter@bo4-graph vs base:last-chapter@bo4-random | violation_rate | +0.0042 | [-0.0583, +0.0625] | no | 8–6 | 30 |
29
+ | base:last-chapter@bo4-graph vs base:last-chapter@bo4-random | restatement_rate | -0.0125 | [-0.0708, +0.0458] | no | 7–10 | 30 |
30
+ | base:last-chapter@bo4-graph vs base:last-chapter@bo4-random | mean_first_violation | +0.3139 | [-1.5890, +2.4112] | no | 12–13 | 30 |
31
+ | base:last-chapter@bo4-graph vs base:last-chapter@bo4-random | total_chars | -658.3667 | [-1738.1367, +358.3625] | no | 14–16 | 30 |
32
+ | base:last-chapter@bo4-graph vs base:last-chapter@bo4-random | prompt_tokens_last | +5.7667 | [-0.1333, +17.0333] | no | 4–4 | 30 |
33
+ | base:last-chapter@bo4-graph vs base:full-context | violation_rate | -0.1000 | [-0.1833, -0.0167] | **yes** | 8–17 | 30 |
34
+ | base:last-chapter@bo4-graph vs base:full-context | restatement_rate | -0.0458 | [-0.1167, +0.0208] | no | 8–12 | 30 |
35
+ | base:last-chapter@bo4-graph vs base:full-context | mean_first_violation | +1.1782 | [-0.4989, +3.1011] | no | 16–11 | 30 |
36
+ | base:last-chapter@bo4-graph vs base:full-context | total_chars | -230.8333 | [-1407.6133, +934.2758] | no | 15–15 | 30 |
37
+ | base:last-chapter@bo4-graph vs base:full-context | prompt_tokens_last | -4342.7667 | [-4398.5342, -4271.4983] | **yes** | 0–30 | 30 |
38
+
39
+ ## Continuity errors on real generated text
40
+
41
+ | condition | ch | fact | contradicting text |
42
+ |---|---|---|---|
43
+ | base:full-context | 20 | `s00f05` | …leather of the book his silent companion. Silas stood at the door of the village inn, his green eyes scanning the shadows, his unc… |
44
+ | base:full-context | 20 | `s01f07` | …he best routes, and we can use that knowledge to rebuild the road." Silas, the bookbinder, leaned back in his chair, his eyes narr… |
45
+ | base:full-context | 20 | `s04f03` | …narrowing. Ceridwen stood up, her hands still clutching the hammer. “Sir, as the village blacksmith, it is my duty to ensure the l… |
46
+ | base:full-context | 20 | `s05f00` | …en glass and shattered shards that littered the floor. Rowena stood before Thaddeus, her hazel eyes locking onto his. The weight o… |
47
+ | base:full-context | 20 | `s07f00` | …errible consequences if it fell into the wrong hands." Odell approached the lantern, his green eyes sharp and focused, his express… |
48
+ | base:full-context | 20 | `s08f00` | …th practiced ease, and Halloran’s posture remained rigid. Ceridwen stood by the door, her amber eyes fixed on the key in Isolde’s … |
49
+ | base:full-context | 20 | `s09f03` | …dwen, and she worked as a clockmaker, her hands steady, her gaze never wavering from the intricate gears and springs that lay… |
50
+ | base:full-context | 20 | `s10f00` | …und of a seagull’s mournful cry. Outside, the world was a blur of grey, and the only signs of life were the occasional flicker of … |
51
+ | base:full-context | 20 | `s11f00` | …rupting the hushed conversation. Hesper sat cross-legged by the hearth, her green eyes unwavering as she watched the dancing light… |
52
+ | base:full-context | 20 | `s13f00` | …Gideon's apothecary shop had been kicked open, and the contents lay in disarray. Miriam's hazel eyes narrowed, her grip tightening… |
53
+ | base:full-context | 20 | `s15f00` | …without a trace. He’s here for a reason, and we must find out what it is." Halloran, his blue eyes reflecting the turmoil in his h… |
54
+ | base:full-context | 20 | `s16f00` | …, and together, they began to prepare for the task ahead. Miriam balanced her scales, her green eyes reflecting the hidden determi… |
55
+
56
+ ## Figures
57
+
58
+ ![canon](figures/gen1-canon-violation.png)
59
+ ![final](figures/gen2-final.png)