reading study: PDNC 1.5B/3B/7B + LitBank: report

.gitattributes

@@ -33,3 +33,13 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+report/litbank-qwen7b/figures/fig1-main-results.png filter=lfs diff=lfs merge=lfs -text
+report/litbank-qwen7b/figures/fig1b-ladder.png filter=lfs diff=lfs merge=lfs -text
+report/litbank-qwen7b/figures/fig3-paired-deltas.png filter=lfs diff=lfs merge=lfs -text
+report/mock-smoke/figures/fig1-main-results.png filter=lfs diff=lfs merge=lfs -text
+report/pdnc-qwen1.5b/figures/fig1-main-results.png filter=lfs diff=lfs merge=lfs -text
+report/pdnc-qwen1.5b/figures/fig1b-ladder.png filter=lfs diff=lfs merge=lfs -text
+report/pdnc-qwen1.5b/figures/fig3-paired-deltas.png filter=lfs diff=lfs merge=lfs -text
+report/pdnc-qwen3b/figures/fig1-main-results.png filter=lfs diff=lfs merge=lfs -text
+report/pdnc-qwen3b/figures/fig1b-ladder.png filter=lfs diff=lfs merge=lfs -text
+report/pdnc-qwen3b/figures/fig3-paired-deltas.png filter=lfs diff=lfs merge=lfs -text

report/comparison/comparison.md

@@ -0,0 +1,55 @@
+# Cross-run comparison
+
+Runs included: litbank-qwen7b, pdnc-qwen1.5b, pdnc-qwen3b
+
+## Extraction yield per run
+
+A smaller model does not merely extract *worse*, it extracts *less*: a window whose output cannot be parsed contributes nothing to any policy. The scale comparison below is therefore partly confounded by yield, so the parse rate is reported alongside it rather than buried.
+
+| run | books | windows | parse failures | parse rate |
+|---|---|---|---|---|
+| litbank-qwen7b | 100 | 881.0 | 1 | 0.999 |
+| pdnc-qwen1.5b | 28 | 4,063.0 | 837 | 0.794 |
+| pdnc-qwen3b | 28 | 4,063.0 | 18 | 0.996 |
+
+## DSG (`dsg-full`) versus the standard incremental pipeline (`append-only`)
+
+| run | books | metric | mean Δ | 95% CI | excludes 0 | wins |
+|---|---|---|---|---|---|---|
+| litbank-qwen7b | 100 | inconsistent_slot_rate | -0.1814 | [-0.2275, -0.1389] | **yes** | 0–55 |
+| litbank-qwen7b | 100 | violations_per_100w | -104.6646 | [-140.1145, -73.0042] | **yes** | 0–55 |
+| litbank-qwen7b | 100 | conll_f1 | -0.0027 | [-0.0090, +0.0034] | no | 9–15 |
+| litbank-qwen7b | 100 | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 |
+| litbank-qwen7b | 100 | rollback | +0.0500 | [+0.0100, +0.1000] | **yes** | 5–0 |
+| litbank-qwen7b | 100 | monotone_fraction | -0.0112 | [-0.0147, -0.0081] | **yes** | 0–44 |
+| pdnc-qwen3b | 28 | inconsistent_slot_rate | -0.4233 | [-0.4694, -0.3744] | **yes** | 0–28 |
+| pdnc-qwen3b | 28 | violations_per_100w | -13826.1650 | [-19694.3909, -8538.2675] | **yes** | 0–28 |
+| pdnc-qwen3b | 28 | conll_f1 | -0.0050 | [-0.0108, +0.0002] | no | 5–15 |
+| pdnc-qwen3b | 28 | mention_acc | -0.0023 | [-0.0168, +0.0115] | no | 10–9 |
+| pdnc-qwen3b | 28 | rollback | +32.9286 | [+24.5000, +42.3214] | **yes** | 28–0 |
+| pdnc-qwen3b | 28 | monotone_fraction | -0.1866 | [-0.2130, -0.1586] | **yes** | 0–28 |
+| pdnc-qwen1.5b | 28 | inconsistent_slot_rate | -0.1067 | [-0.1638, -0.0570] | **yes** | 0–12 |
+| pdnc-qwen1.5b | 28 | violations_per_100w | -61.7864 | [-110.2998, -26.2329] | **yes** | 0–13 |
+| pdnc-qwen1.5b | 28 | conll_f1 | -0.0026 | [-0.0070, +0.0012] | no | 7–8 |
+| pdnc-qwen1.5b | 28 | mention_acc | -0.0171 | [-0.0338, -0.0038] | **yes** | 7–9 |
+| pdnc-qwen1.5b | 28 | rollback | +0.0357 | [+0.0000, +0.1071] | no | 1–0 |
+| pdnc-qwen1.5b | 28 | monotone_fraction | -0.0050 | [-0.0091, -0.0019] | **yes** | 0–11 |
+
+## Deferred versus eager commitment (`dsg-full` − `dsg-eager`)
+
+| run | books | metric | mean Δ | 95% CI | excludes 0 | wins |
+|---|---|---|---|---|---|---|
+| litbank-qwen7b | 100 | rollback | -0.6000 | [-0.7600, -0.4500] | **yes** | 0–42 |
+| litbank-qwen7b | 100 | monotone_fraction | +0.0095 | [+0.0073, +0.0118] | **yes** | 64–0 |
+| litbank-qwen7b | 100 | conll_f1 | -0.0059 | [-0.0111, -0.0016] | **yes** | 3–7 |
+| pdnc-qwen3b | 28 | rollback | -2.5714 | [-3.7143, -1.7500] | **yes** | 0–25 |
+| pdnc-qwen3b | 28 | monotone_fraction | +0.0253 | [+0.0194, +0.0333] | **yes** | 28–0 |
+| pdnc-qwen3b | 28 | conll_f1 | -0.0015 | [-0.0063, +0.0025] | no | 5–4 |
+| pdnc-qwen1.5b | 28 | rollback | -5.5000 | [-6.7143, -4.3214] | **yes** | 0–28 |
+| pdnc-qwen1.5b | 28 | monotone_fraction | +0.0378 | [+0.0307, +0.0463] | **yes** | 28–0 |
+| pdnc-qwen1.5b | 28 | conll_f1 | -0.0001 | [-0.0035, +0.0031] | no | 5–2 |
+
+## Figures
+
+![scale](figures/fig6-model-scale.png)
+![length](figures/fig7-length-dependence.png)

report/litbank-qwen7b/results.md

@@ -0,0 +1,180 @@
+# Dynamic Story Graph — results
+
+Books: **100** · policies: window-only, append-only, dsg-merge, dsg-eager, dsg-full, retrospective · scoring time: 72.2s
+
+## Means over books
+
+| policy | conll_f1 | mention_acc | mention_acc_answered | b3_f1_multi | coverage | fragmentation | conflation | inconsistent_slot_rate | speaker_acc | speaker_acc_matched | violations_per_100w | monotone_fraction | rollback | live_entities |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| window-only | 0.573 | 0.000 | 0.000 | 0.523 | 0.440 | 0.180 | 0.009 | 0.091 | 0.000 | 0.000 | 1092.9 | 1.000 | 0.000 | 47.920 |
+| append-only | 0.654 | 0.000 | 0.000 | 0.586 | 0.440 | 0.084 | 0.038 | 0.181 | 0.000 | 0.000 | 104.7 | 1.000 | 0.000 | 28.710 |
+| dsg-merge | 0.657 | 0.000 | 0.000 | 0.590 | 0.441 | 0.078 | 0.064 | 0.190 | 0.000 | 0.000 | 109.5 | 0.991 | 0.600 | 28.080 |
+| dsg-eager | 0.657 | 0.000 | 0.000 | 0.590 | 0.441 | 0.078 | 0.064 | 0.000 | 0.000 | 0.000 | 1.028 | 0.979 | 0.650 | 28.080 |
+| dsg-full | 0.651 | 0.000 | 0.000 | 0.585 | 0.441 | 0.087 | 0.061 | 0.000 | 0.000 | 0.000 | 0.000 | 0.989 | 0.050 | 28.440 |
+| retrospective | 0.655 | 0.000 | 0.000 | 0.588 | 0.441 | 0.080 | 0.063 | 0.000 | 0.000 | 0.000 | 0.000 | 0.984 | 0.040 | 28.310 |
+
+## Paired comparisons (bootstrap over books)
+
+| comparison | metric | mean Δ | 95% CI | excludes 0 | wins | n |
+|---|---|---|---|---|---|---|
+| append-only vs window-only | conll_f1 | +0.0807 | [+0.0622, +0.1003] | **yes** | 50–9 | 100 |
+| append-only vs window-only | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| append-only vs window-only | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| append-only vs window-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| append-only vs window-only | violations_per_100w | -988.2511 | [-1150.0577, -835.6563] | **yes** | 2–96 | 100 |
+| append-only vs window-only | inconsistent_slot_rate | +0.0908 | [+0.0621, +0.1249] | **yes** | 40–1 | 100 |
+| append-only vs window-only | rollback | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| append-only vs window-only | monotone_fraction | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| append-only vs window-only | fragmentation | -0.0957 | [-0.1233, -0.0706] | **yes** | 0–49 | 100 |
+| append-only vs window-only | conflation | +0.0291 | [+0.0167, +0.0431] | **yes** | 22–0 | 100 |
+| dsg-merge vs append-only | conll_f1 | +0.0032 | [-0.0011, +0.0079] | no | 8–9 | 100 |
+| dsg-merge vs append-only | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-merge vs append-only | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-merge vs append-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-merge vs append-only | violations_per_100w | +4.7889 | [+1.0000, +9.7725] | **yes** | 7–0 | 100 |
+| dsg-merge vs append-only | inconsistent_slot_rate | +0.0086 | [+0.0014, +0.0183] | **yes** | 7–0 | 100 |
+| dsg-merge vs append-only | rollback | +0.6000 | [+0.4500, +0.7600] | **yes** | 42–0 | 100 |
+| dsg-merge vs append-only | monotone_fraction | -0.0086 | [-0.0109, -0.0064] | **yes** | 0–42 | 100 |
+| dsg-merge vs append-only | fragmentation | -0.0067 | [-0.0139, -0.0007] | **yes** | 1–6 | 100 |
+| dsg-merge vs append-only | conflation | +0.0262 | [+0.0125, +0.0427] | **yes** | 16–0 | 100 |
+| dsg-eager vs dsg-merge | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-eager vs dsg-merge | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-eager vs dsg-merge | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-eager vs dsg-merge | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-eager vs dsg-merge | violations_per_100w | -108.4258 | [-144.6174, -76.5869] | **yes** | 0–57 | 100 |
+| dsg-eager vs dsg-merge | inconsistent_slot_rate | -0.1901 | [-0.2359, -0.1473] | **yes** | 0–57 | 100 |
+| dsg-eager vs dsg-merge | rollback | +0.0500 | [+0.0100, +0.1000] | **yes** | 5–0 | 100 |
+| dsg-eager vs dsg-merge | monotone_fraction | -0.0121 | [-0.0160, -0.0087] | **yes** | 0–44 | 100 |
+| dsg-eager vs dsg-merge | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-eager vs dsg-merge | conflation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs dsg-eager | conll_f1 | -0.0059 | [-0.0111, -0.0016] | **yes** | 3–7 | 100 |
+| dsg-full vs dsg-eager | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs dsg-eager | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs dsg-eager | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs dsg-eager | violations_per_100w | -1.0278 | [-2.8333, +0.0000] | no | 0–2 | 100 |
+| dsg-full vs dsg-eager | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs dsg-eager | rollback | -0.6000 | [-0.7600, -0.4500] | **yes** | 0–42 | 100 |
+| dsg-full vs dsg-eager | monotone_fraction | +0.0095 | [+0.0073, +0.0118] | **yes** | 64–0 | 100 |
+| dsg-full vs dsg-eager | fragmentation | +0.0094 | [+0.0026, +0.0185] | **yes** | 7–0 | 100 |
+| dsg-full vs dsg-eager | conflation | -0.0035 | [-0.0076, -0.0003] | **yes** | 0–6 | 100 |
+| retrospective vs dsg-full | conll_f1 | +0.0045 | [+0.0004, +0.0093] | **yes** | 9–1 | 100 |
+| retrospective vs dsg-full | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| retrospective vs dsg-full | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| retrospective vs dsg-full | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| retrospective vs dsg-full | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| retrospective vs dsg-full | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| retrospective vs dsg-full | rollback | -0.0100 | [-0.0300, +0.0000] | no | 0–1 | 100 |
+| retrospective vs dsg-full | monotone_fraction | -0.0047 | [-0.0066, -0.0030] | **yes** | 4–40 | 100 |
+| retrospective vs dsg-full | fragmentation | -0.0066 | [-0.0150, -0.0007] | **yes** | 0–4 | 100 |
+| retrospective vs dsg-full | conflation | +0.0025 | [-0.0068, +0.0131] | no | 3–3 | 100 |
+| dsg-full vs append-only | conll_f1 | -0.0027 | [-0.0090, +0.0034] | no | 9–15 | 100 |
+| dsg-full vs append-only | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs append-only | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs append-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs append-only | violations_per_100w | -104.6646 | [-140.1145, -73.0042] | **yes** | 0–55 | 100 |
+| dsg-full vs append-only | inconsistent_slot_rate | -0.1814 | [-0.2275, -0.1389] | **yes** | 0–55 | 100 |
+| dsg-full vs append-only | rollback | +0.0500 | [+0.0100, +0.1000] | **yes** | 5–0 | 100 |
+| dsg-full vs append-only | monotone_fraction | -0.0112 | [-0.0147, -0.0081] | **yes** | 0–44 | 100 |
+| dsg-full vs append-only | fragmentation | +0.0026 | [-0.0060, +0.0112] | no | 8–5 | 100 |
+| dsg-full vs append-only | conflation | +0.0228 | [+0.0088, +0.0395] | **yes** | 16–4 | 100 |
+| dsg-full vs window-only | conll_f1 | +0.0780 | [+0.0595, +0.0974] | **yes** | 54–14 | 100 |
+| dsg-full vs window-only | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs window-only | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs window-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs window-only | violations_per_100w | -1092.9157 | [-1258.5972, -934.3395] | **yes** | 0–100 | 100 |
+| dsg-full vs window-only | inconsistent_slot_rate | -0.0906 | [-0.1247, -0.0608] | **yes** | 0–36 | 100 |
+| dsg-full vs window-only | rollback | +0.0500 | [+0.0100, +0.1000] | **yes** | 5–0 | 100 |
+| dsg-full vs window-only | monotone_fraction | -0.0112 | [-0.0147, -0.0081] | **yes** | 0–44 | 100 |
+| dsg-full vs window-only | fragmentation | -0.0931 | [-0.1199, -0.0686] | **yes** | 2–51 | 100 |
+| dsg-full vs window-only | conflation | +0.0518 | [+0.0333, +0.0726] | **yes** | 32–2 | 100 |
+| dsg-full vs dsg-merge | conll_f1 | -0.0059 | [-0.0111, -0.0016] | **yes** | 3–7 | 100 |
+| dsg-full vs dsg-merge | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs dsg-merge | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs dsg-merge | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs dsg-merge | violations_per_100w | -109.4535 | [-145.7021, -77.5386] | **yes** | 0–57 | 100 |
+| dsg-full vs dsg-merge | inconsistent_slot_rate | -0.1901 | [-0.2359, -0.1473] | **yes** | 0–57 | 100 |
+| dsg-full vs dsg-merge | rollback | -0.5500 | [-0.7100, -0.3900] | **yes** | 2–40 | 100 |
+| dsg-full vs dsg-merge | monotone_fraction | -0.0026 | [-0.0068, +0.0013] | no | 31–34 | 100 |
+| dsg-full vs dsg-merge | fragmentation | +0.0094 | [+0.0026, +0.0185] | **yes** | 7–0 | 100 |
+| dsg-full vs dsg-merge | conflation | -0.0035 | [-0.0076, -0.0003] | **yes** | 0–6 | 100 |
+| dsg-full vs retrospective | conll_f1 | -0.0045 | [-0.0093, -0.0004] | **yes** | 1–9 | 100 |
+| dsg-full vs retrospective | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs retrospective | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs retrospective | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs retrospective | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs retrospective | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-full vs retrospective | rollback | +0.0100 | [+0.0000, +0.0300] | no | 1–0 | 100 |
+| dsg-full vs retrospective | monotone_fraction | +0.0047 | [+0.0030, +0.0066] | **yes** | 40–4 | 100 |
+| dsg-full vs retrospective | fragmentation | +0.0066 | [+0.0007, +0.0150] | **yes** | 4–0 | 100 |
+| dsg-full vs retrospective | conflation | -0.0025 | [-0.0131, +0.0068] | no | 3–3 | 100 |
+| dsg-eager vs append-only | conll_f1 | +0.0032 | [-0.0011, +0.0079] | no | 8–9 | 100 |
+| dsg-eager vs append-only | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-eager vs append-only | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-eager vs append-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 |
+| dsg-eager vs append-only | violations_per_100w | -103.6369 | [-139.2292, -71.9828] | **yes** | 0–55 | 100 |
+| dsg-eager vs append-only | inconsistent_slot_rate | -0.1814 | [-0.2275, -0.1389] | **yes** | 0–55 | 100 |
+| dsg-eager vs append-only | rollback | +0.6500 | [+0.4900, +0.8200] | **yes** | 44–0 | 100 |
+| dsg-eager vs append-only | monotone_fraction | -0.0207 | [-0.0251, -0.0167] | **yes** | 0–65 | 100 |
+| dsg-eager vs append-only | fragmentation | -0.0067 | [-0.0139, -0.0007] | **yes** | 1–6 | 100 |
+| dsg-eager vs append-only | conflation | +0.0262 | [+0.0125, +0.0427] | **yes** | 16–0 | 100 |
+
+## Identity-resolution latency (measurement instrument)
+
+_not computed_
+
+## One character, traced (dsg-full)
+
+### 1023_bleak_house_brat
+
+_no state recorded for Chancery_
+
+### 105_persuasion_brat
+
+**ELLIOT** — what the reader believes, and when.
+
+| read at | predicate | value | held until | status | source |
+|---|---|---|---|---|---|
+| 0.22 | other:found_enough_in_her_duties_friends_and_children | to attach her to life | — | believed | reported |
+| 0.22 | other:had_judgements_and_conduct | never required indulgence | — | believed | reported |
+| 0.22 | other:humoured_softened_or_concealed_sir_walter_s_failings | for seventeen years | — | believed | reported |
+| 0.22 | other:was_an_excellent_woman | sensible and amiable | — | believed | reported |
+| 0.22 | other:was_not_the_very_happiest_being_in_the_world | herself | — | believed | reported |
+| 0.33 | other:deceased | 13 years ago | — | believed | narrated |
+| 0.33 | other:relies_on | Lady Russell | — | believed | narrated |
+| 0.33 | possesses | daughters | — | believed | narrated |
+| 0.33 | possesses | two eldest daughters | — | believed | narrated |
+
+
+## Revision events on real text (dsg-full)
+
+| op | book | position | change | evidence |
+|---|---|---|---|---|
+| `supersede` | 1023_bleak_house_brat | 0.30 | High Court of Chancery.location: 'Lincoln's Inn Hall' → 'Holborn Hill' | …technicalities, running their goat-hair and horsehair warded heads against walls of words and making a pretence of equity with serious faces, as playe… |
+| `supersede` | 1023_bleak_house_brat | 0.20 | Fog.location: 'down the river' → 'Temple Bar' | …the muddy streets are muddiest near that leaden-headed old obstruction, appropriate ornament for the threshold of a leaden-headed old corporation, Tem… |
+| `supersede` | 1023_bleak_house_brat | 0.00 | Fog.location: 'everywhere' → 'up the river' | …CHAPTER I In Chancery London. Michaelmas term lately over, and the Lord Chancellor sitting in Lincoln's Inn Hall. Implacable November weather. As much… |
+| `supersede` | 1023_bleak_house_brat | 0.00 | Fog.location: 'up the river' → 'down the river' | …CHAPTER I In Chancery London. Michaelmas term lately over, and the Lord Chancellor sitting in Lincoln's Inn Hall. Implacable November weather. As much… |
+| `elaborate` | 1023_bleak_house_brat | 0.20 | provisional -> committed | …e deliriously found themselves made parties in Jarndyce and Jarndyce without knowing how or why; whole families have inherited legendary hatreds with … |
+| `elaborate` | 1023_bleak_house_brat | 0.20 | provisional -> committed | …e deliriously found themselves made parties in Jarndyce and Jarndyce without knowing how or why; whole families have inherited legendary hatreds with … |
+| `elaborate` | 1023_bleak_house_brat | 0.00 | provisional -> committed | …the muddy streets are muddiest near that leaden-headed old obstruction, appropriate ornament for the threshold of a leaden-headed old corporation, Tem… |
+| `elaborate` | 105_persuasion_brat | 0.89 | provisional -> committed | …rawing-back of youth; and, in one of their spring excursions to London, when Elizabeth was in her first bloom, Mr Elliot had been forced into the intr… |
+| `elaborate` | 105_persuasion_brat | 0.44 | provisional -> committed | …y different were her delicate features and mild dark eyes from his own), there could be nothing in them, now that she was faded and thin, to excite hi… |
+| `elaborate` | 105_persuasion_brat | 0.33 | provisional -> committed | …y different were her delicate features and mild dark eyes from his own), there could be nothing in them, now that she was faded and thin, to excite hi… |
+| `elaborate` | 105_persuasion_brat | 0.22 | provisional -> committed | …teen and fourteen, was an awful legacy for a mother to bequeath, an awful charge rather, to confide to the authority and guidance of a conceited, sill… |
+| `elaborate` | 105_persuasion_brat | 0.11 | provisional -> committed | …rawing-back of youth; and, in one of their spring excursions to London, when Elizabeth was in her first bloom, Mr Elliot had been forced into the intr… |
+| `elaborate` | 105_persuasion_brat | 0.11 | provisional -> committed | …y different were her delicate features and mild dark eyes from his own), there could be nothing in them, now that she was faded and thin, to excite hi… |
+| `elaborate` | 105_persuasion_brat | 0.11 | provisional -> committed | …teen and fourteen, was an awful legacy for a mother to bequeath, an awful charge rather, to confide to the authority and guidance of a conceited, sill… |
+| `elaborate` | 105_persuasion_brat | 0.11 | provisional -> committed | …ather, (having met with one or two private disappointments in very unreasonable applications), prided himself on remaining single for his dear daughte… |
+| `elaborate` | 105_persuasion_brat | 0.00 | provisional -> committed | …rawing-back of youth; and, in one of their spring excursions to London, when Elizabeth was in her first bloom, Mr Elliot had been forced into the intr… |
+| `supersede` | 1064_the_masque_of_the_red_death_brat | 0.78 | the company.emotion: 'a sensation' → 'disapprobation' | …ll of the clock; and thus it happened, perhaps, that more of thought crept, with more of time, into the meditations of the thoughtful among those who … |
+| `supersede` | 1064_the_masque_of_the_red_death_brat | 0.78 | the company.emotion: 'disapprobation' → 'surprise' | …ll of the clock; and thus it happened, perhaps, that more of thought crept, with more of time, into the meditations of the thoughtful among those who … |
+| `elaborate` | 1064_the_masque_of_the_red_death_brat | 0.00 | provisional -> committed | …The" Red Death" had long devastated the country. No pestilence had ever been so fatal, or so hideous. Blood was its Avatar and its seal -- the redness… |
+| `merge_provisional` | 110_tess_of_the_durbervilles_a_pure_woman_brat | 0.12 | absorbed e00009: link 'the parson' -> 'Parson Tringham' | …f beer wi' me on the strength o' t, Pa' son Tringham? There's a very pretty brew in tap at The Pure Drop -- though, to be sure, not so good as at Roll… |
+
+## Figures
+
+![main](figures/fig1-main-results.png)
+![ladder](figures/fig1b-ladder.png)
+![prefix](figures/fig2-prefix-curve.png)
+![paired](figures/fig3-paired-deltas.png)
+![revision](figures/fig4-revision-profile.png)
+![violations](figures/fig5-violation-growth.png)

report/mock-smoke/results.md

@@ -0,0 +1,94 @@
+# Dynamic Story Graph — results
+
+Books: **3** · policies: window-only, append-only, dsg-merge, dsg-eager, dsg-full, retrospective · scoring time: 32.3s
+
+## Means over books
+
+| policy | conll_f1 | b3_f1_multi | b3_f1 | coverage | fragmentation | conflation | speaker_acc | speaker_acc_matched | violations_per_100w | monotone_fraction | rollback | live_entities |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| window-only | 0.462 | 0.543 | 0.751 | 0.404 | 0.167 | 0.000 | 0.031 | 0.223 | 29847.6 | 1.000 | 0.000 | 233.0 |
+| append-only | 0.517 | 0.583 | 0.773 | 0.404 | 0.042 | 0.042 | 0.032 | 0.234 | 591.5 | 1.000 | 0.000 | 107.7 |
+| dsg-merge | 0.517 | 0.583 | 0.773 | 0.404 | 0.042 | 0.042 | 0.032 | 0.234 | 591.5 | 1.000 | 0.000 | 107.7 |
+| dsg-eager | 0.517 | 0.583 | 0.773 | 0.404 | 0.042 | 0.042 | 0.032 | 0.234 | 0.000 | 0.956 | 0.000 | 107.7 |
+| dsg-full | 0.517 | 0.583 | 0.773 | 0.404 | 0.042 | 0.042 | 0.032 | 0.234 | 0.000 | 0.965 | 0.000 | 107.7 |
+| retrospective | 0.517 | 0.583 | 0.773 | 0.404 | 0.042 | 0.042 | 0.032 | 0.234 | 0.000 | 0.944 | 0.000 | 107.7 |
+
+## Paired comparisons (bootstrap over books)
+
+| comparison | metric | mean Δ | 95% CI | excludes 0 | wins | n |
+|---|---|---|---|---|---|---|
+| dsg-full vs append-only | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs append-only | b3_f1_multi | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs append-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs append-only | violations_per_100w | -591.4530 | [-1738.4615, +0.0000] | no | 0–2 | 3 |
+| dsg-full vs append-only | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs append-only | monotone_fraction | -0.0355 | [-0.1000, +0.0000] | no | 0–2 | 3 |
+| dsg-full vs window-only | conll_f1 | +0.0546 | [+0.0000, +0.1639] | no | 1–0 | 3 |
+| dsg-full vs window-only | b3_f1_multi | +0.0397 | [+0.0000, +0.1191] | no | 1–0 | 3 |
+| dsg-full vs window-only | speaker_acc_matched | +0.0109 | [+0.0000, +0.0326] | no | 1–0 | 3 |
+| dsg-full vs window-only | violations_per_100w | -29847.6496 | [-35979.4872, -23638.4615] | **yes** | 0–3 | 3 |
+| dsg-full vs window-only | fragmentation | -0.1250 | [-0.3750, +0.0000] | no | 0–1 | 3 |
+| dsg-full vs window-only | monotone_fraction | -0.0355 | [-0.1000, +0.0000] | no | 0–2 | 3 |
+| dsg-full vs dsg-merge | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs dsg-merge | b3_f1_multi | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs dsg-merge | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs dsg-merge | violations_per_100w | -591.4530 | [-1738.4615, +0.0000] | no | 0–2 | 3 |
+| dsg-full vs dsg-merge | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs dsg-merge | monotone_fraction | -0.0355 | [-0.1000, +0.0000] | no | 0–2 | 3 |
+| dsg-full vs dsg-eager | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs dsg-eager | b3_f1_multi | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs dsg-eager | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs dsg-eager | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs dsg-eager | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs dsg-eager | monotone_fraction | +0.0085 | [+0.0000, +0.0240] | no | 2–0 | 3 |
+| dsg-full vs retrospective | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs retrospective | b3_f1_multi | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs retrospective | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs retrospective | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs retrospective | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-full vs retrospective | monotone_fraction | +0.0205 | [+0.0000, +0.0584] | no | 2–0 | 3 |
+| dsg-eager vs append-only | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-eager vs append-only | b3_f1_multi | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-eager vs append-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-eager vs append-only | violations_per_100w | -591.4530 | [-1738.4615, +0.0000] | no | 0–2 | 3 |
+| dsg-eager vs append-only | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+| dsg-eager vs append-only | monotone_fraction | -0.0439 | [-0.1240, +0.0000] | no | 0–2 | 3 |
+| append-only vs window-only | conll_f1 | +0.0546 | [+0.0000, +0.1639] | no | 1–0 | 3 |
+| append-only vs window-only | b3_f1_multi | +0.0397 | [+0.0000, +0.1191] | no | 1–0 | 3 |
+| append-only vs window-only | speaker_acc_matched | +0.0109 | [+0.0000, +0.0326] | no | 1–0 | 3 |
+| append-only vs window-only | violations_per_100w | -29256.1966 | [-34241.0256, -23602.5641] | **yes** | 0–3 | 3 |
+| append-only vs window-only | fragmentation | -0.1250 | [-0.3750, +0.0000] | no | 0–1 | 3 |
+| append-only vs window-only | monotone_fraction | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 |
+
+## Revision events on real text (dsg-full)
+
+| op | book | position | change | evidence |
+|---|---|---|---|---|
+| `elaborate` | AlicesAdventuresInWonderland | 0.97 | provisional -> committed | …ulders, that all the jurors were writing down "stupid things!" on their slates, and she could even make out that one of them didn't know how to spell … |
+| `elaborate` | AlicesAdventuresInWonderland | 0.97 | provisional -> committed | …ulders, that all the jurors were writing down "stupid things!" on their slates, and she could even make out that one of them didn't know how to spell … |
+| `elaborate` | AlicesAdventuresInWonderland | 0.97 | provisional -> committed | …ulders, that all the jurors were writing down "stupid things!" on their slates, and she could even make out that one of them didn't know how to spell … |
+| `elaborate` | AlicesAdventuresInWonderland | 0.90 | provisional -> committed | …have their tails in their mouths--and they're all over crumbs." "You're wrong about the crumbs," said the Mock Turtle: "crumbs would all wash off in t… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.88 | provisional -> committed | …should like to hear her try and repeat something now. Tell her to begin." He looked at the Gryphon as if he thought it had some kind of authority over… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.85 | provisional -> committed | …have their tails in their mouths--and they're all over crumbs." "You're wrong about the crumbs," said the Mock Turtle: "crumbs would all wash off in t… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.85 | provisional -> committed | …have their tails in their mouths--and they're all over crumbs." "You're wrong about the crumbs," said the Mock Turtle: "crumbs would all wash off in t… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.82 | provisional -> committed | …un?" said Alice. "Why, _she_," said the Gryphon. "It's all her fancy, that: they never executes nobody, you know. Come on!" "Everybody says 'come on!'… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.80 | provisional -> committed | …A cheap sort of present!" thought Alice. "I'm glad they don't give birthday presents like that!" But she did not venture to say it out loud. "Thinking… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.78 | provisional -> committed | …dn't be so stingy about it, you know--" She had quite forgotten the Duchess by this time, and was a little startled when she heard her voice close to … |
+| `elaborate` | AlicesAdventuresInWonderland | 0.72 | provisional -> committed | …t like the look of things at all, as the game was in such confusion that she never knew whether it was her turn or not. So she went in search of her h… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.70 | provisional -> committed | …A cheap sort of present!" thought Alice. "I'm glad they don't give birthday presents like that!" But she did not venture to say it out loud. "Thinking… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.70 | provisional -> committed | …with their heads!" and the procession moved on, three of the soldiers remaining behind to execute the unfortunate gardeners, who ran to Alice for prot… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.65 | provisional -> committed | …a thing as a drawing of a muchness?" "Really, now you ask me," said Alice, very much confused, "I don't think--" "Then you shouldn't talk," said the H… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.65 | provisional -> committed | …A cheap sort of present!" thought Alice. "I'm glad they don't give birthday presents like that!" But she did not venture to say it out loud. "Thinking… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.65 | provisional -> committed | …A cheap sort of present!" thought Alice. "I'm glad they don't give birthday presents like that!" But she did not venture to say it out loud. "Thinking… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.60 | provisional -> committed | …g a minute or two. "They couldn't have done that, you know," Alice gently remarked; "they'd have been ill." "So they were," said the Dormouse; "_very_… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.55 | provisional -> committed | …t like the look of things at all, as the game was in such confusion that she never knew whether it was her turn or not. So she went in search of her h… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.55 | provisional -> committed | …g a minute or two. "They couldn't have done that, you know," Alice gently remarked; "they'd have been ill." "So they were," said the Dormouse; "_very_… |
+| `elaborate` | AlicesAdventuresInWonderland | 0.55 | provisional -> committed | …g a minute or two. "They couldn't have done that, you know," Alice gently remarked; "they'd have been ill." "So they were," said the Dormouse; "_very_… |
+
+## Figures
+
+![main](figures/fig1-main-results.png)
+![prefix](figures/fig2-prefix-curve.png)
+![paired](figures/fig3-paired-deltas.png)
+![revision](figures/fig4-revision-profile.png)
+![violations](figures/fig5-violation-growth.png)

report/pdnc-qwen1.5b/results.md

@@ -0,0 +1,196 @@
+# Dynamic Story Graph — results
+
+Books: **28** · policies: window-only, append-only, dsg-merge, dsg-eager, dsg-full, retrospective · scoring time: 1044.2s
+
+## Means over books
+
+| policy | conll_f1 | mention_acc | mention_acc_answered | b3_f1_multi | coverage | fragmentation | conflation | inconsistent_slot_rate | speaker_acc | speaker_acc_matched | violations_per_100w | monotone_fraction | rollback | live_entities |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| window-only | 0.451 | 0.069 | 0.943 | 0.532 | 0.313 | 0.334 | 0.004 | 0.004 | 0.012 | 0.509 | 75607.3 | 1.000 | 0.000 | 322.1 |
+| append-only | 0.514 | 0.383 | 0.898 | 0.572 | 0.304 | 0.134 | 0.111 | 0.107 | 0.013 | 0.559 | 61.786 | 1.000 | 0.000 | 60.893 |
+| dsg-merge | 0.511 | 0.360 | 0.834 | 0.571 | 0.304 | 0.131 | 0.193 | 0.100 | 0.012 | 0.514 | 40.742 | 0.963 | 5.536 | 55.179 |
+| dsg-eager | 0.511 | 0.360 | 0.834 | 0.571 | 0.304 | 0.131 | 0.193 | 0.000 | 0.012 | 0.514 | 3.861 | 0.957 | 5.536 | 55.179 |
+| dsg-full | 0.511 | 0.366 | 0.825 | 0.570 | 0.304 | 0.135 | 0.173 | 0.000 | 0.013 | 0.525 | 0.000 | 0.995 | 0.036 | 57.536 |
+| retrospective | 0.507 | 0.416 | 0.799 | 0.566 | 0.299 | 0.132 | 0.200 | 0.000 | 0.013 | 0.530 | 0.000 | 0.991 | 0.036 | 57.714 |
+
+## Paired comparisons (bootstrap over books)
+
+| comparison | metric | mean Δ | 95% CI | excludes 0 | wins | n |
+|---|---|---|---|---|---|---|
+| append-only vs window-only | conll_f1 | +0.0628 | [+0.0453, +0.0815] | **yes** | 23–0 | 28 |
+| append-only vs window-only | mention_acc | +0.3143 | [+0.2662, +0.3635] | **yes** | 28–0 | 28 |
+| append-only vs window-only | mention_acc_answered | -0.0453 | [-0.0948, -0.0041] | **yes** | 7–18 | 28 |
+| append-only vs window-only | speaker_acc_matched | +0.0503 | [+0.0086, +0.0937] | **yes** | 16–7 | 28 |
+| append-only vs window-only | violations_per_100w | -75545.5557 | [-140608.4932, -28002.6100] | **yes** | 0–28 | 28 |
+| append-only vs window-only | inconsistent_slot_rate | +0.1027 | [+0.0530, +0.1595] | **yes** | 12–0 | 28 |
+| append-only vs window-only | rollback | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| append-only vs window-only | monotone_fraction | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| append-only vs window-only | fragmentation | -0.2002 | [-0.2526, -0.1499] | **yes** | 0–25 | 28 |
+| append-only vs window-only | conflation | +0.1077 | [+0.0601, +0.1682] | **yes** | 18–0 | 28 |
+| dsg-merge vs append-only | conll_f1 | -0.0025 | [-0.0061, +0.0005] | no | 2–9 | 28 |
+| dsg-merge vs append-only | mention_acc | -0.0232 | [-0.0404, -0.0089] | **yes** | 4–10 | 28 |
+| dsg-merge vs append-only | mention_acc_answered | -0.0638 | [-0.1137, -0.0249] | **yes** | 2–16 | 28 |
+| dsg-merge vs append-only | speaker_acc_matched | -0.0445 | [-0.0811, -0.0158] | **yes** | 1–9 | 28 |
+| dsg-merge vs append-only | violations_per_100w | -21.0449 | [-63.7384, +3.0417] | no | 1–2 | 28 |
+| dsg-merge vs append-only | inconsistent_slot_rate | -0.0070 | [-0.0225, +0.0046] | no | 1–2 | 28 |
+| dsg-merge vs append-only | rollback | +5.5357 | [+4.3571, +6.7500] | **yes** | 28–0 | 28 |
+| dsg-merge vs append-only | monotone_fraction | -0.0372 | [-0.0457, -0.0299] | **yes** | 0–28 | 28 |
+| dsg-merge vs append-only | fragmentation | -0.0032 | [-0.0097, +0.0000] | no | 0–1 | 28 |
+| dsg-merge vs append-only | conflation | +0.0816 | [+0.0311, +0.1533] | **yes** | 11–0 | 28 |
+| dsg-eager vs dsg-merge | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| dsg-eager vs dsg-merge | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| dsg-eager vs dsg-merge | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| dsg-eager vs dsg-merge | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| dsg-eager vs dsg-merge | violations_per_100w | -36.8808 | [-60.9653, -16.2603] | **yes** | 0–11 | 28 |
+| dsg-eager vs dsg-merge | inconsistent_slot_rate | -0.0997 | [-0.1563, -0.0501] | **yes** | 0–11 | 28 |
+| dsg-eager vs dsg-merge | rollback | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| dsg-eager vs dsg-merge | monotone_fraction | -0.0057 | [-0.0104, -0.0022] | **yes** | 0–11 | 28 |
+| dsg-eager vs dsg-merge | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| dsg-eager vs dsg-merge | conflation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| dsg-full vs dsg-eager | conll_f1 | -0.0001 | [-0.0035, +0.0031] | no | 5–2 | 28 |
+| dsg-full vs dsg-eager | mention_acc | +0.0062 | [+0.0010, +0.0127] | **yes** | 9–2 | 28 |
+| dsg-full vs dsg-eager | mention_acc_answered | -0.0091 | [-0.0372, +0.0119] | no | 11–5 | 28 |
+| dsg-full vs dsg-eager | speaker_acc_matched | +0.0109 | [+0.0000, +0.0272] | no | 3–0 | 28 |
+| dsg-full vs dsg-eager | violations_per_100w | -3.8607 | [-9.6754, +0.0000] | no | 0–2 | 28 |
+| dsg-full vs dsg-eager | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| dsg-full vs dsg-eager | rollback | -5.5000 | [-6.7143, -4.3214] | **yes** | 0��28 | 28 |
+| dsg-full vs dsg-eager | monotone_fraction | +0.0378 | [+0.0307, +0.0463] | **yes** | 28–0 | 28 |
+| dsg-full vs dsg-eager | fragmentation | +0.0037 | [+0.0000, +0.0094] | no | 2–0 | 28 |
+| dsg-full vs dsg-eager | conflation | -0.0206 | [-0.0393, -0.0061] | **yes** | 0–7 | 28 |
+| retrospective vs dsg-full | conll_f1 | -0.0046 | [-0.0120, +0.0033] | no | 2–9 | 28 |
+| retrospective vs dsg-full | mention_acc | +0.0503 | [+0.0380, +0.0644] | **yes** | 28–0 | 28 |
+| retrospective vs dsg-full | mention_acc_answered | -0.0264 | [-0.0546, -0.0018] | **yes** | 9–18 | 28 |
+| retrospective vs dsg-full | speaker_acc_matched | +0.0043 | [-0.0080, +0.0188] | no | 3–2 | 28 |
+| retrospective vs dsg-full | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| retrospective vs dsg-full | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| retrospective vs dsg-full | rollback | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| retrospective vs dsg-full | monotone_fraction | -0.0042 | [-0.0079, -0.0014] | **yes** | 0–12 | 28 |
+| retrospective vs dsg-full | fragmentation | -0.0029 | [-0.0186, +0.0107] | no | 4–3 | 28 |
+| retrospective vs dsg-full | conflation | +0.0275 | [-0.0009, +0.0632] | no | 7–4 | 28 |
+| dsg-full vs append-only | conll_f1 | -0.0026 | [-0.0070, +0.0012] | no | 7–8 | 28 |
+| dsg-full vs append-only | mention_acc | -0.0171 | [-0.0338, -0.0038] | **yes** | 7–9 | 28 |
+| dsg-full vs append-only | mention_acc_answered | -0.0729 | [-0.1298, -0.0284] | **yes** | 3–16 | 28 |
+| dsg-full vs append-only | speaker_acc_matched | -0.0336 | [-0.0612, -0.0104] | **yes** | 2–7 | 28 |
+| dsg-full vs append-only | violations_per_100w | -61.7864 | [-110.2998, -26.2329] | **yes** | 0–13 | 28 |
+| dsg-full vs append-only | inconsistent_slot_rate | -0.1067 | [-0.1638, -0.0570] | **yes** | 0–12 | 28 |
+| dsg-full vs append-only | rollback | +0.0357 | [+0.0000, +0.1071] | no | 1–0 | 28 |
+| dsg-full vs append-only | monotone_fraction | -0.0050 | [-0.0091, -0.0019] | **yes** | 0–11 | 28 |
+| dsg-full vs append-only | fragmentation | +0.0004 | [-0.0080, +0.0077] | no | 2–1 | 28 |
+| dsg-full vs append-only | conflation | +0.0610 | [+0.0058, +0.1358] | **yes** | 9–6 | 28 |
+| dsg-full vs window-only | conll_f1 | +0.0602 | [+0.0421, +0.0796] | **yes** | 23–1 | 28 |
+| dsg-full vs window-only | mention_acc | +0.2972 | [+0.2427, +0.3512] | **yes** | 28–0 | 28 |
+| dsg-full vs window-only | mention_acc_answered | -0.1182 | [-0.1942, -0.0549] | **yes** | 4–23 | 28 |
+| dsg-full vs window-only | speaker_acc_matched | +0.0167 | [-0.0357, +0.0688] | no | 14–11 | 28 |
+| dsg-full vs window-only | violations_per_100w | -75607.3421 | [-140680.6369, -28042.6731] | **yes** | 0–28 | 28 |
+| dsg-full vs window-only | inconsistent_slot_rate | -0.0040 | [-0.0119, +0.0000] | no | 0–1 | 28 |
+| dsg-full vs window-only | rollback | +0.0357 | [+0.0000, +0.1071] | no | 1–0 | 28 |
+| dsg-full vs window-only | monotone_fraction | -0.0050 | [-0.0091, -0.0019] | **yes** | 0–11 | 28 |
+| dsg-full vs window-only | fragmentation | -0.1997 | [-0.2529, -0.1475] | **yes** | 0–25 | 28 |
+| dsg-full vs window-only | conflation | +0.1687 | [+0.0948, +0.2621] | **yes** | 21–0 | 28 |
+| dsg-full vs dsg-merge | conll_f1 | -0.0001 | [-0.0035, +0.0031] | no | 5–2 | 28 |
+| dsg-full vs dsg-merge | mention_acc | +0.0062 | [+0.0010, +0.0127] | **yes** | 9–2 | 28 |
+| dsg-full vs dsg-merge | mention_acc_answered | -0.0091 | [-0.0372, +0.0119] | no | 11–5 | 28 |
+| dsg-full vs dsg-merge | speaker_acc_matched | +0.0109 | [+0.0000, +0.0272] | no | 3–0 | 28 |
+| dsg-full vs dsg-merge | violations_per_100w | -40.7415 | [-67.4195, -18.3721] | **yes** | 0–12 | 28 |
+| dsg-full vs dsg-merge | inconsistent_slot_rate | -0.0997 | [-0.1563, -0.0501] | **yes** | 0–11 | 28 |
+| dsg-full vs dsg-merge | rollback | -5.5000 | [-6.7143, -4.3214] | **yes** | 0–28 | 28 |
+| dsg-full vs dsg-merge | monotone_fraction | +0.0322 | [+0.0230, +0.0423] | **yes** | 27–1 | 28 |
+| dsg-full vs dsg-merge | fragmentation | +0.0037 | [+0.0000, +0.0094] | no | 2–0 | 28 |
+| dsg-full vs dsg-merge | conflation | -0.0206 | [-0.0393, -0.0061] | **yes** | 0–7 | 28 |
+| dsg-full vs retrospective | conll_f1 | +0.0046 | [-0.0033, +0.0120] | no | 9–2 | 28 |
+| dsg-full vs retrospective | mention_acc | -0.0503 | [-0.0644, -0.0380] | **yes** | 0–28 | 28 |
+| dsg-full vs retrospective | mention_acc_answered | +0.0264 | [+0.0018, +0.0546] | **yes** | 18–9 | 28 |
+| dsg-full vs retrospective | speaker_acc_matched | -0.0043 | [-0.0188, +0.0080] | no | 2–3 | 28 |
+| dsg-full vs retrospective | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| dsg-full vs retrospective | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| dsg-full vs retrospective | rollback | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 |
+| dsg-full vs retrospective | monotone_fraction | +0.0042 | [+0.0014, +0.0079] | **yes** | 12–0 | 28 |
+| dsg-full vs retrospective | fragmentation | +0.0029 | [-0.0107, +0.0186] | no | 3–4 | 28 |
+| dsg-full vs retrospective | conflation | -0.0275 | [-0.0632, +0.0009] | no | 4–7 | 28 |
+| dsg-eager vs append-only | conll_f1 | -0.0025 | [-0.0061, +0.0005] | no | 2–9 | 28 |
+| dsg-eager vs append-only | mention_acc | -0.0232 | [-0.0404, -0.0089] | **yes** | 4–10 | 28 |
+| dsg-eager vs append-only | mention_acc_answered | -0.0638 | [-0.1137, -0.0249] | **yes** | 2–16 | 28 |
+| dsg-eager vs append-only | speaker_acc_matched | -0.0445 | [-0.0811, -0.0158] | **yes** | 1–9 | 28 |
+| dsg-eager vs append-only | violations_per_100w | -57.9257 | [-105.9478, -23.6706] | **yes** | 0–12 | 28 |
+| dsg-eager vs append-only | inconsistent_slot_rate | -0.1067 | [-0.1638, -0.0570] | **yes** | 0–12 | 28 |
+| dsg-eager vs append-only | rollback | +5.5357 | [+4.3571, +6.7500] | **yes** | 28–0 | 28 |
+| dsg-eager vs append-only | monotone_fraction | -0.0428 | [-0.0510, -0.0354] | **yes** | 0–28 | 28 |
+| dsg-eager vs append-only | fragmentation | -0.0032 | [-0.0097, +0.0000] | no | 0–1 | 28 |
+| dsg-eager vs append-only | conflation | +0.0816 | [+0.0311, +0.1533] | **yes** | 11–0 | 28 |
+
+## Identity-resolution latency (measurement instrument)
+
+How long a novel holds a figure unnamed before the reader is told who they are, by narrative person. Unpaired bootstrap on the group means; the first-person group is small, so read the interval, not the point.
+
+| metric | first-person | third-person | Δ | 95% CI | excludes 0 |
+|---|---|---|---|---|---|
+| mean_latency | 0.289 (n=7) | 0.131 (n=21) | +0.158 | [+0.017, +0.343] | **yes** |
+| median_latency | 0.240 (n=7) | 0.074 (n=21) | +0.166 | [+0.010, +0.373] | **yes** |
+| unresolved_fraction | 0.825 (n=7) | 0.845 (n=21) | -0.020 | [-0.084, +0.048] | no |
+| max_latency | 0.529 (n=7) | 0.398 (n=21) | +0.131 | [-0.092, +0.328] | no |
+
+## One character, traced (dsg-full)
+
+### AHandfulOfDust
+
+**Tony Last** — what the reader believes, and when.
+
+| read at | predicate | value | held until | status | source |
+|---|---|---|---|---|---|
+| 0.04 | other:has_a_room | Morgan le Fay | — | believed | narrated |
+| 0.10 | other:wears_a_dark_suit_on_sundays | narrator | — | believed | narrated |
+| 0.86 | other:saw_brenda | John Beaver of Bratt's Club | — | believed | narrated |
+
+### APassageToIndia
+
+**Dr. Aziz** — what the reader believes, and when.
+
+| read at | predicate | value | held until | status | source |
+|---|---|---|---|---|---|
+| 0.16 | identity_of | the city | — | believed | narrated |
+| 0.43 | member_of | the city | — | believed | narrated |
+| 0.50 | resides_in | the city | — | believed | narrated |
+| 0.62 | married_to | the city | — | believed | narrated |
+| 0.62 | other:is | an innocent man | — | believed | narrated |
+| 0.63 | other:is | a | — | believed | narrated |
+| 0.70 | other:followed | prisoner | — | believed | narrated |
+| 0.80 | other:friend | Hassan | — | believed | narrated |
+| 0.85 | other:addicted | to cameos | — | believed | narrated |
+| 0.92 | other:skilful | in the slighter impertinences | — | believed | narrated |
+| 0.93 | other:lives_in_the_guest_house | residing_in | — | believed | narrated |
+| 0.96 | other:started_again | <<END>> | — | believed | narrated |
+
+
+## Revision events on real text (dsg-full)
+
+| op | book | position | change | evidence |
+|---|---|---|---|---|
+| `merge_provisional` | AHandfulOfDust | 0.96 | absorbed e00089: link 'the half-caste prospector' -> 'John Beaver of Bratt's Club' | …f comprehension, and he went away unsatisfied. At their mid-day meal Mr Todd said, "Mr Last, the Indians tell me that you have been trying to speak wi… |
+| `merge_provisional` | AHandfulOfDust | 0.04 | absorbed e00004: link 'Guinevere' -> 'Morgan le Fay' | …t arcs of flying fish ("Look, Ernie, come quick, there's a shark." "That's not a shark, it's a dolphin." "Mr Brink said it was a porpoise." "There he … |
+| `elaborate` | AHandfulOfDust | 0.95 | provisional -> committed | …lly propped the motor-cycle against the wall of the drawbridge and ran in to breakfast. Life at Hetton was busier but simpler since Richard Last's suc… |
+| `elaborate` | AHandfulOfDust | 0.92 | provisional -> committed | …formed by a rough platform erected in the eaves of the roof. Mr Todd propped a ladder against it and mounted. Tony followed, still unsteady after his … |
+| `elaborate` | AHandfulOfDust | 0.69 | provisional -> committed | …nursing. If only we were at a village...." "I'm afraid I'm being a great nuisance." "That's not the point. The thing is to find what is best for us to… |
+| `elaborate` | AHandfulOfDust | 0.66 | bound 'an old horse' to a horse | …t arcs of flying fish ("Look, Ernie, come quick, there's a shark." "That's not a shark, it's a dolphin." "Mr Brink said it was a porpoise." "There he … |
+| `elaborate` | AHandfulOfDust | 0.61 | provisional -> committed | …, he was sauntering among his guests refilling their glasses. "This stuff's all right," he said, showing the label, "it won't hurt you. It's the right… |
+| `elaborate` | AHandfulOfDust | 0.47 | provisional -> committed | …You come in boats. All these stores stay here. All these stores go in boats. You tell men that." Rosa said nothing. "Understand?" "No peoples go in bo… |
+| `elaborate` | AHandfulOfDust | 0.38 | provisional -> committed | …le down with you again." While Allan was there, Tony resolutely refused to listen, but later the words, and the picture they evoked, would not leave h… |
+| `elaborate` | AHandfulOfDust | 0.34 | provisional -> committed | …ins. So many of our guests nowadays are on a diet. I think muffins one of the few things that make the English winter endurable." "Muffins stand for s… |
+| `elaborate` | AHandfulOfDust | 0.32 | provisional -> committed | …e Empire chair against the wall, perfectly still with her hands folded in her lap, like a small well-brought-up child introduced into a room full of g… |
+| `elaborate` | AHandfulOfDust | 0.26 | provisional -> committed | …know." "How?" "Well, my mother let it to her, you know." Tony was greatly surprised and taxed Brenda with this. "You never told me who was behind your… |
+| `elaborate` | AHandfulOfDust | 0.10 | provisional -> committed | …sunset to dawn with the small wick glowing in the beef fat and the palm thatch overhead dimly discernible; but Tony lived in quiet confidence and expe… |
+| `elaborate` | AHandfulOfDust | 0.08 | provisional -> committed | …ap like Beaver. She'll want to come back in a year, just you see. Allan says the same." "I've told Allan. I don't want her back." "Well, that's vindic… |
+| `elaborate` | AHandfulOfDust | 0.07 | provisional -> committed | …scudding along, bowed to the wind, prayer-books clutched in their gloved hands. Four or five rugged old men were hobbling down to bathe, hissing like … |
+| `elaborate` | AHandfulOfDust | 0.07 | provisional -> committed | …aid, "Ambrose, when I'm alone I think in future I'll have dinner in the library." Afterwards he sat with a book in front of the fire but he was unable… |
+| `elaborate` | AHandfulOfDust | 0.04 | provisional -> committed | …t arcs of flying fish ("Look, Ernie, come quick, there's a shark." "That's not a shark, it's a dolphin." "Mr Brink said it was a porpoise." "There he … |
+| `elaborate` | AHandfulOfDust | 0.01 | provisional -> committed | …clock, though the wind had got up, fluttering the papers on which the order of the service was printed and once threatening to unveil the memorial pre… |
+| `merge_provisional` | APassageToIndia | 0.71 | absorbed e00153: link 'forward, forward' -> 'Mahmoud Ali' | …s prey. "Forward, forward," shrieked Mahmoud Ali, whose every utterance had become a yell. "Down with the Collector, down with the Superintendent of P… |
+| `merge_provisional` | APassageToIndia | 0.63 | absorbed e00127: link 'much more power than' -> 'Aziz' | …e." "It is useless discussing Hindus with me. Living with them teaches me no more. When I think I annoy them, I do not. When I think I don't annoy the… |
+
+## Figures
+
+![main](figures/fig1-main-results.png)
+![ladder](figures/fig1b-ladder.png)
+![prefix](figures/fig2-prefix-curve.png)
+![paired](figures/fig3-paired-deltas.png)
+![revision](figures/fig4-revision-profile.png)
+![violations](figures/fig5-violation-growth.png)