diff --git a/.gitattributes b/.gitattributes index a6344aac8c09253b3b630fb776ae94478aa0275b..5830c23a5e1d3ea0033cc7892a2c2634cff23032 100644 --- a/.gitattributes +++ b/.gitattributes @@ -33,3 +33,13 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text *.zip filter=lfs diff=lfs merge=lfs -text *.zst filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text +report/litbank-qwen7b/figures/fig1-main-results.png filter=lfs diff=lfs merge=lfs -text +report/litbank-qwen7b/figures/fig1b-ladder.png filter=lfs diff=lfs merge=lfs -text +report/litbank-qwen7b/figures/fig3-paired-deltas.png filter=lfs diff=lfs merge=lfs -text +report/mock-smoke/figures/fig1-main-results.png filter=lfs diff=lfs merge=lfs -text +report/pdnc-qwen1.5b/figures/fig1-main-results.png filter=lfs diff=lfs merge=lfs -text +report/pdnc-qwen1.5b/figures/fig1b-ladder.png filter=lfs diff=lfs merge=lfs -text +report/pdnc-qwen1.5b/figures/fig3-paired-deltas.png filter=lfs diff=lfs merge=lfs -text +report/pdnc-qwen3b/figures/fig1-main-results.png filter=lfs diff=lfs merge=lfs -text +report/pdnc-qwen3b/figures/fig1b-ladder.png filter=lfs diff=lfs merge=lfs -text +report/pdnc-qwen3b/figures/fig3-paired-deltas.png filter=lfs diff=lfs merge=lfs -text diff --git a/report/comparison/comparison.md b/report/comparison/comparison.md new file mode 100644 index 0000000000000000000000000000000000000000..c973037bc089745c9e3534317f2b16270c7ed47d --- /dev/null +++ b/report/comparison/comparison.md @@ -0,0 +1,55 @@ +# Cross-run comparison + +Runs included: litbank-qwen7b, pdnc-qwen1.5b, pdnc-qwen3b + +## Extraction yield per run + +A smaller model does not merely extract *worse*, it extracts *less*: a window whose output cannot be parsed contributes nothing to any policy. The scale comparison below is therefore partly confounded by yield, so the parse rate is reported alongside it rather than buried. + +| run | books | windows | parse failures | parse rate | +|---|---|---|---|---| +| litbank-qwen7b | 100 | 881.0 | 1 | 0.999 | +| pdnc-qwen1.5b | 28 | 4,063.0 | 837 | 0.794 | +| pdnc-qwen3b | 28 | 4,063.0 | 18 | 0.996 | + +## DSG (`dsg-full`) versus the standard incremental pipeline (`append-only`) + +| run | books | metric | mean Δ | 95% CI | excludes 0 | wins | +|---|---|---|---|---|---|---| +| litbank-qwen7b | 100 | inconsistent_slot_rate | -0.1814 | [-0.2275, -0.1389] | **yes** | 0–55 | +| litbank-qwen7b | 100 | violations_per_100w | -104.6646 | [-140.1145, -73.0042] | **yes** | 0–55 | +| litbank-qwen7b | 100 | conll_f1 | -0.0027 | [-0.0090, +0.0034] | no | 9–15 | +| litbank-qwen7b | 100 | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | +| litbank-qwen7b | 100 | rollback | +0.0500 | [+0.0100, +0.1000] | **yes** | 5–0 | +| litbank-qwen7b | 100 | monotone_fraction | -0.0112 | [-0.0147, -0.0081] | **yes** | 0–44 | +| pdnc-qwen3b | 28 | inconsistent_slot_rate | -0.4233 | [-0.4694, -0.3744] | **yes** | 0–28 | +| pdnc-qwen3b | 28 | violations_per_100w | -13826.1650 | [-19694.3909, -8538.2675] | **yes** | 0–28 | +| pdnc-qwen3b | 28 | conll_f1 | -0.0050 | [-0.0108, +0.0002] | no | 5–15 | +| pdnc-qwen3b | 28 | mention_acc | -0.0023 | [-0.0168, +0.0115] | no | 10–9 | +| pdnc-qwen3b | 28 | rollback | +32.9286 | [+24.5000, +42.3214] | **yes** | 28–0 | +| pdnc-qwen3b | 28 | monotone_fraction | -0.1866 | [-0.2130, -0.1586] | **yes** | 0–28 | +| pdnc-qwen1.5b | 28 | inconsistent_slot_rate | -0.1067 | [-0.1638, -0.0570] | **yes** | 0–12 | +| pdnc-qwen1.5b | 28 | violations_per_100w | -61.7864 | [-110.2998, -26.2329] | **yes** | 0–13 | +| pdnc-qwen1.5b | 28 | conll_f1 | -0.0026 | [-0.0070, +0.0012] | no | 7–8 | +| pdnc-qwen1.5b | 28 | mention_acc | -0.0171 | [-0.0338, -0.0038] | **yes** | 7–9 | +| pdnc-qwen1.5b | 28 | rollback | +0.0357 | [+0.0000, +0.1071] | no | 1–0 | +| pdnc-qwen1.5b | 28 | monotone_fraction | -0.0050 | [-0.0091, -0.0019] | **yes** | 0–11 | + +## Deferred versus eager commitment (`dsg-full` − `dsg-eager`) + +| run | books | metric | mean Δ | 95% CI | excludes 0 | wins | +|---|---|---|---|---|---|---| +| litbank-qwen7b | 100 | rollback | -0.6000 | [-0.7600, -0.4500] | **yes** | 0–42 | +| litbank-qwen7b | 100 | monotone_fraction | +0.0095 | [+0.0073, +0.0118] | **yes** | 64–0 | +| litbank-qwen7b | 100 | conll_f1 | -0.0059 | [-0.0111, -0.0016] | **yes** | 3–7 | +| pdnc-qwen3b | 28 | rollback | -2.5714 | [-3.7143, -1.7500] | **yes** | 0–25 | +| pdnc-qwen3b | 28 | monotone_fraction | +0.0253 | [+0.0194, +0.0333] | **yes** | 28–0 | +| pdnc-qwen3b | 28 | conll_f1 | -0.0015 | [-0.0063, +0.0025] | no | 5–4 | +| pdnc-qwen1.5b | 28 | rollback | -5.5000 | [-6.7143, -4.3214] | **yes** | 0–28 | +| pdnc-qwen1.5b | 28 | monotone_fraction | +0.0378 | [+0.0307, +0.0463] | **yes** | 28–0 | +| pdnc-qwen1.5b | 28 | conll_f1 | -0.0001 | [-0.0035, +0.0031] | no | 5–2 | + +## Figures + +![scale](figures/fig6-model-scale.png) +![length](figures/fig7-length-dependence.png) diff --git a/report/comparison/figures/fig6-model-scale.pdf b/report/comparison/figures/fig6-model-scale.pdf new file mode 100644 index 0000000000000000000000000000000000000000..ec7ff51db12de4f4017192fea68ef5c04158dcc2 Binary files /dev/null and b/report/comparison/figures/fig6-model-scale.pdf differ diff --git a/report/comparison/figures/fig6-model-scale.png b/report/comparison/figures/fig6-model-scale.png new file mode 100644 index 0000000000000000000000000000000000000000..2a0dfa3da2d04420156170a576ea1442a4957e23 Binary files /dev/null and b/report/comparison/figures/fig6-model-scale.png differ diff --git a/report/litbank-qwen7b/figures/fig1-main-results.pdf b/report/litbank-qwen7b/figures/fig1-main-results.pdf new file mode 100644 index 0000000000000000000000000000000000000000..bc576fe967857b3fa7f3bf99ea04e0ae977800b7 Binary files /dev/null and b/report/litbank-qwen7b/figures/fig1-main-results.pdf differ diff --git a/report/litbank-qwen7b/figures/fig1-main-results.png b/report/litbank-qwen7b/figures/fig1-main-results.png new file mode 100644 index 0000000000000000000000000000000000000000..11d968b54d5e8d874b54b3eff8f382a20fc51e88 --- /dev/null +++ b/report/litbank-qwen7b/figures/fig1-main-results.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1de1844a1e6bb5736ed6a991ced5bad668e61631ab3d6fb219fe7ebf92c8f565 +size 144552 diff --git a/report/litbank-qwen7b/figures/fig1b-ladder.pdf b/report/litbank-qwen7b/figures/fig1b-ladder.pdf new file mode 100644 index 0000000000000000000000000000000000000000..e6e745faa0bba4f3c69490f0bd5d222e05ea1a7d Binary files /dev/null and b/report/litbank-qwen7b/figures/fig1b-ladder.pdf differ diff --git a/report/litbank-qwen7b/figures/fig1b-ladder.png b/report/litbank-qwen7b/figures/fig1b-ladder.png new file mode 100644 index 0000000000000000000000000000000000000000..f64544dcad51b49aff36ba3b22319a1fa1735492 --- /dev/null +++ b/report/litbank-qwen7b/figures/fig1b-ladder.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5be9dad46cdd99fb0330a76d117125200b4d170dd0b9cb8a29175e3df052aed3 +size 115063 diff --git a/report/litbank-qwen7b/figures/fig2-prefix-curve.pdf b/report/litbank-qwen7b/figures/fig2-prefix-curve.pdf new file mode 100644 index 0000000000000000000000000000000000000000..94ae03248a407b3ca6d851bed15b69f3e07472a5 Binary files /dev/null and b/report/litbank-qwen7b/figures/fig2-prefix-curve.pdf differ diff --git a/report/litbank-qwen7b/figures/fig2-prefix-curve.png b/report/litbank-qwen7b/figures/fig2-prefix-curve.png new file mode 100644 index 0000000000000000000000000000000000000000..446185f9186abd93c8b194f083e38813668d6cf0 Binary files /dev/null and b/report/litbank-qwen7b/figures/fig2-prefix-curve.png differ diff --git a/report/litbank-qwen7b/figures/fig3-paired-deltas.pdf b/report/litbank-qwen7b/figures/fig3-paired-deltas.pdf new file mode 100644 index 0000000000000000000000000000000000000000..bb03394c9a0b5fa9d83ff370bfcad2b28aa4ed33 Binary files /dev/null and b/report/litbank-qwen7b/figures/fig3-paired-deltas.pdf differ diff --git a/report/litbank-qwen7b/figures/fig3-paired-deltas.png b/report/litbank-qwen7b/figures/fig3-paired-deltas.png new file mode 100644 index 0000000000000000000000000000000000000000..4409e8b36edbe97919c69297d15c2c98ef03d79d --- /dev/null +++ b/report/litbank-qwen7b/figures/fig3-paired-deltas.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:adb53a2c89da1867c4ec179b2e35e361a7c69031297b6640ffca67869e9d39e2 +size 658040 diff --git a/report/litbank-qwen7b/figures/fig4-revision-profile.pdf b/report/litbank-qwen7b/figures/fig4-revision-profile.pdf new file mode 100644 index 0000000000000000000000000000000000000000..294326c0319db84d4ade76ce6c29c17e56284d7b Binary files /dev/null and b/report/litbank-qwen7b/figures/fig4-revision-profile.pdf differ diff --git a/report/litbank-qwen7b/figures/fig4-revision-profile.png b/report/litbank-qwen7b/figures/fig4-revision-profile.png new file mode 100644 index 0000000000000000000000000000000000000000..daeaaee162f07dc7a41f2b70c8fda11d6b72cc42 Binary files /dev/null and b/report/litbank-qwen7b/figures/fig4-revision-profile.png differ diff --git a/report/litbank-qwen7b/figures/fig5-violation-growth.pdf b/report/litbank-qwen7b/figures/fig5-violation-growth.pdf new file mode 100644 index 0000000000000000000000000000000000000000..75b949dd0bc20314b7707ae14d986d361991db5e Binary files /dev/null and b/report/litbank-qwen7b/figures/fig5-violation-growth.pdf differ diff --git a/report/litbank-qwen7b/figures/fig5-violation-growth.png b/report/litbank-qwen7b/figures/fig5-violation-growth.png new file mode 100644 index 0000000000000000000000000000000000000000..4a560287fb4db5189e01867b0d7882e3ebe01360 Binary files /dev/null and b/report/litbank-qwen7b/figures/fig5-violation-growth.png differ diff --git a/report/litbank-qwen7b/results.md b/report/litbank-qwen7b/results.md new file mode 100644 index 0000000000000000000000000000000000000000..f444e473406d641933d5cac1129532ad3c648216 --- /dev/null +++ b/report/litbank-qwen7b/results.md @@ -0,0 +1,180 @@ +# Dynamic Story Graph — results + +Books: **100** · policies: window-only, append-only, dsg-merge, dsg-eager, dsg-full, retrospective · scoring time: 72.2s + +## Means over books + +| policy | conll_f1 | mention_acc | mention_acc_answered | b3_f1_multi | coverage | fragmentation | conflation | inconsistent_slot_rate | speaker_acc | speaker_acc_matched | violations_per_100w | monotone_fraction | rollback | live_entities | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| window-only | 0.573 | 0.000 | 0.000 | 0.523 | 0.440 | 0.180 | 0.009 | 0.091 | 0.000 | 0.000 | 1092.9 | 1.000 | 0.000 | 47.920 | +| append-only | 0.654 | 0.000 | 0.000 | 0.586 | 0.440 | 0.084 | 0.038 | 0.181 | 0.000 | 0.000 | 104.7 | 1.000 | 0.000 | 28.710 | +| dsg-merge | 0.657 | 0.000 | 0.000 | 0.590 | 0.441 | 0.078 | 0.064 | 0.190 | 0.000 | 0.000 | 109.5 | 0.991 | 0.600 | 28.080 | +| dsg-eager | 0.657 | 0.000 | 0.000 | 0.590 | 0.441 | 0.078 | 0.064 | 0.000 | 0.000 | 0.000 | 1.028 | 0.979 | 0.650 | 28.080 | +| dsg-full | 0.651 | 0.000 | 0.000 | 0.585 | 0.441 | 0.087 | 0.061 | 0.000 | 0.000 | 0.000 | 0.000 | 0.989 | 0.050 | 28.440 | +| retrospective | 0.655 | 0.000 | 0.000 | 0.588 | 0.441 | 0.080 | 0.063 | 0.000 | 0.000 | 0.000 | 0.000 | 0.984 | 0.040 | 28.310 | + +## Paired comparisons (bootstrap over books) + +| comparison | metric | mean Δ | 95% CI | excludes 0 | wins | n | +|---|---|---|---|---|---|---| +| append-only vs window-only | conll_f1 | +0.0807 | [+0.0622, +0.1003] | **yes** | 50–9 | 100 | +| append-only vs window-only | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| append-only vs window-only | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| append-only vs window-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| append-only vs window-only | violations_per_100w | -988.2511 | [-1150.0577, -835.6563] | **yes** | 2–96 | 100 | +| append-only vs window-only | inconsistent_slot_rate | +0.0908 | [+0.0621, +0.1249] | **yes** | 40–1 | 100 | +| append-only vs window-only | rollback | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| append-only vs window-only | monotone_fraction | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| append-only vs window-only | fragmentation | -0.0957 | [-0.1233, -0.0706] | **yes** | 0–49 | 100 | +| append-only vs window-only | conflation | +0.0291 | [+0.0167, +0.0431] | **yes** | 22–0 | 100 | +| dsg-merge vs append-only | conll_f1 | +0.0032 | [-0.0011, +0.0079] | no | 8–9 | 100 | +| dsg-merge vs append-only | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-merge vs append-only | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-merge vs append-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-merge vs append-only | violations_per_100w | +4.7889 | [+1.0000, +9.7725] | **yes** | 7–0 | 100 | +| dsg-merge vs append-only | inconsistent_slot_rate | +0.0086 | [+0.0014, +0.0183] | **yes** | 7–0 | 100 | +| dsg-merge vs append-only | rollback | +0.6000 | [+0.4500, +0.7600] | **yes** | 42–0 | 100 | +| dsg-merge vs append-only | monotone_fraction | -0.0086 | [-0.0109, -0.0064] | **yes** | 0–42 | 100 | +| dsg-merge vs append-only | fragmentation | -0.0067 | [-0.0139, -0.0007] | **yes** | 1–6 | 100 | +| dsg-merge vs append-only | conflation | +0.0262 | [+0.0125, +0.0427] | **yes** | 16–0 | 100 | +| dsg-eager vs dsg-merge | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-eager vs dsg-merge | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-eager vs dsg-merge | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-eager vs dsg-merge | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-eager vs dsg-merge | violations_per_100w | -108.4258 | [-144.6174, -76.5869] | **yes** | 0–57 | 100 | +| dsg-eager vs dsg-merge | inconsistent_slot_rate | -0.1901 | [-0.2359, -0.1473] | **yes** | 0–57 | 100 | +| dsg-eager vs dsg-merge | rollback | +0.0500 | [+0.0100, +0.1000] | **yes** | 5–0 | 100 | +| dsg-eager vs dsg-merge | monotone_fraction | -0.0121 | [-0.0160, -0.0087] | **yes** | 0–44 | 100 | +| dsg-eager vs dsg-merge | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-eager vs dsg-merge | conflation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs dsg-eager | conll_f1 | -0.0059 | [-0.0111, -0.0016] | **yes** | 3–7 | 100 | +| dsg-full vs dsg-eager | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs dsg-eager | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs dsg-eager | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs dsg-eager | violations_per_100w | -1.0278 | [-2.8333, +0.0000] | no | 0–2 | 100 | +| dsg-full vs dsg-eager | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs dsg-eager | rollback | -0.6000 | [-0.7600, -0.4500] | **yes** | 0–42 | 100 | +| dsg-full vs dsg-eager | monotone_fraction | +0.0095 | [+0.0073, +0.0118] | **yes** | 64–0 | 100 | +| dsg-full vs dsg-eager | fragmentation | +0.0094 | [+0.0026, +0.0185] | **yes** | 7–0 | 100 | +| dsg-full vs dsg-eager | conflation | -0.0035 | [-0.0076, -0.0003] | **yes** | 0–6 | 100 | +| retrospective vs dsg-full | conll_f1 | +0.0045 | [+0.0004, +0.0093] | **yes** | 9–1 | 100 | +| retrospective vs dsg-full | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| retrospective vs dsg-full | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| retrospective vs dsg-full | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| retrospective vs dsg-full | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| retrospective vs dsg-full | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| retrospective vs dsg-full | rollback | -0.0100 | [-0.0300, +0.0000] | no | 0–1 | 100 | +| retrospective vs dsg-full | monotone_fraction | -0.0047 | [-0.0066, -0.0030] | **yes** | 4–40 | 100 | +| retrospective vs dsg-full | fragmentation | -0.0066 | [-0.0150, -0.0007] | **yes** | 0–4 | 100 | +| retrospective vs dsg-full | conflation | +0.0025 | [-0.0068, +0.0131] | no | 3–3 | 100 | +| dsg-full vs append-only | conll_f1 | -0.0027 | [-0.0090, +0.0034] | no | 9–15 | 100 | +| dsg-full vs append-only | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs append-only | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs append-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs append-only | violations_per_100w | -104.6646 | [-140.1145, -73.0042] | **yes** | 0–55 | 100 | +| dsg-full vs append-only | inconsistent_slot_rate | -0.1814 | [-0.2275, -0.1389] | **yes** | 0–55 | 100 | +| dsg-full vs append-only | rollback | +0.0500 | [+0.0100, +0.1000] | **yes** | 5–0 | 100 | +| dsg-full vs append-only | monotone_fraction | -0.0112 | [-0.0147, -0.0081] | **yes** | 0–44 | 100 | +| dsg-full vs append-only | fragmentation | +0.0026 | [-0.0060, +0.0112] | no | 8–5 | 100 | +| dsg-full vs append-only | conflation | +0.0228 | [+0.0088, +0.0395] | **yes** | 16–4 | 100 | +| dsg-full vs window-only | conll_f1 | +0.0780 | [+0.0595, +0.0974] | **yes** | 54–14 | 100 | +| dsg-full vs window-only | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs window-only | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs window-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs window-only | violations_per_100w | -1092.9157 | [-1258.5972, -934.3395] | **yes** | 0–100 | 100 | +| dsg-full vs window-only | inconsistent_slot_rate | -0.0906 | [-0.1247, -0.0608] | **yes** | 0–36 | 100 | +| dsg-full vs window-only | rollback | +0.0500 | [+0.0100, +0.1000] | **yes** | 5–0 | 100 | +| dsg-full vs window-only | monotone_fraction | -0.0112 | [-0.0147, -0.0081] | **yes** | 0–44 | 100 | +| dsg-full vs window-only | fragmentation | -0.0931 | [-0.1199, -0.0686] | **yes** | 2–51 | 100 | +| dsg-full vs window-only | conflation | +0.0518 | [+0.0333, +0.0726] | **yes** | 32–2 | 100 | +| dsg-full vs dsg-merge | conll_f1 | -0.0059 | [-0.0111, -0.0016] | **yes** | 3–7 | 100 | +| dsg-full vs dsg-merge | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs dsg-merge | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs dsg-merge | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs dsg-merge | violations_per_100w | -109.4535 | [-145.7021, -77.5386] | **yes** | 0–57 | 100 | +| dsg-full vs dsg-merge | inconsistent_slot_rate | -0.1901 | [-0.2359, -0.1473] | **yes** | 0–57 | 100 | +| dsg-full vs dsg-merge | rollback | -0.5500 | [-0.7100, -0.3900] | **yes** | 2–40 | 100 | +| dsg-full vs dsg-merge | monotone_fraction | -0.0026 | [-0.0068, +0.0013] | no | 31–34 | 100 | +| dsg-full vs dsg-merge | fragmentation | +0.0094 | [+0.0026, +0.0185] | **yes** | 7–0 | 100 | +| dsg-full vs dsg-merge | conflation | -0.0035 | [-0.0076, -0.0003] | **yes** | 0–6 | 100 | +| dsg-full vs retrospective | conll_f1 | -0.0045 | [-0.0093, -0.0004] | **yes** | 1–9 | 100 | +| dsg-full vs retrospective | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs retrospective | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs retrospective | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs retrospective | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs retrospective | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-full vs retrospective | rollback | +0.0100 | [+0.0000, +0.0300] | no | 1–0 | 100 | +| dsg-full vs retrospective | monotone_fraction | +0.0047 | [+0.0030, +0.0066] | **yes** | 40–4 | 100 | +| dsg-full vs retrospective | fragmentation | +0.0066 | [+0.0007, +0.0150] | **yes** | 4–0 | 100 | +| dsg-full vs retrospective | conflation | -0.0025 | [-0.0131, +0.0068] | no | 3–3 | 100 | +| dsg-eager vs append-only | conll_f1 | +0.0032 | [-0.0011, +0.0079] | no | 8–9 | 100 | +| dsg-eager vs append-only | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-eager vs append-only | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-eager vs append-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 100 | +| dsg-eager vs append-only | violations_per_100w | -103.6369 | [-139.2292, -71.9828] | **yes** | 0–55 | 100 | +| dsg-eager vs append-only | inconsistent_slot_rate | -0.1814 | [-0.2275, -0.1389] | **yes** | 0–55 | 100 | +| dsg-eager vs append-only | rollback | +0.6500 | [+0.4900, +0.8200] | **yes** | 44–0 | 100 | +| dsg-eager vs append-only | monotone_fraction | -0.0207 | [-0.0251, -0.0167] | **yes** | 0–65 | 100 | +| dsg-eager vs append-only | fragmentation | -0.0067 | [-0.0139, -0.0007] | **yes** | 1–6 | 100 | +| dsg-eager vs append-only | conflation | +0.0262 | [+0.0125, +0.0427] | **yes** | 16–0 | 100 | + +## Identity-resolution latency (measurement instrument) + +_not computed_ + +## One character, traced (dsg-full) + +### 1023_bleak_house_brat + +_no state recorded for Chancery_ + +### 105_persuasion_brat + +**ELLIOT** — what the reader believes, and when. + +| read at | predicate | value | held until | status | source | +|---|---|---|---|---|---| +| 0.22 | other:found_enough_in_her_duties_friends_and_children | to attach her to life | — | believed | reported | +| 0.22 | other:had_judgements_and_conduct | never required indulgence | — | believed | reported | +| 0.22 | other:humoured_softened_or_concealed_sir_walter_s_failings | for seventeen years | — | believed | reported | +| 0.22 | other:was_an_excellent_woman | sensible and amiable | — | believed | reported | +| 0.22 | other:was_not_the_very_happiest_being_in_the_world | herself | — | believed | reported | +| 0.33 | other:deceased | 13 years ago | — | believed | narrated | +| 0.33 | other:relies_on | Lady Russell | — | believed | narrated | +| 0.33 | possesses | daughters | — | believed | narrated | +| 0.33 | possesses | two eldest daughters | — | believed | narrated | + + +## Revision events on real text (dsg-full) + +| op | book | position | change | evidence | +|---|---|---|---|---| +| `supersede` | 1023_bleak_house_brat | 0.30 | High Court of Chancery.location: 'Lincoln's Inn Hall' → 'Holborn Hill' | …technicalities, running their goat-hair and horsehair warded heads against walls of words and making a pretence of equity with serious faces, as playe… | +| `supersede` | 1023_bleak_house_brat | 0.20 | Fog.location: 'down the river' → 'Temple Bar' | …the muddy streets are muddiest near that leaden-headed old obstruction, appropriate ornament for the threshold of a leaden-headed old corporation, Tem… | +| `supersede` | 1023_bleak_house_brat | 0.00 | Fog.location: 'everywhere' → 'up the river' | …CHAPTER I In Chancery London. Michaelmas term lately over, and the Lord Chancellor sitting in Lincoln's Inn Hall. Implacable November weather. As much… | +| `supersede` | 1023_bleak_house_brat | 0.00 | Fog.location: 'up the river' → 'down the river' | …CHAPTER I In Chancery London. Michaelmas term lately over, and the Lord Chancellor sitting in Lincoln's Inn Hall. Implacable November weather. As much… | +| `elaborate` | 1023_bleak_house_brat | 0.20 | provisional -> committed | …e deliriously found themselves made parties in Jarndyce and Jarndyce without knowing how or why; whole families have inherited legendary hatreds with … | +| `elaborate` | 1023_bleak_house_brat | 0.20 | provisional -> committed | …e deliriously found themselves made parties in Jarndyce and Jarndyce without knowing how or why; whole families have inherited legendary hatreds with … | +| `elaborate` | 1023_bleak_house_brat | 0.00 | provisional -> committed | …the muddy streets are muddiest near that leaden-headed old obstruction, appropriate ornament for the threshold of a leaden-headed old corporation, Tem… | +| `elaborate` | 105_persuasion_brat | 0.89 | provisional -> committed | …rawing-back of youth; and, in one of their spring excursions to London, when Elizabeth was in her first bloom, Mr Elliot had been forced into the intr… | +| `elaborate` | 105_persuasion_brat | 0.44 | provisional -> committed | …y different were her delicate features and mild dark eyes from his own), there could be nothing in them, now that she was faded and thin, to excite hi… | +| `elaborate` | 105_persuasion_brat | 0.33 | provisional -> committed | …y different were her delicate features and mild dark eyes from his own), there could be nothing in them, now that she was faded and thin, to excite hi… | +| `elaborate` | 105_persuasion_brat | 0.22 | provisional -> committed | …teen and fourteen, was an awful legacy for a mother to bequeath, an awful charge rather, to confide to the authority and guidance of a conceited, sill… | +| `elaborate` | 105_persuasion_brat | 0.11 | provisional -> committed | …rawing-back of youth; and, in one of their spring excursions to London, when Elizabeth was in her first bloom, Mr Elliot had been forced into the intr… | +| `elaborate` | 105_persuasion_brat | 0.11 | provisional -> committed | …y different were her delicate features and mild dark eyes from his own), there could be nothing in them, now that she was faded and thin, to excite hi… | +| `elaborate` | 105_persuasion_brat | 0.11 | provisional -> committed | …teen and fourteen, was an awful legacy for a mother to bequeath, an awful charge rather, to confide to the authority and guidance of a conceited, sill… | +| `elaborate` | 105_persuasion_brat | 0.11 | provisional -> committed | …ather, (having met with one or two private disappointments in very unreasonable applications), prided himself on remaining single for his dear daughte… | +| `elaborate` | 105_persuasion_brat | 0.00 | provisional -> committed | …rawing-back of youth; and, in one of their spring excursions to London, when Elizabeth was in her first bloom, Mr Elliot had been forced into the intr… | +| `supersede` | 1064_the_masque_of_the_red_death_brat | 0.78 | the company.emotion: 'a sensation' → 'disapprobation' | …ll of the clock; and thus it happened, perhaps, that more of thought crept, with more of time, into the meditations of the thoughtful among those who … | +| `supersede` | 1064_the_masque_of_the_red_death_brat | 0.78 | the company.emotion: 'disapprobation' → 'surprise' | …ll of the clock; and thus it happened, perhaps, that more of thought crept, with more of time, into the meditations of the thoughtful among those who … | +| `elaborate` | 1064_the_masque_of_the_red_death_brat | 0.00 | provisional -> committed | …The" Red Death" had long devastated the country. No pestilence had ever been so fatal, or so hideous. Blood was its Avatar and its seal -- the redness… | +| `merge_provisional` | 110_tess_of_the_durbervilles_a_pure_woman_brat | 0.12 | absorbed e00009: link 'the parson' -> 'Parson Tringham' | …f beer wi' me on the strength o' t, Pa' son Tringham? There's a very pretty brew in tap at The Pure Drop -- though, to be sure, not so good as at Roll… | + +## Figures + +![main](figures/fig1-main-results.png) +![ladder](figures/fig1b-ladder.png) +![prefix](figures/fig2-prefix-curve.png) +![paired](figures/fig3-paired-deltas.png) +![revision](figures/fig4-revision-profile.png) +![violations](figures/fig5-violation-growth.png) diff --git a/report/mock-smoke/figures/fig1-main-results.pdf b/report/mock-smoke/figures/fig1-main-results.pdf new file mode 100644 index 0000000000000000000000000000000000000000..3e35d5dbc76769722a9e4c25d48eed7f403016c8 Binary files /dev/null and b/report/mock-smoke/figures/fig1-main-results.pdf differ diff --git a/report/mock-smoke/figures/fig1-main-results.png b/report/mock-smoke/figures/fig1-main-results.png new file mode 100644 index 0000000000000000000000000000000000000000..4a10a7e737c0f74fdc55d615e047f08e560eebad --- /dev/null +++ b/report/mock-smoke/figures/fig1-main-results.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e1de283f1b41dc09deae73aa4661ee566229afc9ba79dc9d85e57271f7c01a55 +size 146174 diff --git a/report/mock-smoke/figures/fig2-prefix-curve.pdf b/report/mock-smoke/figures/fig2-prefix-curve.pdf new file mode 100644 index 0000000000000000000000000000000000000000..9839f5419ce6f6359b6d4e2c5e5fc21f007f0b1b Binary files /dev/null and b/report/mock-smoke/figures/fig2-prefix-curve.pdf differ diff --git a/report/mock-smoke/figures/fig2-prefix-curve.png b/report/mock-smoke/figures/fig2-prefix-curve.png new file mode 100644 index 0000000000000000000000000000000000000000..51fe958032c04a80a9d9fb30fc99ab54b9dcd693 Binary files /dev/null and b/report/mock-smoke/figures/fig2-prefix-curve.png differ diff --git a/report/mock-smoke/figures/fig3-paired-deltas.pdf b/report/mock-smoke/figures/fig3-paired-deltas.pdf new file mode 100644 index 0000000000000000000000000000000000000000..b2eb4bdeb3295ada5f4b0c02da3312e13f9bdc4c Binary files /dev/null and b/report/mock-smoke/figures/fig3-paired-deltas.pdf differ diff --git a/report/mock-smoke/figures/fig3-paired-deltas.png b/report/mock-smoke/figures/fig3-paired-deltas.png new file mode 100644 index 0000000000000000000000000000000000000000..3c1829c1d303e9f8d4c9da7581901c90d04ecba8 Binary files /dev/null and b/report/mock-smoke/figures/fig3-paired-deltas.png differ diff --git a/report/mock-smoke/figures/fig4-revision-profile.pdf b/report/mock-smoke/figures/fig4-revision-profile.pdf new file mode 100644 index 0000000000000000000000000000000000000000..f9b78e3c26209ee06d61f63d5708ea4509c39deb Binary files /dev/null and b/report/mock-smoke/figures/fig4-revision-profile.pdf differ diff --git a/report/mock-smoke/figures/fig4-revision-profile.png b/report/mock-smoke/figures/fig4-revision-profile.png new file mode 100644 index 0000000000000000000000000000000000000000..2937ccbbd7e1fec4c7d8030fa37b30d6d3d6eab4 Binary files /dev/null and b/report/mock-smoke/figures/fig4-revision-profile.png differ diff --git a/report/mock-smoke/figures/fig5-violation-growth.pdf b/report/mock-smoke/figures/fig5-violation-growth.pdf new file mode 100644 index 0000000000000000000000000000000000000000..78b933ff1eb3afd5d339590fb1a4c803e289ef30 Binary files /dev/null and b/report/mock-smoke/figures/fig5-violation-growth.pdf differ diff --git a/report/mock-smoke/figures/fig5-violation-growth.png b/report/mock-smoke/figures/fig5-violation-growth.png new file mode 100644 index 0000000000000000000000000000000000000000..83c5fe865ec73ce0c09de9ee97dd2f64b8c1e931 Binary files /dev/null and b/report/mock-smoke/figures/fig5-violation-growth.png differ diff --git a/report/mock-smoke/results.md b/report/mock-smoke/results.md new file mode 100644 index 0000000000000000000000000000000000000000..07f1fc77c57353a925330cf34b01bd3a274341fa --- /dev/null +++ b/report/mock-smoke/results.md @@ -0,0 +1,94 @@ +# Dynamic Story Graph — results + +Books: **3** · policies: window-only, append-only, dsg-merge, dsg-eager, dsg-full, retrospective · scoring time: 32.3s + +## Means over books + +| policy | conll_f1 | b3_f1_multi | b3_f1 | coverage | fragmentation | conflation | speaker_acc | speaker_acc_matched | violations_per_100w | monotone_fraction | rollback | live_entities | +|---|---|---|---|---|---|---|---|---|---|---|---|---| +| window-only | 0.462 | 0.543 | 0.751 | 0.404 | 0.167 | 0.000 | 0.031 | 0.223 | 29847.6 | 1.000 | 0.000 | 233.0 | +| append-only | 0.517 | 0.583 | 0.773 | 0.404 | 0.042 | 0.042 | 0.032 | 0.234 | 591.5 | 1.000 | 0.000 | 107.7 | +| dsg-merge | 0.517 | 0.583 | 0.773 | 0.404 | 0.042 | 0.042 | 0.032 | 0.234 | 591.5 | 1.000 | 0.000 | 107.7 | +| dsg-eager | 0.517 | 0.583 | 0.773 | 0.404 | 0.042 | 0.042 | 0.032 | 0.234 | 0.000 | 0.956 | 0.000 | 107.7 | +| dsg-full | 0.517 | 0.583 | 0.773 | 0.404 | 0.042 | 0.042 | 0.032 | 0.234 | 0.000 | 0.965 | 0.000 | 107.7 | +| retrospective | 0.517 | 0.583 | 0.773 | 0.404 | 0.042 | 0.042 | 0.032 | 0.234 | 0.000 | 0.944 | 0.000 | 107.7 | + +## Paired comparisons (bootstrap over books) + +| comparison | metric | mean Δ | 95% CI | excludes 0 | wins | n | +|---|---|---|---|---|---|---| +| dsg-full vs append-only | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs append-only | b3_f1_multi | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs append-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs append-only | violations_per_100w | -591.4530 | [-1738.4615, +0.0000] | no | 0–2 | 3 | +| dsg-full vs append-only | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs append-only | monotone_fraction | -0.0355 | [-0.1000, +0.0000] | no | 0–2 | 3 | +| dsg-full vs window-only | conll_f1 | +0.0546 | [+0.0000, +0.1639] | no | 1–0 | 3 | +| dsg-full vs window-only | b3_f1_multi | +0.0397 | [+0.0000, +0.1191] | no | 1–0 | 3 | +| dsg-full vs window-only | speaker_acc_matched | +0.0109 | [+0.0000, +0.0326] | no | 1–0 | 3 | +| dsg-full vs window-only | violations_per_100w | -29847.6496 | [-35979.4872, -23638.4615] | **yes** | 0–3 | 3 | +| dsg-full vs window-only | fragmentation | -0.1250 | [-0.3750, +0.0000] | no | 0–1 | 3 | +| dsg-full vs window-only | monotone_fraction | -0.0355 | [-0.1000, +0.0000] | no | 0–2 | 3 | +| dsg-full vs dsg-merge | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs dsg-merge | b3_f1_multi | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs dsg-merge | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs dsg-merge | violations_per_100w | -591.4530 | [-1738.4615, +0.0000] | no | 0–2 | 3 | +| dsg-full vs dsg-merge | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs dsg-merge | monotone_fraction | -0.0355 | [-0.1000, +0.0000] | no | 0–2 | 3 | +| dsg-full vs dsg-eager | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs dsg-eager | b3_f1_multi | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs dsg-eager | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs dsg-eager | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs dsg-eager | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs dsg-eager | monotone_fraction | +0.0085 | [+0.0000, +0.0240] | no | 2–0 | 3 | +| dsg-full vs retrospective | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs retrospective | b3_f1_multi | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs retrospective | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs retrospective | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs retrospective | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-full vs retrospective | monotone_fraction | +0.0205 | [+0.0000, +0.0584] | no | 2–0 | 3 | +| dsg-eager vs append-only | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-eager vs append-only | b3_f1_multi | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-eager vs append-only | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-eager vs append-only | violations_per_100w | -591.4530 | [-1738.4615, +0.0000] | no | 0–2 | 3 | +| dsg-eager vs append-only | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | +| dsg-eager vs append-only | monotone_fraction | -0.0439 | [-0.1240, +0.0000] | no | 0–2 | 3 | +| append-only vs window-only | conll_f1 | +0.0546 | [+0.0000, +0.1639] | no | 1–0 | 3 | +| append-only vs window-only | b3_f1_multi | +0.0397 | [+0.0000, +0.1191] | no | 1–0 | 3 | +| append-only vs window-only | speaker_acc_matched | +0.0109 | [+0.0000, +0.0326] | no | 1–0 | 3 | +| append-only vs window-only | violations_per_100w | -29256.1966 | [-34241.0256, -23602.5641] | **yes** | 0–3 | 3 | +| append-only vs window-only | fragmentation | -0.1250 | [-0.3750, +0.0000] | no | 0–1 | 3 | +| append-only vs window-only | monotone_fraction | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 3 | + +## Revision events on real text (dsg-full) + +| op | book | position | change | evidence | +|---|---|---|---|---| +| `elaborate` | AlicesAdventuresInWonderland | 0.97 | provisional -> committed | …ulders, that all the jurors were writing down "stupid things!" on their slates, and she could even make out that one of them didn't know how to spell … | +| `elaborate` | AlicesAdventuresInWonderland | 0.97 | provisional -> committed | …ulders, that all the jurors were writing down "stupid things!" on their slates, and she could even make out that one of them didn't know how to spell … | +| `elaborate` | AlicesAdventuresInWonderland | 0.97 | provisional -> committed | …ulders, that all the jurors were writing down "stupid things!" on their slates, and she could even make out that one of them didn't know how to spell … | +| `elaborate` | AlicesAdventuresInWonderland | 0.90 | provisional -> committed | …have their tails in their mouths--and they're all over crumbs." "You're wrong about the crumbs," said the Mock Turtle: "crumbs would all wash off in t… | +| `elaborate` | AlicesAdventuresInWonderland | 0.88 | provisional -> committed | …should like to hear her try and repeat something now. Tell her to begin." He looked at the Gryphon as if he thought it had some kind of authority over… | +| `elaborate` | AlicesAdventuresInWonderland | 0.85 | provisional -> committed | …have their tails in their mouths--and they're all over crumbs." "You're wrong about the crumbs," said the Mock Turtle: "crumbs would all wash off in t… | +| `elaborate` | AlicesAdventuresInWonderland | 0.85 | provisional -> committed | …have their tails in their mouths--and they're all over crumbs." "You're wrong about the crumbs," said the Mock Turtle: "crumbs would all wash off in t… | +| `elaborate` | AlicesAdventuresInWonderland | 0.82 | provisional -> committed | …un?" said Alice. "Why, _she_," said the Gryphon. "It's all her fancy, that: they never executes nobody, you know. Come on!" "Everybody says 'come on!'… | +| `elaborate` | AlicesAdventuresInWonderland | 0.80 | provisional -> committed | …A cheap sort of present!" thought Alice. "I'm glad they don't give birthday presents like that!" But she did not venture to say it out loud. "Thinking… | +| `elaborate` | AlicesAdventuresInWonderland | 0.78 | provisional -> committed | …dn't be so stingy about it, you know--" She had quite forgotten the Duchess by this time, and was a little startled when she heard her voice close to … | +| `elaborate` | AlicesAdventuresInWonderland | 0.72 | provisional -> committed | …t like the look of things at all, as the game was in such confusion that she never knew whether it was her turn or not. So she went in search of her h… | +| `elaborate` | AlicesAdventuresInWonderland | 0.70 | provisional -> committed | …A cheap sort of present!" thought Alice. "I'm glad they don't give birthday presents like that!" But she did not venture to say it out loud. "Thinking… | +| `elaborate` | AlicesAdventuresInWonderland | 0.70 | provisional -> committed | …with their heads!" and the procession moved on, three of the soldiers remaining behind to execute the unfortunate gardeners, who ran to Alice for prot… | +| `elaborate` | AlicesAdventuresInWonderland | 0.65 | provisional -> committed | …a thing as a drawing of a muchness?" "Really, now you ask me," said Alice, very much confused, "I don't think--" "Then you shouldn't talk," said the H… | +| `elaborate` | AlicesAdventuresInWonderland | 0.65 | provisional -> committed | …A cheap sort of present!" thought Alice. "I'm glad they don't give birthday presents like that!" But she did not venture to say it out loud. "Thinking… | +| `elaborate` | AlicesAdventuresInWonderland | 0.65 | provisional -> committed | …A cheap sort of present!" thought Alice. "I'm glad they don't give birthday presents like that!" But she did not venture to say it out loud. "Thinking… | +| `elaborate` | AlicesAdventuresInWonderland | 0.60 | provisional -> committed | …g a minute or two. "They couldn't have done that, you know," Alice gently remarked; "they'd have been ill." "So they were," said the Dormouse; "_very_… | +| `elaborate` | AlicesAdventuresInWonderland | 0.55 | provisional -> committed | …t like the look of things at all, as the game was in such confusion that she never knew whether it was her turn or not. So she went in search of her h… | +| `elaborate` | AlicesAdventuresInWonderland | 0.55 | provisional -> committed | …g a minute or two. "They couldn't have done that, you know," Alice gently remarked; "they'd have been ill." "So they were," said the Dormouse; "_very_… | +| `elaborate` | AlicesAdventuresInWonderland | 0.55 | provisional -> committed | …g a minute or two. "They couldn't have done that, you know," Alice gently remarked; "they'd have been ill." "So they were," said the Dormouse; "_very_… | + +## Figures + +![main](figures/fig1-main-results.png) +![prefix](figures/fig2-prefix-curve.png) +![paired](figures/fig3-paired-deltas.png) +![revision](figures/fig4-revision-profile.png) +![violations](figures/fig5-violation-growth.png) diff --git a/report/pdnc-qwen1.5b/figures/fig1-main-results.pdf b/report/pdnc-qwen1.5b/figures/fig1-main-results.pdf new file mode 100644 index 0000000000000000000000000000000000000000..028b4f0668bd4bbe418f5b0121aab3de0e4ba432 Binary files /dev/null and b/report/pdnc-qwen1.5b/figures/fig1-main-results.pdf differ diff --git a/report/pdnc-qwen1.5b/figures/fig1-main-results.png b/report/pdnc-qwen1.5b/figures/fig1-main-results.png new file mode 100644 index 0000000000000000000000000000000000000000..94d538a393f1ed1c9a5711f12c4fb4d20351ccfd --- /dev/null +++ b/report/pdnc-qwen1.5b/figures/fig1-main-results.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9fd4af9cf473fbe29d8036008e41140bfbac7584c3b632816f8afcdf2c4bc69f +size 151295 diff --git a/report/pdnc-qwen1.5b/figures/fig1b-ladder.pdf b/report/pdnc-qwen1.5b/figures/fig1b-ladder.pdf new file mode 100644 index 0000000000000000000000000000000000000000..f2b10ba7efaba3d52d711b63b816071882ba5c97 Binary files /dev/null and b/report/pdnc-qwen1.5b/figures/fig1b-ladder.pdf differ diff --git a/report/pdnc-qwen1.5b/figures/fig1b-ladder.png b/report/pdnc-qwen1.5b/figures/fig1b-ladder.png new file mode 100644 index 0000000000000000000000000000000000000000..11b31495b6f6805c615bd590a68e9bf0fd984996 --- /dev/null +++ b/report/pdnc-qwen1.5b/figures/fig1b-ladder.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:573fa3d7a4cfa3652bb8a80a48aca84ee4790c848233dff877147933fad7a847 +size 100026 diff --git a/report/pdnc-qwen1.5b/figures/fig2-prefix-curve.pdf b/report/pdnc-qwen1.5b/figures/fig2-prefix-curve.pdf new file mode 100644 index 0000000000000000000000000000000000000000..bb309db14d4d54e35031eaa40faacee665aefed2 Binary files /dev/null and b/report/pdnc-qwen1.5b/figures/fig2-prefix-curve.pdf differ diff --git a/report/pdnc-qwen1.5b/figures/fig2-prefix-curve.png b/report/pdnc-qwen1.5b/figures/fig2-prefix-curve.png new file mode 100644 index 0000000000000000000000000000000000000000..3d6b207fd929c24192ffb1547fa3d5d68ef5ab27 Binary files /dev/null and b/report/pdnc-qwen1.5b/figures/fig2-prefix-curve.png differ diff --git a/report/pdnc-qwen1.5b/figures/fig3-paired-deltas.pdf b/report/pdnc-qwen1.5b/figures/fig3-paired-deltas.pdf new file mode 100644 index 0000000000000000000000000000000000000000..0f942500b5361f7eea7297ccf338762c4f6c4033 Binary files /dev/null and b/report/pdnc-qwen1.5b/figures/fig3-paired-deltas.pdf differ diff --git a/report/pdnc-qwen1.5b/figures/fig3-paired-deltas.png b/report/pdnc-qwen1.5b/figures/fig3-paired-deltas.png new file mode 100644 index 0000000000000000000000000000000000000000..df2e4d2d6b4a3d1c668f2269f87a107283397591 --- /dev/null +++ b/report/pdnc-qwen1.5b/figures/fig3-paired-deltas.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ca0e5e10e4d54e375a3dd6891751b531a115f64047d9b9f7a5b3fce7c0bfddb0 +size 147141 diff --git a/report/pdnc-qwen1.5b/figures/fig4-revision-profile.pdf b/report/pdnc-qwen1.5b/figures/fig4-revision-profile.pdf new file mode 100644 index 0000000000000000000000000000000000000000..4c449c301c586d50dd1b5e29eb5b5084fe2c48ba Binary files /dev/null and b/report/pdnc-qwen1.5b/figures/fig4-revision-profile.pdf differ diff --git a/report/pdnc-qwen1.5b/figures/fig4-revision-profile.png b/report/pdnc-qwen1.5b/figures/fig4-revision-profile.png new file mode 100644 index 0000000000000000000000000000000000000000..cc9c8c0e8d36db8d2e7cdfe316af96684968aa64 Binary files /dev/null and b/report/pdnc-qwen1.5b/figures/fig4-revision-profile.png differ diff --git a/report/pdnc-qwen1.5b/figures/fig5-violation-growth.pdf b/report/pdnc-qwen1.5b/figures/fig5-violation-growth.pdf new file mode 100644 index 0000000000000000000000000000000000000000..95b1764175646e9bb139cdc984d24b4a4ac238e9 Binary files /dev/null and b/report/pdnc-qwen1.5b/figures/fig5-violation-growth.pdf differ diff --git a/report/pdnc-qwen1.5b/figures/fig5-violation-growth.png b/report/pdnc-qwen1.5b/figures/fig5-violation-growth.png new file mode 100644 index 0000000000000000000000000000000000000000..4f0c7b1a9ef0afd33c81d4e20ceb9850b0705cac Binary files /dev/null and b/report/pdnc-qwen1.5b/figures/fig5-violation-growth.png differ diff --git a/report/pdnc-qwen1.5b/results.md b/report/pdnc-qwen1.5b/results.md new file mode 100644 index 0000000000000000000000000000000000000000..5a1c1a57836dac32bd27dcbaca109f28ee1d627f --- /dev/null +++ b/report/pdnc-qwen1.5b/results.md @@ -0,0 +1,196 @@ +# Dynamic Story Graph — results + +Books: **28** · policies: window-only, append-only, dsg-merge, dsg-eager, dsg-full, retrospective · scoring time: 1044.2s + +## Means over books + +| policy | conll_f1 | mention_acc | mention_acc_answered | b3_f1_multi | coverage | fragmentation | conflation | inconsistent_slot_rate | speaker_acc | speaker_acc_matched | violations_per_100w | monotone_fraction | rollback | live_entities | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| window-only | 0.451 | 0.069 | 0.943 | 0.532 | 0.313 | 0.334 | 0.004 | 0.004 | 0.012 | 0.509 | 75607.3 | 1.000 | 0.000 | 322.1 | +| append-only | 0.514 | 0.383 | 0.898 | 0.572 | 0.304 | 0.134 | 0.111 | 0.107 | 0.013 | 0.559 | 61.786 | 1.000 | 0.000 | 60.893 | +| dsg-merge | 0.511 | 0.360 | 0.834 | 0.571 | 0.304 | 0.131 | 0.193 | 0.100 | 0.012 | 0.514 | 40.742 | 0.963 | 5.536 | 55.179 | +| dsg-eager | 0.511 | 0.360 | 0.834 | 0.571 | 0.304 | 0.131 | 0.193 | 0.000 | 0.012 | 0.514 | 3.861 | 0.957 | 5.536 | 55.179 | +| dsg-full | 0.511 | 0.366 | 0.825 | 0.570 | 0.304 | 0.135 | 0.173 | 0.000 | 0.013 | 0.525 | 0.000 | 0.995 | 0.036 | 57.536 | +| retrospective | 0.507 | 0.416 | 0.799 | 0.566 | 0.299 | 0.132 | 0.200 | 0.000 | 0.013 | 0.530 | 0.000 | 0.991 | 0.036 | 57.714 | + +## Paired comparisons (bootstrap over books) + +| comparison | metric | mean Δ | 95% CI | excludes 0 | wins | n | +|---|---|---|---|---|---|---| +| append-only vs window-only | conll_f1 | +0.0628 | [+0.0453, +0.0815] | **yes** | 23–0 | 28 | +| append-only vs window-only | mention_acc | +0.3143 | [+0.2662, +0.3635] | **yes** | 28–0 | 28 | +| append-only vs window-only | mention_acc_answered | -0.0453 | [-0.0948, -0.0041] | **yes** | 7–18 | 28 | +| append-only vs window-only | speaker_acc_matched | +0.0503 | [+0.0086, +0.0937] | **yes** | 16–7 | 28 | +| append-only vs window-only | violations_per_100w | -75545.5557 | [-140608.4932, -28002.6100] | **yes** | 0–28 | 28 | +| append-only vs window-only | inconsistent_slot_rate | +0.1027 | [+0.0530, +0.1595] | **yes** | 12–0 | 28 | +| append-only vs window-only | rollback | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| append-only vs window-only | monotone_fraction | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| append-only vs window-only | fragmentation | -0.2002 | [-0.2526, -0.1499] | **yes** | 0–25 | 28 | +| append-only vs window-only | conflation | +0.1077 | [+0.0601, +0.1682] | **yes** | 18–0 | 28 | +| dsg-merge vs append-only | conll_f1 | -0.0025 | [-0.0061, +0.0005] | no | 2–9 | 28 | +| dsg-merge vs append-only | mention_acc | -0.0232 | [-0.0404, -0.0089] | **yes** | 4–10 | 28 | +| dsg-merge vs append-only | mention_acc_answered | -0.0638 | [-0.1137, -0.0249] | **yes** | 2–16 | 28 | +| dsg-merge vs append-only | speaker_acc_matched | -0.0445 | [-0.0811, -0.0158] | **yes** | 1–9 | 28 | +| dsg-merge vs append-only | violations_per_100w | -21.0449 | [-63.7384, +3.0417] | no | 1–2 | 28 | +| dsg-merge vs append-only | inconsistent_slot_rate | -0.0070 | [-0.0225, +0.0046] | no | 1–2 | 28 | +| dsg-merge vs append-only | rollback | +5.5357 | [+4.3571, +6.7500] | **yes** | 28–0 | 28 | +| dsg-merge vs append-only | monotone_fraction | -0.0372 | [-0.0457, -0.0299] | **yes** | 0–28 | 28 | +| dsg-merge vs append-only | fragmentation | -0.0032 | [-0.0097, +0.0000] | no | 0–1 | 28 | +| dsg-merge vs append-only | conflation | +0.0816 | [+0.0311, +0.1533] | **yes** | 11–0 | 28 | +| dsg-eager vs dsg-merge | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-eager vs dsg-merge | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-eager vs dsg-merge | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-eager vs dsg-merge | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-eager vs dsg-merge | violations_per_100w | -36.8808 | [-60.9653, -16.2603] | **yes** | 0–11 | 28 | +| dsg-eager vs dsg-merge | inconsistent_slot_rate | -0.0997 | [-0.1563, -0.0501] | **yes** | 0–11 | 28 | +| dsg-eager vs dsg-merge | rollback | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-eager vs dsg-merge | monotone_fraction | -0.0057 | [-0.0104, -0.0022] | **yes** | 0–11 | 28 | +| dsg-eager vs dsg-merge | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-eager vs dsg-merge | conflation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-full vs dsg-eager | conll_f1 | -0.0001 | [-0.0035, +0.0031] | no | 5–2 | 28 | +| dsg-full vs dsg-eager | mention_acc | +0.0062 | [+0.0010, +0.0127] | **yes** | 9–2 | 28 | +| dsg-full vs dsg-eager | mention_acc_answered | -0.0091 | [-0.0372, +0.0119] | no | 11–5 | 28 | +| dsg-full vs dsg-eager | speaker_acc_matched | +0.0109 | [+0.0000, +0.0272] | no | 3–0 | 28 | +| dsg-full vs dsg-eager | violations_per_100w | -3.8607 | [-9.6754, +0.0000] | no | 0–2 | 28 | +| dsg-full vs dsg-eager | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-full vs dsg-eager | rollback | -5.5000 | [-6.7143, -4.3214] | **yes** | 0–28 | 28 | +| dsg-full vs dsg-eager | monotone_fraction | +0.0378 | [+0.0307, +0.0463] | **yes** | 28–0 | 28 | +| dsg-full vs dsg-eager | fragmentation | +0.0037 | [+0.0000, +0.0094] | no | 2–0 | 28 | +| dsg-full vs dsg-eager | conflation | -0.0206 | [-0.0393, -0.0061] | **yes** | 0–7 | 28 | +| retrospective vs dsg-full | conll_f1 | -0.0046 | [-0.0120, +0.0033] | no | 2–9 | 28 | +| retrospective vs dsg-full | mention_acc | +0.0503 | [+0.0380, +0.0644] | **yes** | 28–0 | 28 | +| retrospective vs dsg-full | mention_acc_answered | -0.0264 | [-0.0546, -0.0018] | **yes** | 9–18 | 28 | +| retrospective vs dsg-full | speaker_acc_matched | +0.0043 | [-0.0080, +0.0188] | no | 3–2 | 28 | +| retrospective vs dsg-full | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| retrospective vs dsg-full | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| retrospective vs dsg-full | rollback | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| retrospective vs dsg-full | monotone_fraction | -0.0042 | [-0.0079, -0.0014] | **yes** | 0–12 | 28 | +| retrospective vs dsg-full | fragmentation | -0.0029 | [-0.0186, +0.0107] | no | 4–3 | 28 | +| retrospective vs dsg-full | conflation | +0.0275 | [-0.0009, +0.0632] | no | 7–4 | 28 | +| dsg-full vs append-only | conll_f1 | -0.0026 | [-0.0070, +0.0012] | no | 7–8 | 28 | +| dsg-full vs append-only | mention_acc | -0.0171 | [-0.0338, -0.0038] | **yes** | 7–9 | 28 | +| dsg-full vs append-only | mention_acc_answered | -0.0729 | [-0.1298, -0.0284] | **yes** | 3–16 | 28 | +| dsg-full vs append-only | speaker_acc_matched | -0.0336 | [-0.0612, -0.0104] | **yes** | 2–7 | 28 | +| dsg-full vs append-only | violations_per_100w | -61.7864 | [-110.2998, -26.2329] | **yes** | 0–13 | 28 | +| dsg-full vs append-only | inconsistent_slot_rate | -0.1067 | [-0.1638, -0.0570] | **yes** | 0–12 | 28 | +| dsg-full vs append-only | rollback | +0.0357 | [+0.0000, +0.1071] | no | 1–0 | 28 | +| dsg-full vs append-only | monotone_fraction | -0.0050 | [-0.0091, -0.0019] | **yes** | 0–11 | 28 | +| dsg-full vs append-only | fragmentation | +0.0004 | [-0.0080, +0.0077] | no | 2–1 | 28 | +| dsg-full vs append-only | conflation | +0.0610 | [+0.0058, +0.1358] | **yes** | 9–6 | 28 | +| dsg-full vs window-only | conll_f1 | +0.0602 | [+0.0421, +0.0796] | **yes** | 23–1 | 28 | +| dsg-full vs window-only | mention_acc | +0.2972 | [+0.2427, +0.3512] | **yes** | 28–0 | 28 | +| dsg-full vs window-only | mention_acc_answered | -0.1182 | [-0.1942, -0.0549] | **yes** | 4–23 | 28 | +| dsg-full vs window-only | speaker_acc_matched | +0.0167 | [-0.0357, +0.0688] | no | 14–11 | 28 | +| dsg-full vs window-only | violations_per_100w | -75607.3421 | [-140680.6369, -28042.6731] | **yes** | 0–28 | 28 | +| dsg-full vs window-only | inconsistent_slot_rate | -0.0040 | [-0.0119, +0.0000] | no | 0–1 | 28 | +| dsg-full vs window-only | rollback | +0.0357 | [+0.0000, +0.1071] | no | 1–0 | 28 | +| dsg-full vs window-only | monotone_fraction | -0.0050 | [-0.0091, -0.0019] | **yes** | 0–11 | 28 | +| dsg-full vs window-only | fragmentation | -0.1997 | [-0.2529, -0.1475] | **yes** | 0–25 | 28 | +| dsg-full vs window-only | conflation | +0.1687 | [+0.0948, +0.2621] | **yes** | 21–0 | 28 | +| dsg-full vs dsg-merge | conll_f1 | -0.0001 | [-0.0035, +0.0031] | no | 5–2 | 28 | +| dsg-full vs dsg-merge | mention_acc | +0.0062 | [+0.0010, +0.0127] | **yes** | 9–2 | 28 | +| dsg-full vs dsg-merge | mention_acc_answered | -0.0091 | [-0.0372, +0.0119] | no | 11–5 | 28 | +| dsg-full vs dsg-merge | speaker_acc_matched | +0.0109 | [+0.0000, +0.0272] | no | 3–0 | 28 | +| dsg-full vs dsg-merge | violations_per_100w | -40.7415 | [-67.4195, -18.3721] | **yes** | 0–12 | 28 | +| dsg-full vs dsg-merge | inconsistent_slot_rate | -0.0997 | [-0.1563, -0.0501] | **yes** | 0–11 | 28 | +| dsg-full vs dsg-merge | rollback | -5.5000 | [-6.7143, -4.3214] | **yes** | 0–28 | 28 | +| dsg-full vs dsg-merge | monotone_fraction | +0.0322 | [+0.0230, +0.0423] | **yes** | 27–1 | 28 | +| dsg-full vs dsg-merge | fragmentation | +0.0037 | [+0.0000, +0.0094] | no | 2–0 | 28 | +| dsg-full vs dsg-merge | conflation | -0.0206 | [-0.0393, -0.0061] | **yes** | 0–7 | 28 | +| dsg-full vs retrospective | conll_f1 | +0.0046 | [-0.0033, +0.0120] | no | 9–2 | 28 | +| dsg-full vs retrospective | mention_acc | -0.0503 | [-0.0644, -0.0380] | **yes** | 0–28 | 28 | +| dsg-full vs retrospective | mention_acc_answered | +0.0264 | [+0.0018, +0.0546] | **yes** | 18–9 | 28 | +| dsg-full vs retrospective | speaker_acc_matched | -0.0043 | [-0.0188, +0.0080] | no | 2–3 | 28 | +| dsg-full vs retrospective | violations_per_100w | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-full vs retrospective | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-full vs retrospective | rollback | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-full vs retrospective | monotone_fraction | +0.0042 | [+0.0014, +0.0079] | **yes** | 12–0 | 28 | +| dsg-full vs retrospective | fragmentation | +0.0029 | [-0.0107, +0.0186] | no | 3–4 | 28 | +| dsg-full vs retrospective | conflation | -0.0275 | [-0.0632, +0.0009] | no | 4–7 | 28 | +| dsg-eager vs append-only | conll_f1 | -0.0025 | [-0.0061, +0.0005] | no | 2–9 | 28 | +| dsg-eager vs append-only | mention_acc | -0.0232 | [-0.0404, -0.0089] | **yes** | 4–10 | 28 | +| dsg-eager vs append-only | mention_acc_answered | -0.0638 | [-0.1137, -0.0249] | **yes** | 2–16 | 28 | +| dsg-eager vs append-only | speaker_acc_matched | -0.0445 | [-0.0811, -0.0158] | **yes** | 1–9 | 28 | +| dsg-eager vs append-only | violations_per_100w | -57.9257 | [-105.9478, -23.6706] | **yes** | 0–12 | 28 | +| dsg-eager vs append-only | inconsistent_slot_rate | -0.1067 | [-0.1638, -0.0570] | **yes** | 0–12 | 28 | +| dsg-eager vs append-only | rollback | +5.5357 | [+4.3571, +6.7500] | **yes** | 28–0 | 28 | +| dsg-eager vs append-only | monotone_fraction | -0.0428 | [-0.0510, -0.0354] | **yes** | 0–28 | 28 | +| dsg-eager vs append-only | fragmentation | -0.0032 | [-0.0097, +0.0000] | no | 0–1 | 28 | +| dsg-eager vs append-only | conflation | +0.0816 | [+0.0311, +0.1533] | **yes** | 11–0 | 28 | + +## Identity-resolution latency (measurement instrument) + +How long a novel holds a figure unnamed before the reader is told who they are, by narrative person. Unpaired bootstrap on the group means; the first-person group is small, so read the interval, not the point. + +| metric | first-person | third-person | Δ | 95% CI | excludes 0 | +|---|---|---|---|---|---| +| mean_latency | 0.289 (n=7) | 0.131 (n=21) | +0.158 | [+0.017, +0.343] | **yes** | +| median_latency | 0.240 (n=7) | 0.074 (n=21) | +0.166 | [+0.010, +0.373] | **yes** | +| unresolved_fraction | 0.825 (n=7) | 0.845 (n=21) | -0.020 | [-0.084, +0.048] | no | +| max_latency | 0.529 (n=7) | 0.398 (n=21) | +0.131 | [-0.092, +0.328] | no | + +## One character, traced (dsg-full) + +### AHandfulOfDust + +**Tony Last** — what the reader believes, and when. + +| read at | predicate | value | held until | status | source | +|---|---|---|---|---|---| +| 0.04 | other:has_a_room | Morgan le Fay | — | believed | narrated | +| 0.10 | other:wears_a_dark_suit_on_sundays | narrator | — | believed | narrated | +| 0.86 | other:saw_brenda | John Beaver of Bratt's Club | — | believed | narrated | + +### APassageToIndia + +**Dr. Aziz** — what the reader believes, and when. + +| read at | predicate | value | held until | status | source | +|---|---|---|---|---|---| +| 0.16 | identity_of | the city | — | believed | narrated | +| 0.43 | member_of | the city | — | believed | narrated | +| 0.50 | resides_in | the city | — | believed | narrated | +| 0.62 | married_to | the city | — | believed | narrated | +| 0.62 | other:is | an innocent man | — | believed | narrated | +| 0.63 | other:is | a | — | believed | narrated | +| 0.70 | other:followed | prisoner | — | believed | narrated | +| 0.80 | other:friend | Hassan | — | believed | narrated | +| 0.85 | other:addicted | to cameos | — | believed | narrated | +| 0.92 | other:skilful | in the slighter impertinences | — | believed | narrated | +| 0.93 | other:lives_in_the_guest_house | residing_in | — | believed | narrated | +| 0.96 | other:started_again | <> | — | believed | narrated | + + +## Revision events on real text (dsg-full) + +| op | book | position | change | evidence | +|---|---|---|---|---| +| `merge_provisional` | AHandfulOfDust | 0.96 | absorbed e00089: link 'the half-caste prospector' -> 'John Beaver of Bratt's Club' | …f comprehension, and he went away unsatisfied. At their mid-day meal Mr Todd said, "Mr Last, the Indians tell me that you have been trying to speak wi… | +| `merge_provisional` | AHandfulOfDust | 0.04 | absorbed e00004: link 'Guinevere' -> 'Morgan le Fay' | …t arcs of flying fish ("Look, Ernie, come quick, there's a shark." "That's not a shark, it's a dolphin." "Mr Brink said it was a porpoise." "There he … | +| `elaborate` | AHandfulOfDust | 0.95 | provisional -> committed | …lly propped the motor-cycle against the wall of the drawbridge and ran in to breakfast. Life at Hetton was busier but simpler since Richard Last's suc… | +| `elaborate` | AHandfulOfDust | 0.92 | provisional -> committed | …formed by a rough platform erected in the eaves of the roof. Mr Todd propped a ladder against it and mounted. Tony followed, still unsteady after his … | +| `elaborate` | AHandfulOfDust | 0.69 | provisional -> committed | …nursing. If only we were at a village...." "I'm afraid I'm being a great nuisance." "That's not the point. The thing is to find what is best for us to… | +| `elaborate` | AHandfulOfDust | 0.66 | bound 'an old horse' to a horse | …t arcs of flying fish ("Look, Ernie, come quick, there's a shark." "That's not a shark, it's a dolphin." "Mr Brink said it was a porpoise." "There he … | +| `elaborate` | AHandfulOfDust | 0.61 | provisional -> committed | …, he was sauntering among his guests refilling their glasses. "This stuff's all right," he said, showing the label, "it won't hurt you. It's the right… | +| `elaborate` | AHandfulOfDust | 0.47 | provisional -> committed | …You come in boats. All these stores stay here. All these stores go in boats. You tell men that." Rosa said nothing. "Understand?" "No peoples go in bo… | +| `elaborate` | AHandfulOfDust | 0.38 | provisional -> committed | …le down with you again." While Allan was there, Tony resolutely refused to listen, but later the words, and the picture they evoked, would not leave h… | +| `elaborate` | AHandfulOfDust | 0.34 | provisional -> committed | …ins. So many of our guests nowadays are on a diet. I think muffins one of the few things that make the English winter endurable." "Muffins stand for s… | +| `elaborate` | AHandfulOfDust | 0.32 | provisional -> committed | …e Empire chair against the wall, perfectly still with her hands folded in her lap, like a small well-brought-up child introduced into a room full of g… | +| `elaborate` | AHandfulOfDust | 0.26 | provisional -> committed | …know." "How?" "Well, my mother let it to her, you know." Tony was greatly surprised and taxed Brenda with this. "You never told me who was behind your… | +| `elaborate` | AHandfulOfDust | 0.10 | provisional -> committed | …sunset to dawn with the small wick glowing in the beef fat and the palm thatch overhead dimly discernible; but Tony lived in quiet confidence and expe… | +| `elaborate` | AHandfulOfDust | 0.08 | provisional -> committed | …ap like Beaver. She'll want to come back in a year, just you see. Allan says the same." "I've told Allan. I don't want her back." "Well, that's vindic… | +| `elaborate` | AHandfulOfDust | 0.07 | provisional -> committed | …scudding along, bowed to the wind, prayer-books clutched in their gloved hands. Four or five rugged old men were hobbling down to bathe, hissing like … | +| `elaborate` | AHandfulOfDust | 0.07 | provisional -> committed | …aid, "Ambrose, when I'm alone I think in future I'll have dinner in the library." Afterwards he sat with a book in front of the fire but he was unable… | +| `elaborate` | AHandfulOfDust | 0.04 | provisional -> committed | …t arcs of flying fish ("Look, Ernie, come quick, there's a shark." "That's not a shark, it's a dolphin." "Mr Brink said it was a porpoise." "There he … | +| `elaborate` | AHandfulOfDust | 0.01 | provisional -> committed | …clock, though the wind had got up, fluttering the papers on which the order of the service was printed and once threatening to unveil the memorial pre… | +| `merge_provisional` | APassageToIndia | 0.71 | absorbed e00153: link 'forward, forward' -> 'Mahmoud Ali' | …s prey. "Forward, forward," shrieked Mahmoud Ali, whose every utterance had become a yell. "Down with the Collector, down with the Superintendent of P… | +| `merge_provisional` | APassageToIndia | 0.63 | absorbed e00127: link 'much more power than' -> 'Aziz' | …e." "It is useless discussing Hindus with me. Living with them teaches me no more. When I think I annoy them, I do not. When I think I don't annoy the… | + +## Figures + +![main](figures/fig1-main-results.png) +![ladder](figures/fig1b-ladder.png) +![prefix](figures/fig2-prefix-curve.png) +![paired](figures/fig3-paired-deltas.png) +![revision](figures/fig4-revision-profile.png) +![violations](figures/fig5-violation-growth.png) diff --git a/report/pdnc-qwen3b/figures/fig1-main-results.pdf b/report/pdnc-qwen3b/figures/fig1-main-results.pdf new file mode 100644 index 0000000000000000000000000000000000000000..1477b82d67aa47c3407cc2f73e7d928ed688e2d7 Binary files /dev/null and b/report/pdnc-qwen3b/figures/fig1-main-results.pdf differ diff --git a/report/pdnc-qwen3b/figures/fig1-main-results.png b/report/pdnc-qwen3b/figures/fig1-main-results.png new file mode 100644 index 0000000000000000000000000000000000000000..08797b821b37793f85f216e1b743972743e76833 --- /dev/null +++ b/report/pdnc-qwen3b/figures/fig1-main-results.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:32aab10cd382fd882be2c25427059ea1806e6a6284d3877292a7ffc8e8f2f5d8 +size 151226 diff --git a/report/pdnc-qwen3b/figures/fig1b-ladder.pdf b/report/pdnc-qwen3b/figures/fig1b-ladder.pdf new file mode 100644 index 0000000000000000000000000000000000000000..d428dd8f77fe8d632e68796c755c5b9ad6cfeee3 Binary files /dev/null and b/report/pdnc-qwen3b/figures/fig1b-ladder.pdf differ diff --git a/report/pdnc-qwen3b/figures/fig1b-ladder.png b/report/pdnc-qwen3b/figures/fig1b-ladder.png new file mode 100644 index 0000000000000000000000000000000000000000..daaa7b0bf2da0c554c656046bbda536327b95c1c --- /dev/null +++ b/report/pdnc-qwen3b/figures/fig1b-ladder.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e2bc0a4bbf2e3d59b09321eed76e47f268fd960bdfaa979e96d1f4d840df76be +size 100633 diff --git a/report/pdnc-qwen3b/figures/fig2-prefix-curve.pdf b/report/pdnc-qwen3b/figures/fig2-prefix-curve.pdf new file mode 100644 index 0000000000000000000000000000000000000000..6dd4221dc99477f5aca5f3647a7612037730eefc Binary files /dev/null and b/report/pdnc-qwen3b/figures/fig2-prefix-curve.pdf differ diff --git a/report/pdnc-qwen3b/figures/fig2-prefix-curve.png b/report/pdnc-qwen3b/figures/fig2-prefix-curve.png new file mode 100644 index 0000000000000000000000000000000000000000..cbbbe8936f5bd5246b04a558faa5c7849aa2ad8e Binary files /dev/null and b/report/pdnc-qwen3b/figures/fig2-prefix-curve.png differ diff --git a/report/pdnc-qwen3b/figures/fig3-paired-deltas.pdf b/report/pdnc-qwen3b/figures/fig3-paired-deltas.pdf new file mode 100644 index 0000000000000000000000000000000000000000..70711fa2d733e4e41157fc0a346356aed2f9c189 Binary files /dev/null and b/report/pdnc-qwen3b/figures/fig3-paired-deltas.pdf differ diff --git a/report/pdnc-qwen3b/figures/fig3-paired-deltas.png b/report/pdnc-qwen3b/figures/fig3-paired-deltas.png new file mode 100644 index 0000000000000000000000000000000000000000..6f8cb65b0cfc2a60fb88faade97f27faae68779c --- /dev/null +++ b/report/pdnc-qwen3b/figures/fig3-paired-deltas.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:af80e480b69576f40129c587b2c7163839fb9f8f28b7e276ed0e589e310c2fce +size 146711 diff --git a/report/pdnc-qwen3b/figures/fig4-revision-profile.pdf b/report/pdnc-qwen3b/figures/fig4-revision-profile.pdf new file mode 100644 index 0000000000000000000000000000000000000000..d7db9d0672a91368bb55dc5da9fc619651742d10 Binary files /dev/null and b/report/pdnc-qwen3b/figures/fig4-revision-profile.pdf differ diff --git a/report/pdnc-qwen3b/figures/fig4-revision-profile.png b/report/pdnc-qwen3b/figures/fig4-revision-profile.png new file mode 100644 index 0000000000000000000000000000000000000000..dd3cf88579b3c6db07870d3cff7c9d11b28c606b Binary files /dev/null and b/report/pdnc-qwen3b/figures/fig4-revision-profile.png differ diff --git a/report/pdnc-qwen3b/figures/fig5-violation-growth.pdf b/report/pdnc-qwen3b/figures/fig5-violation-growth.pdf new file mode 100644 index 0000000000000000000000000000000000000000..272836a775eea240eb7e4ba40c89160f120483ab Binary files /dev/null and b/report/pdnc-qwen3b/figures/fig5-violation-growth.pdf differ diff --git a/report/pdnc-qwen3b/figures/fig5-violation-growth.png b/report/pdnc-qwen3b/figures/fig5-violation-growth.png new file mode 100644 index 0000000000000000000000000000000000000000..7a928089a4b392abefd0bffad70f936653cb208a Binary files /dev/null and b/report/pdnc-qwen3b/figures/fig5-violation-growth.png differ diff --git a/report/pdnc-qwen3b/results.md b/report/pdnc-qwen3b/results.md new file mode 100644 index 0000000000000000000000000000000000000000..e614c2c56371921acea083db2c8b2271cee97868 --- /dev/null +++ b/report/pdnc-qwen3b/results.md @@ -0,0 +1,206 @@ +# Dynamic Story Graph — results + +Books: **28** · policies: window-only, append-only, dsg-merge, dsg-eager, dsg-full, retrospective · scoring time: 128.8s + +## Means over books + +| policy | conll_f1 | mention_acc | mention_acc_answered | b3_f1_multi | coverage | fragmentation | conflation | inconsistent_slot_rate | speaker_acc | speaker_acc_matched | violations_per_100w | monotone_fraction | rollback | live_entities | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| window-only | 0.447 | 0.096 | 0.959 | 0.530 | 0.357 | 0.361 | 0.001 | 0.101 | 0.037 | 0.587 | 56247.2 | 1.000 | 0.000 | 141.9 | +| append-only | 0.536 | 0.406 | 0.898 | 0.585 | 0.345 | 0.116 | 0.094 | 0.423 | 0.035 | 0.572 | 13826.3 | 1.000 | 0.000 | 24.143 | +| dsg-merge | 0.532 | 0.390 | 0.830 | 0.584 | 0.352 | 0.106 | 0.245 | 0.433 | 0.032 | 0.523 | 13542.4 | 0.987 | 2.357 | 20.464 | +| dsg-eager | 0.532 | 0.390 | 0.830 | 0.584 | 0.352 | 0.106 | 0.245 | 0.000 | 0.032 | 0.523 | 0.397 | 0.788 | 35.500 | 20.464 | +| dsg-full | 0.531 | 0.404 | 0.850 | 0.582 | 0.352 | 0.117 | 0.215 | 0.000 | 0.035 | 0.560 | 0.182 | 0.813 | 32.929 | 21.071 | +| retrospective | 0.535 | 0.435 | 0.843 | 0.584 | 0.353 | 0.123 | 0.187 | 0.000 | 0.035 | 0.559 | 0.000 | 0.793 | 33.464 | 21.321 | + +## Paired comparisons (bootstrap over books) + +| comparison | metric | mean Δ | 95% CI | excludes 0 | wins | n | +|---|---|---|---|---|---|---| +| append-only vs window-only | conll_f1 | +0.0884 | [+0.0657, +0.1111] | **yes** | 23–0 | 28 | +| append-only vs window-only | mention_acc | +0.3100 | [+0.2651, +0.3532] | **yes** | 28–0 | 28 | +| append-only vs window-only | mention_acc_answered | -0.0606 | [-0.0950, -0.0293] | **yes** | 6–19 | 28 | +| append-only vs window-only | speaker_acc_matched | -0.0149 | [-0.0292, -0.0012] | **yes** | 3–13 | 28 | +| append-only vs window-only | violations_per_100w | -42420.8364 | [-59687.1868, -27206.8249] | **yes** | 0–28 | 28 | +| append-only vs window-only | inconsistent_slot_rate | +0.3226 | [+0.2838, +0.3577] | **yes** | 28–0 | 28 | +| append-only vs window-only | rollback | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| append-only vs window-only | monotone_fraction | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| append-only vs window-only | fragmentation | -0.2445 | [-0.3051, -0.1851] | **yes** | 0–24 | 28 | +| append-only vs window-only | conflation | +0.0933 | [+0.0517, +0.1410] | **yes** | 14–0 | 28 | +| dsg-merge vs append-only | conll_f1 | -0.0035 | [-0.0074, +0.0008] | no | 3–16 | 28 | +| dsg-merge vs append-only | mention_acc | -0.0163 | [-0.0283, -0.0053] | **yes** | 6–15 | 28 | +| dsg-merge vs append-only | mention_acc_answered | -0.0684 | [-0.1265, -0.0261] | **yes** | 2–24 | 28 | +| dsg-merge vs append-only | speaker_acc_matched | -0.0496 | [-0.0905, -0.0165] | **yes** | 3–11 | 28 | +| dsg-merge vs append-only | violations_per_100w | -283.9537 | [-595.7302, -60.6291] | **yes** | 10–12 | 28 | +| dsg-merge vs append-only | inconsistent_slot_rate | +0.0099 | [-0.0025, +0.0238] | no | 10–6 | 28 | +| dsg-merge vs append-only | rollback | +2.3571 | [+1.6786, +3.3929] | **yes** | 26–0 | 28 | +| dsg-merge vs append-only | monotone_fraction | -0.0131 | [-0.0206, -0.0081] | **yes** | 0–26 | 28 | +| dsg-merge vs append-only | fragmentation | -0.0102 | [-0.0193, -0.0029] | **yes** | 0–8 | 28 | +| dsg-merge vs append-only | conflation | +0.1503 | [+0.0738, +0.2604] | **yes** | 19–0 | 28 | +| dsg-eager vs dsg-merge | conll_f1 | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-eager vs dsg-merge | mention_acc | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-eager vs dsg-merge | mention_acc_answered | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-eager vs dsg-merge | speaker_acc_matched | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-eager vs dsg-merge | violations_per_100w | -13541.9963 | [-19396.4653, -8342.3598] | **yes** | 0–28 | 28 | +| dsg-eager vs dsg-merge | inconsistent_slot_rate | -0.4332 | [-0.4779, -0.3855] | **yes** | 0–28 | 28 | +| dsg-eager vs dsg-merge | rollback | +33.1429 | [+24.7857, +42.4286] | **yes** | 28–0 | 28 | +| dsg-eager vs dsg-merge | monotone_fraction | -0.1988 | [-0.2259, -0.1699] | **yes** | 0–28 | 28 | +| dsg-eager vs dsg-merge | fragmentation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-eager vs dsg-merge | conflation | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-full vs dsg-eager | conll_f1 | -0.0015 | [-0.0063, +0.0025] | no | 5–4 | 28 | +| dsg-full vs dsg-eager | mention_acc | +0.0139 | [+0.0052, +0.0245] | **yes** | 10–0 | 28 | +| dsg-full vs dsg-eager | mention_acc_answered | +0.0200 | [-0.0016, +0.0499] | no | 8–4 | 28 | +| dsg-full vs dsg-eager | speaker_acc_matched | +0.0373 | [+0.0091, +0.0750] | **yes** | 8–0 | 28 | +| dsg-full vs dsg-eager | violations_per_100w | -0.2149 | [-0.6448, +0.0000] | no | 0–1 | 28 | +| dsg-full vs dsg-eager | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-full vs dsg-eager | rollback | -2.5714 | [-3.7143, -1.7500] | **yes** | 0–25 | 28 | +| dsg-full vs dsg-eager | monotone_fraction | +0.0253 | [+0.0194, +0.0333] | **yes** | 28–0 | 28 | +| dsg-full vs dsg-eager | fragmentation | +0.0110 | [+0.0016, +0.0237] | **yes** | 4–0 | 28 | +| dsg-full vs dsg-eager | conflation | -0.0295 | [-0.0526, -0.0109] | **yes** | 0–9 | 28 | +| retrospective vs dsg-full | conll_f1 | +0.0046 | [-0.0005, +0.0107] | no | 10–8 | 28 | +| retrospective vs dsg-full | mention_acc | +0.0310 | [+0.0178, +0.0452] | **yes** | 23–4 | 28 | +| retrospective vs dsg-full | mention_acc_answered | -0.0067 | [-0.0394, +0.0240] | no | 13–13 | 28 | +| retrospective vs dsg-full | speaker_acc_matched | -0.0010 | [-0.0360, +0.0256] | no | 8–8 | 28 | +| retrospective vs dsg-full | violations_per_100w | -0.1819 | [-0.5456, +0.0000] | no | 0–1 | 28 | +| retrospective vs dsg-full | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| retrospective vs dsg-full | rollback | +0.5357 | [-0.4286, +1.5357] | no | 9–4 | 28 | +| retrospective vs dsg-full | monotone_fraction | -0.0201 | [-0.0261, -0.0146] | **yes** | 2–26 | 28 | +| retrospective vs dsg-full | fragmentation | +0.0055 | [-0.0058, +0.0170] | no | 6–3 | 28 | +| retrospective vs dsg-full | conflation | -0.0278 | [-0.0698, +0.0164] | no | 3–11 | 28 | +| dsg-full vs append-only | conll_f1 | -0.0050 | [-0.0108, +0.0002] | no | 5–15 | 28 | +| dsg-full vs append-only | mention_acc | -0.0023 | [-0.0168, +0.0115] | no | 10–9 | 28 | +| dsg-full vs append-only | mention_acc_answered | -0.0484 | [-0.0879, -0.0138] | **yes** | 3–23 | 28 | +| dsg-full vs append-only | speaker_acc_matched | -0.0123 | [-0.0336, +0.0053] | no | 8–6 | 28 | +| dsg-full vs append-only | violations_per_100w | -13826.1650 | [-19694.3909, -8538.2675] | **yes** | 0–28 | 28 | +| dsg-full vs append-only | inconsistent_slot_rate | -0.4233 | [-0.4694, -0.3744] | **yes** | 0–28 | 28 | +| dsg-full vs append-only | rollback | +32.9286 | [+24.5000, +42.3214] | **yes** | 28–0 | 28 | +| dsg-full vs append-only | monotone_fraction | -0.1866 | [-0.2130, -0.1586] | **yes** | 0–28 | 28 | +| dsg-full vs append-only | fragmentation | +0.0008 | [-0.0101, +0.0136] | no | 2–5 | 28 | +| dsg-full vs append-only | conflation | +0.1208 | [+0.0368, +0.2375] | **yes** | 14–4 | 28 | +| dsg-full vs window-only | conll_f1 | +0.0834 | [+0.0618, +0.1050] | **yes** | 23–2 | 28 | +| dsg-full vs window-only | mention_acc | +0.3077 | [+0.2627, +0.3492] | **yes** | 28–0 | 28 | +| dsg-full vs window-only | mention_acc_answered | -0.1090 | [-0.1471, -0.0741] | **yes** | 2–25 | 28 | +| dsg-full vs window-only | speaker_acc_matched | -0.0272 | [-0.0531, -0.0047] | **yes** | 7–13 | 28 | +| dsg-full vs window-only | violations_per_100w | -56247.0014 | [-77477.5923, -36997.9241] | **yes** | 0–28 | 28 | +| dsg-full vs window-only | inconsistent_slot_rate | -0.1006 | [-0.1208, -0.0804] | **yes** | 0–26 | 28 | +| dsg-full vs window-only | rollback | +32.9286 | [+24.5000, +42.3214] | **yes** | 28–0 | 28 | +| dsg-full vs window-only | monotone_fraction | -0.1866 | [-0.2130, -0.1586] | **yes** | 0–28 | 28 | +| dsg-full vs window-only | fragmentation | -0.2437 | [-0.3017, -0.1861] | **yes** | 0–25 | 28 | +| dsg-full vs window-only | conflation | +0.2141 | [+0.1361, +0.3214] | **yes** | 23–0 | 28 | +| dsg-full vs dsg-merge | conll_f1 | -0.0015 | [-0.0063, +0.0025] | no | 5–4 | 28 | +| dsg-full vs dsg-merge | mention_acc | +0.0139 | [+0.0052, +0.0245] | **yes** | 10–0 | 28 | +| dsg-full vs dsg-merge | mention_acc_answered | +0.0200 | [-0.0016, +0.0499] | no | 8–4 | 28 | +| dsg-full vs dsg-merge | speaker_acc_matched | +0.0373 | [+0.0091, +0.0750] | **yes** | 8–0 | 28 | +| dsg-full vs dsg-merge | violations_per_100w | -13542.2113 | [-19396.6802, -8342.3598] | **yes** | 0–28 | 28 | +| dsg-full vs dsg-merge | inconsistent_slot_rate | -0.4332 | [-0.4779, -0.3855] | **yes** | 0–28 | 28 | +| dsg-full vs dsg-merge | rollback | +30.5714 | [+22.3929, +39.6786] | **yes** | 27–0 | 28 | +| dsg-full vs dsg-merge | monotone_fraction | -0.1735 | [-0.2021, -0.1430] | **yes** | 1–27 | 28 | +| dsg-full vs dsg-merge | fragmentation | +0.0110 | [+0.0016, +0.0237] | **yes** | 4–0 | 28 | +| dsg-full vs dsg-merge | conflation | -0.0295 | [-0.0526, -0.0109] | **yes** | 0–9 | 28 | +| dsg-full vs retrospective | conll_f1 | -0.0046 | [-0.0107, +0.0005] | no | 8–10 | 28 | +| dsg-full vs retrospective | mention_acc | -0.0310 | [-0.0452, -0.0178] | **yes** | 4–23 | 28 | +| dsg-full vs retrospective | mention_acc_answered | +0.0067 | [-0.0240, +0.0394] | no | 13–13 | 28 | +| dsg-full vs retrospective | speaker_acc_matched | +0.0010 | [-0.0256, +0.0360] | no | 8–8 | 28 | +| dsg-full vs retrospective | violations_per_100w | +0.1819 | [+0.0000, +0.5456] | no | 1–0 | 28 | +| dsg-full vs retrospective | inconsistent_slot_rate | +0.0000 | [+0.0000, +0.0000] | no | 0–0 | 28 | +| dsg-full vs retrospective | rollback | -0.5357 | [-1.5357, +0.4286] | no | 4–9 | 28 | +| dsg-full vs retrospective | monotone_fraction | +0.0201 | [+0.0146, +0.0261] | **yes** | 26–2 | 28 | +| dsg-full vs retrospective | fragmentation | -0.0055 | [-0.0170, +0.0058] | no | 3–6 | 28 | +| dsg-full vs retrospective | conflation | +0.0278 | [-0.0164, +0.0698] | no | 11–3 | 28 | +| dsg-eager vs append-only | conll_f1 | -0.0035 | [-0.0074, +0.0008] | no | 3–16 | 28 | +| dsg-eager vs append-only | mention_acc | -0.0163 | [-0.0283, -0.0053] | **yes** | 6–15 | 28 | +| dsg-eager vs append-only | mention_acc_answered | -0.0684 | [-0.1265, -0.0261] | **yes** | 2–24 | 28 | +| dsg-eager vs append-only | speaker_acc_matched | -0.0496 | [-0.0905, -0.0165] | **yes** | 3–11 | 28 | +| dsg-eager vs append-only | violations_per_100w | -13825.9500 | [-19694.3855, -8538.2621] | **yes** | 0–28 | 28 | +| dsg-eager vs append-only | inconsistent_slot_rate | -0.4233 | [-0.4694, -0.3744] | **yes** | 0–28 | 28 | +| dsg-eager vs append-only | rollback | +35.5000 | [+26.8563, +45.0357] | **yes** | 28–0 | 28 | +| dsg-eager vs append-only | monotone_fraction | -0.2119 | [-0.2382, -0.1838] | **yes** | 0–28 | 28 | +| dsg-eager vs append-only | fragmentation | -0.0102 | [-0.0193, -0.0029] | **yes** | 0–8 | 28 | +| dsg-eager vs append-only | conflation | +0.1503 | [+0.0738, +0.2604] | **yes** | 19–0 | 28 | + +## Where the reader's model is refined versus corrected + +Spearman correlation between position in the book and the share of update operations of each kind. The shuffled row permutes reading order, which preserves state growth exactly while destroying narrative order: a trend that survives the shuffle is an artefact of the state getting larger, one that disappears is a property of the text. + +| ordering | operation | rho | p | first bin | last bin | +|---|---|---|---|---|---| +| narrative | elaboration | -0.586 | 0.0066 | 0.187 | 0.175 | +| narrative | revision | +0.821 | 9.1e-06 | 0.046 | 0.298 | +| shuffled | elaboration | +0.077 | 0.75 | 0.119 | 0.180 | +| shuffled | revision | +0.690 | 0.00076 | 0.066 | 0.228 | + +## Identity-resolution latency (measurement instrument) + +How long a novel holds a figure unnamed before the reader is told who they are, by narrative person. Unpaired bootstrap on the group means; the first-person group is small, so read the interval, not the point. + +| metric | first-person | third-person | Δ | 95% CI | excludes 0 | +|---|---|---|---|---|---| +| mean_latency | 0.064 (n=7) | 0.060 (n=21) | +0.004 | [-0.066, +0.085] | no | +| median_latency | 0.064 (n=7) | 0.049 (n=21) | +0.016 | [-0.056, +0.096] | no | +| unresolved_fraction | 0.616 (n=7) | 0.538 (n=21) | +0.078 | [-0.187, +0.324] | no | +| max_latency | 0.090 (n=7) | 0.127 (n=21) | -0.037 | [-0.147, +0.088] | no | + +## One character, traced (dsg-full) + +### AHandfulOfDust + +**Tony Last** — what the reader believes, and when. + +| read at | predicate | value | held until | status | source | +|---|---|---|---|---|---| +| 0.10 | emotion | feels awful | window 13 | superseded | narrated | +| 0.11 | emotion | annoyed | window 30 | superseded | narrated | +| 0.12 | true_name | Tendril "e do speak uncommon "igh of the | window 30 | retracted | narrated | +| 0.25 | emotion | slightly torpid | window 33 | superseded | narrated | +| 0.25 | true_name | Tendril | window 83 | retracted | narrated | +| 0.27 | emotion | low | window 48 | superseded | narrated | +| 0.39 | emotion | depressed | window 91 | superseded | narrated | +| 0.40 | identity_of | Mrs Rattery | window 50 | retracted | narrated | +| 0.41 | identity_of | Tony | window 52 | retracted | narrated | +| 0.43 | identity_of | the young American | window 58 | retracted | narrated | +| 0.47 | identity_of | Albert | window 69 | retracted | narrated | +| 0.53 | knows | John Beaver | — | believed | narrated | + +### APassageToIndia + +**Dr. Aziz** — what the reader believes, and when. + +| read at | predicate | value | held until | status | source | +|---|---|---|---|---|---| +| 0.30 | identity_of | Hassan | — | believed | narrated | +| 0.30 | member_of | the Indian community | — | believed | narrated | + + +## Revision events on real text (dsg-full) + +| op | book | position | change | evidence | +|---|---|---|---|---| +| `revise` | AHandfulOfDust | 0.99 | Reggie St Cloud.true_name: 'Reginald St Cloud' → 'Princess Abdul Akbar' | …clock, though the wind had got up, fluttering the papers on which the order of the service was printed and once threatening to unveil the memorial pre… | +| `revise` | AHandfulOfDust | 0.97 | Tony.identity_of: 'the host' → 'Tony' | …sunset to dawn with the small wick glowing in the beef fat and the palm thatch overhead dimly discernible; but Tony lived in quiet confidence and expe… | +| `revise` | AHandfulOfDust | 0.95 | Tony.identity_of: 'the old man' → 'the guest' | …hanked his guest courteously. "I enjoyed that _very_ much. It was an extremely distressing chapter. But, if I remember it rightly, it will all turn ou… | +| `revise` | AHandfulOfDust | 0.95 | Tony.identity_of: 'the guest' → 'the friend' | …hanked his guest courteously. "I enjoyed that _very_ much. It was an extremely distressing chapter. But, if I remember it rightly, it will all turn ou… | +| `revise` | AHandfulOfDust | 0.95 | Tony.identity_of: 'the friend' → 'the host' | …hanked his guest courteously. "I enjoyed that _very_ much. It was an extremely distressing chapter. But, if I remember it rightly, it will all turn ou… | +| `revise` | AHandfulOfDust | 0.95 | Tony.identity_of: 'the host' → 'the reader' | …hanked his guest courteously. "I enjoyed that _very_ much. It was an extremely distressing chapter. But, if I remember it rightly, it will all turn ou… | +| `revise` | AHandfulOfDust | 0.95 | Tony.identity_of: 'the reader' → 'the traveler' | …hanked his guest courteously. "I enjoyed that _very_ much. It was an extremely distressing chapter. But, if I remember it rightly, it will all turn ou… | +| `revise` | AHandfulOfDust | 0.95 | Tony.identity_of: 'the traveler' → 'the guest' | …hanked his guest courteously. "I enjoyed that _very_ much. It was an extremely distressing chapter. But, if I remember it rightly, it will all turn ou… | +| `revise` | AHandfulOfDust | 0.95 | Tony.identity_of: 'the guest' → 'the host' | …hanked his guest courteously. "I enjoyed that _very_ much. It was an extremely distressing chapter. But, if I remember it rightly, it will all turn ou… | +| `revise` | AHandfulOfDust | 0.92 | Tony.identity_of: 'Dr Messinger' → 'the old man' | …it ran through rapids, always dangerous and at most seasons of the year impassable, to join the upper waters of the river where Dr Messinger had come … | +| `revise` | AHandfulOfDust | 0.92 | Tony.true_name: 'John Beaver' → 'Winterbourne' | …it ran through rapids, always dangerous and at most seasons of the year impassable, to join the upper waters of the river where Dr Messinger had come … | +| `revise` | AHandfulOfDust | 0.90 | Brenda.identity_of: 'Jock' → 'Princess Abdul Akbar' | …nces but a constant round of more modest entertaining; that August in London was the gayest time of all (he rewrote this annually in slightly differen… | +| `revise` | AHandfulOfDust | 0.85 | Tony.child_of: 'Reggie St Cloud' → 'Hetton' | …t's quite light," she added. "After five. Oughtn't we to go to bed?" "Yes." "Only another week and then all the parties will be over," said Beaver. "Y… | +| `revise` | AHandfulOfDust | 0.84 | Tony.identity_of: 'John Beaver' → 'Tony' | …ost three and sixpence each, and Tony remembered vividly the embarrassment with which he had witnessed their demonstration on the floor of the toy dep… | +| `revise` | AHandfulOfDust | 0.84 | Tony.identity_of: 'Tony' → 'Dr Messinger' | …ost three and sixpence each, and Tony remembered vividly the embarrassment with which he had witnessed their demonstration on the floor of the toy dep… | +| `revise` | AHandfulOfDust | 0.84 | Rosa.identity_of: 'Mr Forbes' companion' → 'Tony' | …You come in boats. All these stores stay here. All these stores go in boats. You tell men that." Rosa said nothing. "Understand?" "No peoples go in bo… | +| `revise` | AHandfulOfDust | 0.83 | Rosa.identity_of: 'Rosa' → 'Mr Forbes' companion' | …he part of the camp where they ate and slept. "Rosa, we have decided to take you down the river with us. We need you to help us talk to the men. Under… | +| `revise` | AHandfulOfDust | 0.80 | Tony.identity_of: 'Winterbourne' → 'John Beaver' | …But almost immediately they crossed a stream flowing in the opposite direction. "Very curious," said Dr Messinger. "A discovery of genuine scientific … | +| `revise` | AHandfulOfDust | 0.80 | Jenny.identity_of: 'Princess Abdul Akbar' → 'Veronica' | …lackballed?" "I shouldn't know. Gentlemen are so odd about their clubs." "I thought that you were going to make Allan and Reggie support me." "I asked… | +| `revise` | AHandfulOfDust | 0.75 | Tony.identity_of: 'Princess Abdul Akbar' → 'Winterbourne' | …found a way into the buttonholes of his shirt and the laces of his breeches; mosquitoes had got him at the ankles when he changed into slacks for the … | + +## Figures + +![main](figures/fig1-main-results.png) +![ladder](figures/fig1b-ladder.png) +![prefix](figures/fig2-prefix-curve.png) +![paired](figures/fig3-paired-deltas.png) +![revision](figures/fig4-revision-profile.png) +![violations](figures/fig5-violation-growth.png)