Generation findings
Four runs, 18,600 generated chapters, 30 stories, conditions paired within story. Canon is planted in chapter 1 and withheld afterwards; violations are deterministic string tests near a mention of the subject. No language model appears in any metric.
Summary. Five ways of using a narrative graph at generation time were tested. None of them helps. The fifth, ranking candidates, is the one the project's own negative results predicted should work, and it does not either.
1. What was tried
| use of the graph | design | outcome |
|---|---|---|
| carried as context | serialised state digest in the prompt | worse than no memory |
| carried plus recent text | digest and the previous chapter | loses to the previous chapter alone |
| queried by the next beat | retrieve only what the beat implicates | worse than no memory |
| queried plus recent text | retrieval and the previous chapter | ties the previous chapter alone |
| used to rank candidates | draw four, pick the most consistent | indistinguishable from picking blind |
2. The ranking result, and why the control mattered
Ranking was motivated by the finding in METHODOLOGY.md section 5 that absolute
contradiction counts are dominated by extraction noise while relative
comparisons on identical input are sound. Ranking four candidates for the same
chapter position is the relative case, so the noise should cancel in the
ordering.
| condition | violations | prompt tokens |
|---|---|---|
| previous chapter, one sample | 0.354 | 1,005 |
| previous chapter, four samples chosen blind | 0.296 | 1,012 |
| previous chapter, four samples ranked by the graph | 0.300 | 1,017 |
| full transcript, one sample | 0.400 | 5,360 |
| comparison | difference | interval | excludes zero |
|---|---|---|---|
| graph ranked minus one sample | -0.054 | [-0.129, +0.025] | no |
| blind choice minus one sample | -0.058 | [-0.142, +0.025] | no |
| graph ranked minus blind choice | +0.004 | [-0.058, +0.063] | no |
The graph adds nothing over choosing blind from the same four candidates. Any apparent gain belongs to the extra sampling, and even that does not clear the interval.
Without the blind arm this would have been reported as a reduction of 0.054 from graph ranking. The control is what separates a mechanism from a budget.
3. A correction, and a limit of the benchmark
Earlier notes in this project stated that the previous chapter beats the full transcript, and separately that the full transcript is best. Both were read from single runs and neither is safe.
| condition | run 2 | run 3 | run 4 | spread |
|---|---|---|---|---|
| no memory | 0.504 | 0.487 | not run | 0.017 |
| previous chapter | 0.308 | 0.312 | 0.354 | 0.046 |
| full transcript | 0.329 | 0.254 | 0.400 | 0.146 |
The full transcript condition moves by 0.146 across runs, which is larger than almost every effect discussed in this project. Its ordering against the previous chapter is therefore not established, and the honest statement is that both beat no memory while their ranking against each other is unresolved at this sample size.
What survives the variance is the part that is large and repeated: no memory is clearly worst, and every graph conditioning design lands at or below the memory baselines rather than above them.
Implication for the benchmark. Conditions whose prompt grows without bound are unstable across seeds. Effects below roughly 0.15 on such conditions need several seeds before they are reported. Within run paired comparisons remain valid, because premises and sampling seeds are shared; cross run comparisons of a single condition are not.
4. The false positive caught in run 1
Run 1 included a LoRA fine tuned on 6,876 state conditioned examples. It scored 0.054 against 0.367 for the base model, an apparent sevenfold win. It was not real: the model had degenerated into pastiche of its training corpus, reproducing even the hard wrap indentation of Project Gutenberg, and a model that stops writing about the premise cannot violate its canon.
| base conditions | tuned conditions | |
|---|---|---|
| chapters mentioning any premise character | 1.000 | 0.765 to 0.812 |
| canon facts actively restated | 0.838 to 0.887 | 0.092 to 0.163 |
on_premise is now a first class metric. The tuned arm is excluded from every
conclusion.
5. Why the conditioning line is closed
Five designs, four runs, 18,600 chapters, each falsifier stated in advance.
Three separate controls each removed an apparent effect: on_premise removed a
sevenfold win, the blind selection arm removed a 0.054 gain, and the cross run
spread removed a baseline ordering.
The state held 0.492 of the planted canon at the moment of writing and conditioning on it still added nothing, so this is not an extraction ceiling. The serialised digest result also replicates a published negative, the Narrative World Model paper's State Memory condition, with a 3B open model.
The reading result in METHODOLOGY.md section 2 is unaffected. It compares
policies on identical input, where a shared noise floor cancels, and it holds at
three extractor sizes.