# Generation findings

Four runs, 18,600 generated chapters, 30 stories, conditions paired within
story. Canon is planted in chapter 1 and withheld afterwards; violations are
deterministic string tests near a mention of the subject. No language model
appears in any metric.

**Summary. Five ways of using a narrative graph at generation time were tested.
None of them helps. The fifth, ranking candidates, is the one the project's own
negative results predicted should work, and it does not either.**

## 1. What was tried

| use of the graph | design | outcome |
|---|---|---|
| carried as context | serialised state digest in the prompt | worse than no memory |
| carried plus recent text | digest and the previous chapter | loses to the previous chapter alone |
| queried by the next beat | retrieve only what the beat implicates | worse than no memory |
| queried plus recent text | retrieval and the previous chapter | ties the previous chapter alone |
| used to rank candidates | draw four, pick the most consistent | indistinguishable from picking blind |

## 2. The ranking result, and why the control mattered

Ranking was motivated by the finding in `METHODOLOGY.md` section 5 that absolute
contradiction counts are dominated by extraction noise while relative
comparisons on identical input are sound. Ranking four candidates for the same
chapter position is the relative case, so the noise should cancel in the
ordering.

| condition | violations | prompt tokens |
|---|---|---|
| previous chapter, one sample | 0.354 | 1,005 |
| previous chapter, four samples chosen blind | 0.296 | 1,012 |
| previous chapter, four samples ranked by the graph | 0.300 | 1,017 |
| full transcript, one sample | 0.400 | 5,360 |

| comparison | difference | interval | excludes zero |
|---|---|---|---|
| graph ranked minus one sample | -0.054 | [-0.129, +0.025] | no |
| blind choice minus one sample | -0.058 | [-0.142, +0.025] | no |
| **graph ranked minus blind choice** | **+0.004** | [-0.058, +0.063] | **no** |

The graph adds nothing over choosing blind from the same four candidates. Any
apparent gain belongs to the extra sampling, and even that does not clear the
interval.

Without the blind arm this would have been reported as a reduction of 0.054
from graph ranking. The control is what separates a mechanism from a budget.

## 3. A correction, and a limit of the benchmark

Earlier notes in this project stated that the previous chapter beats the full
transcript, and separately that the full transcript is best. Both were read from
single runs and neither is safe.

| condition | run 2 | run 3 | run 4 | spread |
|---|---|---|---|---|
| no memory | 0.504 | 0.487 | not run | 0.017 |
| previous chapter | 0.308 | 0.312 | 0.354 | 0.046 |
| **full transcript** | 0.329 | 0.254 | 0.400 | **0.146** |

The full transcript condition moves by 0.146 across runs, which is larger than
almost every effect discussed in this project. Its ordering against the previous
chapter is therefore **not established**, and the honest statement is that both
beat no memory while their ranking against each other is unresolved at this
sample size.

What survives the variance is the part that is large and repeated: no memory is
clearly worst, and every graph conditioning design lands at or below the memory
baselines rather than above them.

**Implication for the benchmark.** Conditions whose prompt grows without bound
are unstable across seeds. Effects below roughly 0.15 on such conditions need
several seeds before they are reported. Within run paired comparisons remain
valid, because premises and sampling seeds are shared; cross run comparisons of
a single condition are not.

## 4. The false positive caught in run 1

Run 1 included a LoRA fine tuned on 6,876 state conditioned examples. It scored
0.054 against 0.367 for the base model, an apparent sevenfold win. It was not
real: the model had degenerated into pastiche of its training corpus,
reproducing even the hard wrap indentation of Project Gutenberg, and a model
that stops writing about the premise cannot violate its canon.

| | base conditions | tuned conditions |
|---|---|---|
| chapters mentioning any premise character | 1.000 | 0.765 to 0.812 |
| canon facts actively restated | 0.838 to 0.887 | 0.092 to 0.163 |

`on_premise` is now a first class metric. The tuned arm is excluded from every
conclusion.

## 5. Why the conditioning line is closed

Five designs, four runs, 18,600 chapters, each falsifier stated in advance.
Three separate controls each removed an apparent effect: `on_premise` removed a
sevenfold win, the blind selection arm removed a 0.054 gain, and the cross run
spread removed a baseline ordering.

The state held 0.492 of the planted canon at the moment of writing and
conditioning on it still added nothing, so this is not an extraction ceiling.
The serialised digest result also replicates a published negative, the Narrative
World Model paper's State Memory condition, with a 3B open model.

The reading result in `METHODOLOGY.md` section 2 is unaffected. It compares
policies on identical input, where a shared noise floor cancels, and it holds at
three extractor sizes.