File size: 1,858 Bytes
9f5fbb5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | # Creative-bench methodology
Writing-voice evidence for two model files. Creative generations run at
temperature 0.7: capability discipline (greedy, deterministic) would
strangle voice; writing needs room. Seed fixed per file so runs are
reproducible short of sampler noise.
## Prompts
32 prompts from EQ-Bench Creative Writing v3
(`creative-bench/prompts.json`, vendored from
[EQ-bench/creative-writing-bench](https://github.com/EQ-bench/creative-writing-bench),
used with citation). One generation per prompt per file, 6000-token budget
(generous on purpose: every char is data), user turn only — some models
support no system role.
## What is measured (no judge model)
- **Shared trigram vocabulary** per prompt (Gryphe-style): size of
trigram-set intersection over union, file A vs. file B. Lower means more
distinct voices. Reported as mean and max over paired prompts.
- **Cliche density**: substring hits per 100 words against a plain-text
list (`audition/cliches-starter.txt` documents its own provenance;
bring your own list for serious work).
- **Texture stats**: lengths, sentence rhythm, lexical diversity,
top-trigram fingerprint per file.
- **Blind pairs**: every prompt exported with both texts labeled A/B only.
The reader judges; nothing is scored.
## What is archived
Per file, `creative-bench/runs/file-<x>-creative.jsonl`: prompt metadata,
full response text, full reasoning trace, finish reason, temp/seed/budget.
No personal information: server model ids are replaced with run labels
before saving.
## Reproduce
```bash
BENCH_LABEL=file-c python3 creative-bench/generate.py \
--prompts creative-bench/prompts.json \
--out creative-bench/runs/file-c-creative.jsonl
python3 audition/voice-taste.py creative-bench/runs/file-a-creative.jsonl \
creative-bench/runs/file-b-creative.jsonl --out taste-out
```
|