Creative-bench methodology

Writing-voice evidence for two model files. Creative generations run at temperature 0.7: capability discipline (greedy, deterministic) would strangle voice; writing needs room. Seed fixed per file so runs are reproducible short of sampler noise.

Prompts

32 prompts from EQ-Bench Creative Writing v3 (creative-bench/prompts.json, vendored from EQ-bench/creative-writing-bench, used with citation). One generation per prompt per file, 6000-token budget (generous on purpose: every char is data), user turn only — some models support no system role.

What is measured (no judge model)

Shared trigram vocabulary per prompt (Gryphe-style): size of trigram-set intersection over union, file A vs. file B. Lower means more distinct voices. Reported as mean and max over paired prompts.
Cliche density: substring hits per 100 words against a plain-text list (audition/cliches-starter.txt documents its own provenance; bring your own list for serious work).
Texture stats: lengths, sentence rhythm, lexical diversity, top-trigram fingerprint per file.
Blind pairs: every prompt exported with both texts labeled A/B only. The reader judges; nothing is scored.

What is archived

Per file, creative-bench/runs/file-<x>-creative.jsonl: prompt metadata, full response text, full reasoning trace, finish reason, temp/seed/budget. No personal information: server model ids are replaced with run labels before saving.

Reproduce

BENCH_LABEL=file-c python3 creative-bench/generate.py \
    --prompts creative-bench/prompts.json \
    --out creative-bench/runs/file-c-creative.jsonl
python3 audition/voice-taste.py creative-bench/runs/file-a-creative.jsonl \
    creative-bench/runs/file-b-creative.jsonl --out taste-out