| # Creative-bench methodology |
|
|
| Writing-voice evidence for two model files. Creative generations run at |
| temperature 0.7: capability discipline (greedy, deterministic) would |
| strangle voice; writing needs room. Seed fixed per file so runs are |
| reproducible short of sampler noise. |
| ## Prompts |
|
|
| 32 prompts from EQ-Bench Creative Writing v3 |
| (`creative-bench/prompts.json`, vendored from |
| [EQ-bench/creative-writing-bench](https://github.com/EQ-bench/creative-writing-bench), |
| used with citation). One generation per prompt per file, 6000-token budget |
| (generous on purpose: every char is data), user turn only — some models |
| support no system role. |
|
|
| ## What is measured (no judge model) |
|
|
| - **Shared trigram vocabulary** per prompt (Gryphe-style): size of |
| trigram-set intersection over union, file A vs. file B. Lower means more |
| distinct voices. Reported as mean and max over paired prompts. |
| - **Cliche density**: substring hits per 100 words against a plain-text |
| list (`audition/cliches-starter.txt` documents its own provenance; |
| bring your own list for serious work). |
| - **Texture stats**: lengths, sentence rhythm, lexical diversity, |
| top-trigram fingerprint per file. |
| - **Blind pairs**: every prompt exported with both texts labeled A/B only. |
| The reader judges; nothing is scored. |
|
|
| ## What is archived |
|
|
| Per file, `creative-bench/runs/file-<x>-creative.jsonl`: prompt metadata, |
| full response text, full reasoning trace, finish reason, temp/seed/budget. |
| No personal information: server model ids are replaced with run labels |
| before saving. |
|
|
| ## Reproduce |
|
|
| ```bash |
| BENCH_LABEL=file-c python3 creative-bench/generate.py \ |
| --prompts creative-bench/prompts.json \ |
| --out creative-bench/runs/file-c-creative.jsonl |
| python3 audition/voice-taste.py creative-bench/runs/file-a-creative.jsonl \ |
| creative-bench/runs/file-b-creative.jsonl --out taste-out |
| ``` |
|
|
