code/JUDGING.md

# How the judging was run

`ts300.py sample` writes `judge_packet_partN.md` files (10 story pairs each, A/B order randomized and balanced,
grading rubric included in every packet) and a separate `judge_key.json`.

Each packet was graded by a fresh Claude Opus 5.5 instance with no other context, run as a Claude Code subagent
with this prompt (paths adjusted per packet). The packets were copied to a folder without the key first.

> You are acting as a blind judge for a story-grading task. Read exactly one file: <packet path>. Follow the
> grading instructions inside it exactly, and grade every item in it yourself; don't write or run any scripts
> to grade. Write your complete reply (the notes for each item, then the CSV block with the exact header the
> file specifies) to <reply path> using the Write tool. Do not open, list, search or run anything else: no
> other files or directories and no shell commands. In particular, never look for or open any answer key.
> When finished, reply with only the word: written

Pasting each packet into a new chat with any capable LLM works the same way. Concatenate the replies in part
order into `judge_replies.txt`, then:

    python ts300.py score --key judge_key.json --judge judge_replies.txt
    python paired_stats.py judge_key.json judge_replies.txt