How the judging was run
ts300.py sample writes judge_packet_partN.md files (10 story pairs each, A/B order randomized and balanced,
grading rubric included in every packet) and a separate judge_key.json.
Each packet was graded by a fresh Claude Opus 5.5 instance with no other context, run as a Claude Code subagent with this prompt (paths adjusted per packet). The packets were copied to a folder without the key first.
You are acting as a blind judge for a story-grading task. Read exactly one file: . Follow the grading instructions inside it exactly, and grade every item in it yourself; don't write or run any scripts to grade. Write your complete reply (the notes for each item, then the CSV block with the exact header the file specifies) to using the Write tool. Do not open, list, search or run anything else: no other files or directories and no shell commands. In particular, never look for or open any answer key. When finished, reply with only the word: written
Pasting each packet into a new chat with any capable LLM works the same way. Concatenate the replies in part
order into judge_replies.txt, then:
python ts300.py score --key judge_key.json --judge judge_replies.txt
python paired_stats.py judge_key.json judge_replies.txt