data/writeup1.md

Judge calibration: held-out results

Judge: claude-haiku-5-5, effort low, prompt 2 (regradar/judge.py). Scored once (eval/judge/heldout.lock). Written by python -m regradar.judge summary from calibration_heldout.json and eval/human/judge_labels.jsonl.

Cases	n	Label	Agreement	Cohen's kappa [95% CI]	Human yes	Judge yes
all labelled	30	correct	93.3 [83.3, 100.0]	0.76 [0.32, 1.00]	83%	83%
all labelled	30	faithful	83.3 [70.0, 96.7]	not defined (no human 'no' labels)	100%	83%
natural sample	28	correct	96.4 [89.3, 100.0]	0.84 [0.00, 1.00]	86%	89%
natural sample	28	faithful	89.3 [75.0, 100.0]	not defined (no human 'no' labels)	100%	89%

Planted errors marked unfaithful: 15/18 = 83.3% [66.7, 100.0]; by kind condition 1/3, entity 5/6, negation 5/5, number 3/3, wrong_chunk 1/1. Missed: q039condition, q070condition, q120~entity.