Commit History
-
[NOTICKET] feat: assert on invariants, report a band, classify the genre (Steps 0 + 5)
Rifqi Hafizuddin +1: Claude Opus 5Step 0 -- measurement hardening. `temperature=0` is argmax over logits a shared inference server does not hand back identically twice, and this corpus has a recorded case of identical input scoring 6/7 one day and 0/12 the next. Three changes follow from that: * `eval/knowledge/invariants.py` asserts on FIELDS AND INVARIANTS, never on exact text: a span that cannot be located verbatim, a missing or duplicate entity id, a rejected field that still carries a value (rejected-never-repaired is only checkable from outside), a page out of range, queue ranks that are not contiguous, a leaked sort key. A violation means a control leaked, so it fails the run (exit 5) regardless of how the scores looked. * `--runs N` repeats ONLY the paid stage -- the free stages are deterministic -- and reports min/median/max with the MEDIAN as the value, because one failed run drags a mean into a region no run occupied. It also checks every run minted identical entity ids, which is the K11…f1d64b1