Spaces:
Running
Impact pipeline against the gap key
Model claude-haiku-5-5; 12 circulars, every one named in company/GAP_KEY.md. Generated by python -m regradar.evaluate_impact score from eval/impact/*.json. API cost of the run: $0.65.
| Measure | Result |
|---|---|
| Planted gaps found, strict: the flag states the rule's key fact | 11/12 = 92% [65, 99] (Wilson 95%) |
| Planted gaps found, lenient: a flag cites the right policy section | 12/12 = 100% [76, 100] |
| Traps flagged as gaps (N1-N3; lower is better), strict / lenient | 2/3 / 2/3 |
| Gap or partial flags in all (share of applicable obligations) | 328 (29%) |
| Flags matching no key row (need a person) | 234 |
| Obligations extracted / rated not applicable / not covered | 1492 / 369 / 660 |
| Items in the review queue | 72 |
| Reports failing a code check | 0 |
Per key row (Y = found or, for N rows, flagged):
| Match | G1 | G2 | G3 | G4 | G5 | G6 | G7 | G8 | G9 | G10 | G11 | G12 | N1 | N2 | N3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| strict | Y | Y | Y | Y | Y | - | Y | Y | Y | Y | Y | Y | Y | - | Y |
| lenient | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | - | Y |
Matching: a gap or partial rating from the key's circular that cites the key's policy in the same section (heading number); strict also requires the finding or action to state the rule's key fact (KEY_FACT in regradar/evaluate_impact.py, taken from the key's rule text). Recall is not the whole story: the pipeline flags many obligations, and precision is not measured. The key lists what was planted, not every shortfall, so unkeyed flags are neither right nor wrong until a person reviews them; they are listed in impact_<model>.json.