Spaces:
Running
Running
| # Impact pipeline against the gap key | |
| Model `claude-haiku-5-5`; 12 circulars, every one named in [`company/GAP_KEY.md`](https://github.com/nihardamera/RegRadar/blob/main/company/GAP_KEY.md). Generated by `python -m regradar.evaluate_impact score` from `eval/impact/*.json`. API cost of the run: $0.65. | |
| | Measure | Result | | |
| | --- | --- | | |
| | Planted gaps found, strict: the flag states the rule's key fact | **11/12** = 92% [65, 99] (Wilson 95%) | | |
| | Planted gaps found, lenient: a flag cites the right policy section | 12/12 = 100% [76, 100] | | |
| | Traps flagged as gaps (N1-N3; lower is better), strict / lenient | **2/3** / 2/3 | | |
| | Gap or partial flags in all (share of applicable obligations) | 328 (29%) | | |
| | Flags matching no key row (need a person) | 234 | | |
| | Obligations extracted / rated not applicable / not covered | 1492 / 369 / 660 | | |
| | Items in the review queue | 72 | | |
| | Reports failing a code check | 0 | | |
| Per key row (Y = found or, for N rows, flagged): | |
| | Match | G1 | G2 | G3 | G4 | G5 | G6 | G7 | G8 | G9 | G10 | G11 | G12 | N1 | N2 | N3 | | |
| | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | |
| | strict | Y | Y | Y | Y | Y | - | Y | Y | Y | Y | Y | Y | Y | - | Y | | |
| | lenient | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | Y | - | Y | | |
| Matching: a gap or partial rating from the key's circular that cites the key's policy in the same section (heading number); strict also requires the finding or action to state the rule's key fact (`KEY_FACT` in `regradar/evaluate_impact.py`, taken from the key's rule text). **Recall is not the whole story:** the pipeline flags many obligations, and precision is not measured. The key lists what was planted, not every shortfall, so unkeyed flags are neither right nor wrong until a person reviews them; they are listed in `impact_<model>.json`. | |