EVALUATION.md

# Checkpoint-matched evaluation

CNN checkpoint SHA-256: `a6b944d56580b64dfb526da5b3888a137b6783ea29d8a719a0fea3f4e3f7ee76`.

These are curated historical results, not a new evaluation or a claim about a later attention model. Exact counts were recovered from the checkpoint-bound historical record. Full numeric values, source-record hashes and dataset-file hashes are in [evaluation.json](evaluation.json). Original records and full banks are not included in this inference release.

| Bank | Exact supported plans | Supported rejected | Negative false accepts | Correct full decisions |
| --- | ---: | ---: | ---: | ---: |
| Fresh, same training grammar | 622/625 | 3/625 | 5/575 | 1192/1200 |
| Schema + reserved phrasing transfer | 497/612 | 115/612 | 8/588 | 1077/1200 |
| AI-authored transformed bank | 40/50 | 10/50 | 13/70 | 97/120 |

## Definitions

- **Exact supported plans:** predicted plan equals the annotated plan; denominator is supported requests, including rejected ones.
- **Supported rejected:** genuinely supported requests producing diagnostics.
- **Negative false accepts:** unsupported/invalid requests producing plans; denominator is all annotated negatives.
- **Correct full decisions:** exact supported plans plus correctly rejected negatives, divided by all requests. This is not acceptance rate.
- Supported fake-tool execution counts equal exact-plan counts on these three banks. Fake execution is not real-workflow reliability or isolation.

## Interpretation and comparison

The authored bank has 24 bases with five variants each, including 10 supported bases. The 40/50 exact supported cases correspond to 8/10 supported bases; two core phrasings failed in every variant. The 13/70 negative false accepts include mixed ordering, unfamiliar prohibition, unknown fields and loop/sorting contexts. Correlated variants are not independent samples.

The historical phrase-rule comparator scored 503/612 supported transfer plans and 41/50 authored supported plans, slightly above this CNN's 497/612 and 40/50. There is no demonstrated general neural advantage over strong rules or a main-model baseline.

Training, validation and evaluation inputs were synthetic/AI-authored, not human task traffic. Exact normalized-input deduplication does not establish semantic independence or representative English coverage. Synthetic labels are not independently established human intent. The banks are now exposed; do not tune on them and call them new held-out tests.

No production-safety, broad-English, demographic-fairness or worst-case latency claim is made. Publication preparation checks packaging/API compatibility, not new model capability.