Checkpoint-matched evaluation
CNN checkpoint SHA-256: a6b944d56580b64dfb526da5b3888a137b6783ea29d8a719a0fea3f4e3f7ee76.
These are curated historical results, not a new evaluation or a claim about a later attention model. Exact counts were recovered from the checkpoint-bound historical record. Full numeric values, source-record hashes and dataset-file hashes are in evaluation.json. Original records and full banks are not included in this inference release.
| Bank | Exact supported plans | Supported rejected | Negative false accepts | Correct full decisions |
|---|---|---|---|---|
| Fresh, same training grammar | 622/625 | 3/625 | 5/575 | 1192/1200 |
| Schema + reserved phrasing transfer | 497/612 | 115/612 | 8/588 | 1077/1200 |
| AI-authored transformed bank | 40/50 | 10/50 | 13/70 | 97/120 |
Definitions
- Exact supported plans: predicted plan equals the annotated plan; denominator is supported requests, including rejected ones.
- Supported rejected: genuinely supported requests producing diagnostics.
- Negative false accepts: unsupported/invalid requests producing plans; denominator is all annotated negatives.
- Correct full decisions: exact supported plans plus correctly rejected negatives, divided by all requests. This is not acceptance rate.
- Supported fake-tool execution counts equal exact-plan counts on these three banks. Fake execution is not real-workflow reliability or isolation.
Interpretation and comparison
The authored bank has 24 bases with five variants each, including 10 supported bases. The 40/50 exact supported cases correspond to 8/10 supported bases; two core phrasings failed in every variant. The 13/70 negative false accepts include mixed ordering, unfamiliar prohibition, unknown fields and loop/sorting contexts. Correlated variants are not independent samples.
The historical phrase-rule comparator scored 503/612 supported transfer plans and 41/50 authored supported plans, slightly above this CNN's 497/612 and 40/50. There is no demonstrated general neural advantage over strong rules or a main-model baseline.
Training, validation and evaluation inputs were synthetic/AI-authored, not human task traffic. Exact normalized-input deduplication does not establish semantic independence or representative English coverage. Synthetic labels are not independently established human intent. The banks are now exposed; do not tune on them and call them new held-out tests.
No production-safety, broad-English, demographic-fairness or worst-case latency claim is made. Publication preparation checks packaging/API compatibility, not new model capability.