| # Checkpoint-matched evaluation |
|
|
| CNN checkpoint SHA-256: `a6b944d56580b64dfb526da5b3888a137b6783ea29d8a719a0fea3f4e3f7ee76`. |
|
|
| These are curated historical results, not a new evaluation or a claim about a later attention model. Exact counts were recovered from the checkpoint-bound historical record. Full numeric values, source-record hashes and dataset-file hashes are in [evaluation.json](evaluation.json). Original records and full banks are not included in this inference release. |
|
|
| | Bank | Exact supported plans | Supported rejected | Negative false accepts | Correct full decisions | |
| | --- | ---: | ---: | ---: | ---: | |
| | Fresh, same training grammar | 622/625 | 3/625 | 5/575 | 1192/1200 | |
| | Schema + reserved phrasing transfer | 497/612 | 115/612 | 8/588 | 1077/1200 | |
| | AI-authored transformed bank | 40/50 | 10/50 | 13/70 | 97/120 | |
|
|
| ## Definitions |
|
|
| - **Exact supported plans:** predicted plan equals the annotated plan; denominator is supported requests, including rejected ones. |
| - **Supported rejected:** genuinely supported requests producing diagnostics. |
| - **Negative false accepts:** unsupported/invalid requests producing plans; denominator is all annotated negatives. |
| - **Correct full decisions:** exact supported plans plus correctly rejected negatives, divided by all requests. This is not acceptance rate. |
| - Supported fake-tool execution counts equal exact-plan counts on these three banks. Fake execution is not real-workflow reliability or isolation. |
|
|
| ## Interpretation and comparison |
|
|
| The authored bank has 24 bases with five variants each, including 10 supported bases. The 40/50 exact supported cases correspond to 8/10 supported bases; two core phrasings failed in every variant. The 13/70 negative false accepts include mixed ordering, unfamiliar prohibition, unknown fields and loop/sorting contexts. Correlated variants are not independent samples. |
|
|
| The historical phrase-rule comparator scored 503/612 supported transfer plans and 41/50 authored supported plans, slightly above this CNN's 497/612 and 40/50. There is no demonstrated general neural advantage over strong rules or a main-model baseline. |
|
|
| Training, validation and evaluation inputs were synthetic/AI-authored, not human task traffic. Exact normalized-input deduplication does not establish semantic independence or representative English coverage. Synthetic labels are not independently established human intent. The banks are now exposed; do not tune on them and call them new held-out tests. |
|
|
| No production-safety, broad-English, demographic-fairness or worst-case latency claim is made. Publication preparation checks packaging/API compatibility, not new model capability. |
|
|
