| # Lesson 08 RESULTS: structured output and constrained decoding (pankh-decider-v1) |
|
|
| Run (after grading fixes): 2026-10-11, Apple Silicon Mac, PyTorch 2.14.1, MPS training (GPU shared), CPU latency. papermill, `lesson_08.executed.ipynb` local. Whole notebook **96.5 s**. Seed 42. Not timed on a T4. Warm-started from the re-executed Lesson 07 v0; supersedes the first run. |
|
|
| ## Data |
| - Same gen.py sets and shape-hash split as Lesson 07 (fit 15,205, calibration 4,795, test 2,000). |
| - Real slot values: gen.py replayed with a recorder (read with ast.literal_eval, never executed); asserted byte-identical (22,000 questions). Gold objects failing the validator: 0. Unpointable training pairs: 0. |
| - Leakage: 0 identical questions, 0 shared shapes; 271/2,000 test questions also fit a loose training-template pattern (as in Lesson 07). |
| |
| ## Model |
| Warm start from v0, vocabulary 210 -> 284. 837,888 params; 1,500 steps x 64, word dropout 0.15 (markers exempt), JSON-only loss; 71.9 s on shared MPS; final loss 0.0357. Pointers (`<code0>`) are replaced by the question's real values before parsing. |
| |
| ## Stencils |
| - Catalog stencil: router.yaml structure, structural feasibility (pankhllm's slot-fit rule), names in catalog order without skipping a required one, listed enum words, pattern-fitting markers, integers in range, `}}` only when required params present. |
| - Grounded stencil: + an enum value only if it or a router.yaml alias appears in the question. |
| - Gold answers allowed: catalog 100.0%, grounded 100.0%. |
| - Validator: operation, names, enums, patterns, ranges, required params, and every string parameter must appear verbatim in the question. |
| |
| ## Numbers (held-out templates, 2,000, greedy) |
| | Decoding | Valid | Exact match | Wrong but valid | Op accuracy | |
| |---|---|---|---|---| |
| | Free | 91.6% | 47.3% | 44.4% | 84.5% | |
| | Catalog stencil | 100.0% | 50.5% | 49.5% | 84.5% | |
| | Grounded stencil | 100.0% | 71.3% | 28.7% | 90.8% | |
| |
| **CHECK (Lesson 08):** with the stencil on, validity **100.0%**; wrong-but-valid **49.5%** (catalog) and **28.7%** (grounded); free decoding 91.6% valid, 44.4% wrong but valid. |
| - The catalog stencil (pankhllm's slot-fit check plus structure) left operation accuracy unchanged (84.5% -> 84.5%): it bought validity, not precision. The lift came from grounding enum words. |
| - Exact match is literal. Filling router.yaml defaults on both sides makes 67 of 887 free, 81 of 989 catalog and 29 of 574 grounded wrong-but-valid answers equal gold; semantic wrong-but-valid: free 41.0%, catalog 45.4%, grounded 27.3%. |
| - Free failures (167): unknown param window 44, not JSON 43, unknown param benchmark 34, direction 22, grain 16. |
| - Grounded wrong-but-valid: 390 right op with wrong/missing parameter, 184 wrong op. |
| |
| ## Decisions at a threshold (operation; confidence renormalised over feasible ops with the stencil) |
| | Model | Fixed threshold | Decided | Op precision | Wrong | JSON exact among decided | Oracle 98% | |
| |---|---|---|---|---|---|---| |
| | v1 free | 0.9978 | 7.6% | 100.00% | 0 | 54.90% | 39.2% | |
| | v1 grounded stencil | 0.9777 | 72.1% | 98.54% | 21 | 77.05% | 85.4% | |
| (The first run of this notebook gave 93.0% at 93.49% for the grounded row: with renormalisation a single feasible operation gets confidence 1.0, so the result swings between runs.) |
| |
| ## 14 benchmark questions (grounded) |
| Op correct 11/14; decided 12/14, 2 wrong. "Why did calls fall in T-112 in 2026-08?" -> kpi_value 0.996, decided and wrong. "revenue of T-401" -> prescriber_list (0.512); "Which 10 territories... most revenue" -> UNSUPPORTED (1.000): revenue is not a catalog metric. |
| |
| ## Latency (CPU, batch 1, 300 questions) |
| Full JSON, grounded: p50 5.993 ms, p95 9.345 ms, 10.7 tokens (~1,785 tokens/s, no KV cache). Op only: p50 0.873 ms, p95 1.001 ms. 3.3 MB fp32. |
| |
| ## History |
| First drafts: wrong replay order, unquoted op names, a mislabelled reason count and a grammar dead end, all fixed before the first full run. After grading: widget gained a pointer-substitution line and hides the final JSON until the last step; the header notes T4 not timed; defaults-adjusted scoring added; re-executed. |
| - After this re-execution, one display-only fix was applied to the widget JavaScript in both the clean and the executed notebook without re-running (re-running would change every number down the chain): operation names are now quoted in the "after pointer substitution" line. No computation changed. |
| |
