Lesson 08 RESULTS: structured output and constrained decoding (pankh-decider-v1)
Run (after grading fixes): 2026-10-11, Apple Silicon Mac, PyTorch 2.14.1, MPS training (GPU shared), CPU latency. papermill, lesson_08.executed.ipynb local. Whole notebook 96.5 s. Seed 42. Not timed on a T4. Warm-started from the re-executed Lesson 07 v0; supersedes the first run.
Data
- Same gen.py sets and shape-hash split as Lesson 07 (fit 15,205, calibration 4,795, test 2,000).
- Real slot values: gen.py replayed with a recorder (read with ast.literal_eval, never executed); asserted byte-identical (22,000 questions). Gold objects failing the validator: 0. Unpointable training pairs: 0.
- Leakage: 0 identical questions, 0 shared shapes; 271/2,000 test questions also fit a loose training-template pattern (as in Lesson 07).
Model
Warm start from v0, vocabulary 210 -> 284. 837,888 params; 1,500 steps x 64, word dropout 0.15 (markers exempt), JSON-only loss; 71.9 s on shared MPS; final loss 0.0357. Pointers (<code0>) are replaced by the question's real values before parsing.
Stencils
- Catalog stencil: router.yaml structure, structural feasibility (pankhllm's slot-fit rule), names in catalog order without skipping a required one, listed enum words, pattern-fitting markers, integers in range,
}}only when required params present. - Grounded stencil: + an enum value only if it or a router.yaml alias appears in the question.
- Gold answers allowed: catalog 100.0%, grounded 100.0%.
- Validator: operation, names, enums, patterns, ranges, required params, and every string parameter must appear verbatim in the question.
Numbers (held-out templates, 2,000, greedy)
| Decoding | Valid | Exact match | Wrong but valid | Op accuracy |
|---|---|---|---|---|
| Free | 91.6% | 47.3% | 44.4% | 84.5% |
| Catalog stencil | 100.0% | 50.5% | 49.5% | 84.5% |
| Grounded stencil | 100.0% | 71.3% | 28.7% | 90.8% |
CHECK (Lesson 08): with the stencil on, validity 100.0%; wrong-but-valid 49.5% (catalog) and 28.7% (grounded); free decoding 91.6% valid, 44.4% wrong but valid.
- The catalog stencil (pankhllm's slot-fit check plus structure) left operation accuracy unchanged (84.5% -> 84.5%): it bought validity, not precision. The lift came from grounding enum words.
- Exact match is literal. Filling router.yaml defaults on both sides makes 67 of 887 free, 81 of 989 catalog and 29 of 574 grounded wrong-but-valid answers equal gold; semantic wrong-but-valid: free 41.0%, catalog 45.4%, grounded 27.3%.
- Free failures (167): unknown param window 44, not JSON 43, unknown param benchmark 34, direction 22, grain 16.
- Grounded wrong-but-valid: 390 right op with wrong/missing parameter, 184 wrong op.
Decisions at a threshold (operation; confidence renormalised over feasible ops with the stencil)
| Model | Fixed threshold | Decided | Op precision | Wrong | JSON exact among decided | Oracle 98% |
|---|---|---|---|---|---|---|
| v1 free | 0.9978 | 7.6% | 100.00% | 0 | 54.90% | 39.2% |
| v1 grounded stencil | 0.9777 | 72.1% | 98.54% | 21 | 77.05% | 85.4% |
| (The first run of this notebook gave 93.0% at 93.49% for the grounded row: with renormalisation a single feasible operation gets confidence 1.0, so the result swings between runs.) |
14 benchmark questions (grounded)
Op correct 11/14; decided 12/14, 2 wrong. "Why did calls fall in T-112 in 2026-08?" -> kpi_value 0.996, decided and wrong. "revenue of T-401" -> prescriber_list (0.512); "Which 10 territories... most revenue" -> UNSUPPORTED (1.000): revenue is not a catalog metric.
Latency (CPU, batch 1, 300 questions)
Full JSON, grounded: p50 5.993 ms, p95 9.345 ms, 10.7 tokens (~1,785 tokens/s, no KV cache). Op only: p50 0.873 ms, p95 1.001 ms. 3.3 MB fp32.
History
First drafts: wrong replay order, unquoted op names, a mislabelled reason count and a grammar dead end, all fixed before the first full run. After grading: widget gained a pointer-substitution line and hides the final JSON until the last step; the header notes T4 not timed; defaults-adjusted scoring added; re-executed.
- After this re-execution, one display-only fix was applied to the widget JavaScript in both the clean and the executed notebook without re-running (re-running would change every number down the chain): operation names are now quoted in the "after pointer substitution" line. No computation changed.