Add RESULTS.md

RESULTS.md

@@ -1,53 +1,47 @@
 # Lesson 08 RESULTS: structured output and constrained decoding (pankh-decider-v1)
 
-Run: 2026-10-11, Apple M4 Max, PyTorch 2.14.1, training on `mps`, latency on CPU. Executed end to end with papermill (`lesson_08.executed.ipynb`, local). Whole notebook: **34.8 s** (papermill wall clock 36 s). Seed 42.
+Run (after grading fixes): 2026-10-11, Apple M4 Max, PyTorch 2.14.1, MPS training (GPU shared), CPU latency. papermill, `lesson_08.executed.ipynb` local. Whole notebook **96.5 s**. Seed 42. Not timed on a T4. Warm-started from the re-executed Lesson 07 v0; supersedes the first run.
 
 ## Data
-- Same pankhllm gen.py sets as Lesson 07 (20,000 training-template questions, 2,000 held-out-template questions), same shape-hash split: fit 15,205, calibration 4,795.
-- Real slot values: the notebook replays gen.py's recipe with a recorder (gen.py read with `ast.literal_eval`, never executed) and asserts the 22,000 questions are byte-identical to gen.py's own output. They were.
-- Gold JSON: recorded surface words mapped to catalog values (router.yaml aliases plus explicit tables in the notebook); parameters the operation lacks are dropped (for example the time window in ranking questions). `direction` is set only when the question states it ({t}, {b} or a literal "top"). Gold objects failing the catalog validator: 0. Training pairs whose values could not be pointed at: 0.
-- Leakage: identical questions across train/test 0; shared shapes 0 (Lesson 07 check, repeated here); 271/2,000 test questions also fit a loose training-template pattern (same as Lesson 07).
+- Same gen.py sets and shape-hash split as Lesson 07 (fit 15,205, calibration 4,795, test 2,000).
+- Real slot values: gen.py replayed with a recorder (read with ast.literal_eval, never executed); asserted byte-identical (22,000 questions). Gold objects failing the validator: 0. Unpointable training pairs: 0.
+- Leakage: 0 identical questions, 0 shared shapes; 271/2,000 test questions also fit a loose training-template pattern (as in Lesson 07).
 
 ## Model
-- Warm start from pankh-decider-v0 (Lesson 07): all transformer weights kept; vocabulary grown 210 -> 284 tokens (numbered slot markers `<code0>`..., JSON pieces, one key token per parameter, one token per enum value, `<eos>`). New markers initialised from v0's unnumbered markers. The notebook tried the Hub first; the v0 repo is private, so it fell back to the local folder (RepositoryNotFoundError, expected).
-- 837,888 parameters. 1,500 steps x batch 64, word dropout 0.15 on non-marker input words, loss on the JSON tokens only. 20.5 s on MPS. Final loss 0.0359. Longest target 19 tokens.
-- Values are kept real by pointers: the model writes `<code0>`, the decoder copies the question's first code back in before parsing.
+Warm start from v0, vocabulary 210 -> 284. 837,888 params; 1,500 steps x 64, word dropout 0.15 (markers exempt), JSON-only loss; 71.9 s on shared MPS; final loss 0.0357. Pointers (`<code0>`) are replaced by the question's real values before parsing.
 
-## Stencils (constrained decoding)
-- Catalog stencil: prefix-validity grammar from router.yaml: `{"op":` -> structurally feasible operation -> `,"params":{` -> parameter names of that operation in catalog order (a required one may not be skipped) -> legal values (listed enum words; markers whose value fits the pattern; integer markers inside the range) -> `}}` only when required parameters are present -> `<eos>`.
-- Grounded stencil: as above, plus an enum value is allowed only if it or one of its router.yaml aliases is written in the question (a plural "s" may be missing).
-- Gold answers the stencil allows on the 2,000 held-out questions: catalog 100.0%, grounded 100.0%.
+## Stencils
+- Catalog stencil: router.yaml structure, structural feasibility (pankhllm's slot-fit rule), names in catalog order without skipping a required one, listed enum words, pattern-fitting markers, integers in range, `}}` only when required params present.
+- Grounded stencil: + an enum value only if it or a router.yaml alias appears in the question.
+- Gold answers allowed: catalog 100.0%, grounded 100.0%.
+- Validator: operation, names, enums, patterns, ranges, required params, and every string parameter must appear verbatim in the question.
 
-## Numbers (held-out templates, 2,000 questions, greedy decoding)
-| Decoding | Valid against catalog | Exact match (op + params) | Wrong but valid | Op accuracy |
+## Numbers (held-out templates, 2,000, greedy)
+| Decoding | Valid | Exact match | Wrong but valid | Op accuracy |
 |---|---|---|---|---|
-| Free | 91.1% | 47.9% | 43.2% | 83.8% |
-| Catalog stencil | 100.0% | 51.1% | 48.9% | 83.8% |
-| Grounded stencil | 100.0% | 74.7% | 25.3% | 89.6% |
-
-**CHECK (Lesson 08):** with the stencil on, validity is **100.0%**; the wrong-but-valid rate is **48.9%** with the catalog-only stencil and **25.3%** with the grounded stencil (free decoding: 91.1% valid, 43.2% wrong but valid).
-
-Why free outputs failed (178 of 2,000): unknown param `benchmark` 64, not JSON 64, unknown param `direction` 18, unknown param `compare` 11, missing `compare` 8.
-Wrong-but-valid under the grounded stencil: 298 right operation with a wrong or missing parameter, 208 wrong operation. The commonest: omitting `direction: bottom`, omitting `window`/`product`/`n`, and calling a KPI question UNSUPPORTED.
-
-## Decisions at a threshold (operation only)
-Fixed threshold chosen on the calibration shapes (lowest holding >= 98%); oracle = swept on the test set.
-| Model | Fixed threshold | Decided @ fixed | Op precision @ fixed | Wrong | JSON exact among decided | Decided @ oracle 98% |
+| Free | 91.6% | 47.3% | 44.4% | 84.5% |
+| Catalog stencil | 100.0% | 50.5% | 49.5% | 84.5% |
+| Grounded stencil | 100.0% | 71.3% | 28.7% | 90.8% |
+
+**CHECK (Lesson 08):** with the stencil on, validity **100.0%**; wrong-but-valid **49.5%** (catalog) and **28.7%** (grounded); free decoding 91.6% valid, 44.4% wrong but valid.
+- The catalog stencil (pankhllm's slot-fit check plus structure) left operation accuracy unchanged (84.5% -> 84.5%): it bought validity, not precision. The lift came from grounding enum words.
+- Exact match is literal. Filling router.yaml defaults on both sides makes 67 of 887 free, 81 of 989 catalog and 29 of 574 grounded wrong-but-valid answers equal gold; semantic wrong-but-valid: free 41.0%, catalog 45.4%, grounded 27.3%.
+- Free failures (167): unknown param window 44, not JSON 43, unknown param benchmark 34, direction 22, grain 16.
+- Grounded wrong-but-valid: 390 right op with wrong/missing parameter, 184 wrong op.
+
+## Decisions at a threshold (operation; confidence renormalised over feasible ops with the stencil)
+| Model | Fixed threshold | Decided | Op precision | Wrong | JSON exact among decided | Oracle 98% |
 |---|---|---|---|---|---|---|
-| v1 free | 0.9950 | 25.4% | 100.00% | 0 | 71.40% | 40.2% (100%) |
-| v1 grounded stencil | 0.6851 | 93.0% | 93.49% | 121 | 78.00% | 80.8% (98.02%) |
-With the stencil, the confidence is renormalised over feasible operations only, so a question where only UNSUPPORTED is feasible gets confidence 1.0. The calibration shapes are easy, so the fixed threshold came out low (0.685) and did not hold 98% on held-out templates.
-
-## 14 benchmark questions (operation mapped as in Lesson 07; grounded stencil)
-Operation correct 11/14; decided at the fixed threshold 13/14, 2 wrong. The grounded stencil makes "revenue" questions infeasible for the KPI operations ("revenue" is not a metric in the pharma catalog): "Which 10 territories had the most revenue" -> UNSUPPORTED (1.000), "revenue of T-401" -> coverage_owner (0.401). "Why did calls fall in T-112 in 2026-08?" -> kpi_value (0.996), wrong.
+| v1 free | 0.9978 | 7.6% | 100.00% | 0 | 54.90% | 39.2% |
+| v1 grounded stencil | 0.9777 | 72.1% | 98.54% | 21 | 77.05% | 85.4% |
+(The first run of this notebook gave 93.0% at 93.49% for the grounded row: with renormalisation a single feasible operation gets confidence 1.0, so the result swings between runs.)
 
-## Latency (CPU, batch 1, 300 held-out questions)
-- Full JSON with the grounded stencil: p50 5.732 ms, p95 8.871 ms; 10.9 tokens per answer -> about 1,898 tokens/s at p50 (no KV cache; each step re-reads the prefix).
-- Operation only (one forward pass + feasibility check): p50 0.813 ms, p95 0.921 ms.
-- Model file 3.3 MB fp32.
+## 14 benchmark questions (grounded)
+Op correct 11/14; decided 12/14, 2 wrong. "Why did calls fall in T-112 in 2026-08?" -> kpi_value 0.996, decided and wrong. "revenue of T-401" -> prescriber_list (0.512); "Which 10 territories... most revenue" -> UNSUPPORTED (1.000): revenue is not a catalog metric.
 
-## What went wrong along the way
-- First run: replayed gen.py draws in the wrong order (assertion caught it; fixed). Second: operation names were not quoted in the JSON (0% valid; fixed). Third: the validator labelled valid UNSUPPORTED outputs with a failure reason (counts only) and the grammar could dead-end by skipping a required key; both fixed, grounded stencil added. Numbers above are from the final run only.
-- Each re-run on MPS gives slightly different numbers (training is not bit-exact).
+## Latency (CPU, batch 1, 300 questions)
+Full JSON, grounded: p50 5.993 ms, p95 9.345 ms, 10.7 tokens (~1,785 tokens/s, no KV cache). Op only: p50 0.873 ms, p95 1.001 ms. 3.3 MB fp32.
 
-- After execution, one self-check question in the closing markdown cell was reworded in both the clean and the executed notebook, without re-running. No code changed.
+## History
+First drafts: wrong replay order, unquoted op names, a mislabelled reason count and a grammar dead end, all fixed before the first full run. After grading: widget gained a pointer-substitution line and hides the final JSON until the last step; the header notes T4 not timed; defaults-adjusted scoring added; re-executed.
+- After this re-execution, one display-only fix was applied to the widget JavaScript in both the clean and the executed notebook without re-running (re-running would change every number down the chain): operation names are now quoted in the "after pointer substitution" line. No computation changed.