docs/tutorials/06-evaluation.md

M5: evaluate the frozen study and demonstrate routing

Status: final inference, analysis, latency v2 and all-example integration demonstration completed. The aggregate inspection below is verified, and the independent final review passed. This chapter reports the retained first attempt, including unfavorable evidence.

M4 fitted operating policies on calibration and committed the final protocol at 3c1431b5ec2313394f83ffdc00543a9913b4ee71. M5 measures those exact choices on the untouched final set and explains their practical limits. Read the calibration tutorial first.

1. Make the final boundary explicit

The source runs, dataset, fitted policies, analysis code, primary seed and uncertainty method are already fixed. The final launcher compares protocol bytes with their Git commit and verifies source and input identities. A filename alone would not establish that the same protocol preceded the measurements.

Model selection is over. Do not choose another seed, lower a threshold or edit a prompt to improve the final result. New hypotheses suggested by failures need a new holdout after further development.

2. Retain the first prediction attempt

The completed recorded command was:

uv run --locked --extra ml python scripts/predict_selected.py \
  --selection configs/final-protocol.json \
  --protocol-commit 3c1431b \
  --partition test \
  --output artifacts/m5-predictions-v1

All eight contenders completed, with 360 finite aligned outputs each and zero technical failures. The original output directories already exist and must not be overwritten. For a new full reproduction, use a fresh directory and your own previously committed protocol; the public source archive does not contain the private Git history. Preserve launch receipts, subprocess outputs and partial first-answer journals. A failed model call is part of the outcome, not permission to erase an inconvenient example.

The final set contains 360 cases in 30 scenario groups, balanced across the three labels. Familiar and held-out-family cohorts each contain 180 cases in 15 groups. The correct grouping unit is the authored scenario, including all 12 related views.

3. Analyze saved logits under the committed policy

The completed analysis used saved logits without another model call:

uv run --locked --extra ml python scripts/analyze_study.py \
  --protocol configs/final-protocol.json \
  --protocol-commit 3c1431b \
  --prediction-root artifacts/m5-predictions-v1 \
  --output artifacts/m5-analysis-v1

Analysis checks frozen identities before opening labels. Raw argmax, calibrated probabilities and routed decisions remain distinct outputs. The final report explains all eight methods, cohorts, family support, errors and costs; the compact summary preserves numerical results. For CPU-only public reproduction without private Git history, use the saved-prediction audit. That recomputes evidence from published predictions and labels; it does not independently reproduce training or inference provenance.

The primary denominator includes all ambiguous requests: deferring one counts as a missed clarification. False clarification is measured over all complete supported requests. Coverage and covered-only accuracy describe different tradeoffs and cannot replace those denominators.

4. Read the paired group interval

The frozen analysis uses 2,000 paired whole-scenario bootstrap replicates with seed 101. Each replicate resamples the same groups for both compared models, preserving both within-scenario dependence and the paired comparison. Resampling individual rendered rows would overstate the amount of independent evidence.

The primary adapted model recovered 120/120 ambiguous requests versus 62/120 for unchanged Kai, with 0/120 versus 3/120 unnecessary questions. The recall difference is +48.3 percentage points, with a paired 95% interval of +40.2 to +56.8 points. The predefined positive-result criterion—a strictly positive lower recall bound and adapted false clarification no greater than 10%—was met. Descriptive cohort/family results explain heterogeneity; they do not replace that primary rule.

Verify the portable aggregates without loading a model:

uv run --locked --extra ml python - <<'PY'
import json
from pathlib import Path

summary = json.loads(Path("reports/m5-summary.json").read_text())
assert summary["protocol_commit"] == "3c1431b"
assert len(summary["models"]) == 8
for name, result in summary["models"].items():
    assert result["n"] == 360 and result["groups"] == 30
    assert result["failed_prediction_rows"] == 0
    routed = result["routed"]
    print(name, "clarify", routed["confusion"]["clarify"]["clarify"],
          "false questions", routed["confusion"]["suggest"]["clarify"],
          "coverage", routed["coverage"])
interval = summary["primary_difference"]["clarify_recall"]
primary = summary["models"]["kai-adapted-s42"]["routed"]
assert interval["lower_95"] > 0 and primary["false_clarify_rate"] <= 0.10
print("primary recall interval:", interval)
PY

5. Explain calibration, errors and seed variation

Primary adapted Kai's raw argmax classified 360/360 synthetic rows correctly, while its frozen route deferred 28 (21 complete supported and seven unsupported). Its routed accuracy and coverage are both 92.2%. Calling the entire system “100% accurate” would hide these unanswered cases and overstate what the test measures.

Temperature scaling worsened primary final log loss from 0.001395 to 0.021902, despite improving calibration-split log loss. It also worsened both other adapted seeds. Inspect reliability and coverage beside the raw/routed tables. Do not disable temperature or deferral after seeing this result.

Calibration fitting used separate data, but its empirical 90% coverage and 10% false-question constraints are not test guarantees. MiniLM, unchanged Kai and adapted seeds 43/44 fell below 90% final coverage. The primary model met that coverage target, and all three adapted models had no wrong covered routes, but seeds 43/44 deferred seven/one ambiguous requests. A final violation must be reported without refitting.

All three training seeds remain visible. A better seed-44 result would not change seed 42's primary role. Controlled synthetic language can make a result appear stronger than performance on natural requests.

6. Measure the complete routing path

The recorded v2 command was:

uv run --locked --extra ml python scripts/benchmark_routes.py \
  --protocol configs/final-protocol.json \
  --output artifacts/m5-latency-v2

The protocol uses 60 fixed validation requests, batch size one, three warmup requests and separate model-load timing. Timed work includes encoding, inference, policy routing, strict output validation and JSON serialization. GPU forward time alone would omit work a caller actually pays for.

Median/p95 were 0.021/0.028 ms for rules, 6.410/8.598 ms for CPU MiniLM, 8.650/9.022 ms for unchanged CUDA Kai and 14.744/15.016 ms for adapted CUDA Kai. Adapted Kai's allocator peak was 2.28 GiB. The latency record retains first-request and loading costs, device information and all request timings. The earlier v1 timing attempt predates the final protocol integrity correction; v2 is the final reported measurement.

Keep CPU MiniLM and CUDA Kai hardware explicit. The machine is shared; measured latencies describe this environment and scope rather than a universal speed ordering. Inspect numerical drift between batched validation and single-request execution separately from timing.

7. Demonstrate without executing a command

The completed local staging and demonstration interfaces are:

uv run --locked --extra ml python scripts/build_release.py \
  --run artifacts/m3/lora-n1008-lr0.0001-s42 \
  --protocol configs/final-protocol.json \
  --model-id kai-adapted-s42 \
  --output artifacts/m5-demo-release

uv run --locked --extra ml python scripts/demo_policy.py \
  --release artifacts/m5-demo-release \
  --base artifacts/kai-package \
  --output artifacts/m5-demo.json

Use fresh destinations to reproduce; the recorded ones already exist. The staged adaptation's reference logits matched exactly. This local staging smoke is separate from M6's installed public download verification.

The demonstration runs all 12 approved policy examples, not a curated subset of successes. It produced eight expected routes, three deferrals and one wrong route. P04 asks to display local lines containing the literal docker run and explicitly says not to launch Docker; it should suggest, but returned unsupported with calibrated probability 0.986556. P03's complete deletion request, P06's missing move destination and P08's contradictory Git approvals deferred. The full demonstration record retains every input and output.

This confident literal-text/negation failure is more informative about a limitation than the perfect synthetic raw score alone. These policy examples were already known and are illustrative, not a new unbiased benchmark. Preserve the failure without tuning from it.

The classifier emits a suggestion_request containing request/context for a downstream generator; it does not itself call Bashcraft or generate a command. A fixed clarification message does not demonstrate tailored question generation. The other workflows are scope messaging and human review. Every output records command_generated: false and command_executed: false.

This CPU-only check verifies that the saved demo retained all examples and did not claim command execution:

uv run --locked --extra ml python - <<'PY'
import json
from pathlib import Path

demo = json.loads(Path("reports/m5-demo.json").read_text())
assert len(demo["examples"]) == 12
assert demo["command_execution"] is False
for example in demo["examples"]:
    workflow = example["output"]["workflow"]
    assert workflow["command_generated"] is False
    assert workflow["command_executed"] is False
    print(example["id"], example["policy_label"], example["output"]["decision"])
PY

Troubleshooting and alternatives

• A protocol, source or dataset hash mismatch is a provenance failure; preserve it and diagnose the cause before using scores.
• Existing output directories protect the first attempt. Do not overwrite them to obtain a cleaner result.
• Missing or malformed final outputs remain failed/deferred rows with eligible denominators intact.
• High covered-only accuracy can accompany poor clarification recall when many cases are deferred.
• A negative primary finding can complete an honest study. Additional tuning is a new experiment, not a repair to its final score.
• Raw row count does not describe natural-language diversity: the study contains 150 authored scenarios and shared rendering structures.

Exercises

1. Explain why a confidence interval based on individual rows can be too narrow here.
2. Find a tradeoff that raw accuracy alone would hide.
3. Separate evidence of checkpoint reload, calibrated routing, final generalization and nonexecuting integration.