nanodiff-350m-typed-decisions-lam1

Proper-scoring arm (λ = 1.0) of the typed-decision calibration study: identical to the control except the decision-slot CE adds to the loss (a log-score — proper scoring rule — on the answer token). See the full report.

• 3,000 steps · seed 1337 · batch 32 × seq 512 × grad-accum 2 (32k tok/step) · A100-80GB, ≈ 27.6 min
• Best-calibration checkpoint: step 2000 (eval_calibration_step2000.json here) — the arm overtrains past ~2k steps
• Checkpoints: nanodiff-350m-typed-decisions-lam1.pt (step 3000) + the step-2000 revision (commit abf2adb715; torch.load → state dict; load exactly as in code/eval_calibration.py)
• Eval artifacts in this repo: eval_calibration.json (+ logits), eval_calibration_step2000.json (+ logits)

Eval — step-3000 checkpoint (test, decision-level; T=2.09 fit on cal)

group	n	acc	ece_hard	ece_soft	ece_soft|temp	AUROC	acc@.5
ALL	5214	0.676	0.324	0.058	0.048	0.915	0.957
choice (1-of-10)	1195	0.122	0.878	0.195	0.084	0.518	0.127
noul	231	0.662	0.338	0.188	0.088	0.662	0.793
severity (Score)	338	0.713	0.287	0.048	0.083	0.800	0.929
workflow4 (k=4)	2436	0.858	0.142	0.020	0.073	0.875	0.998

Eval — step-2000 checkpoint (best calibrated; T=1.52)

group	n	acc	ece_hard	ece_soft	ece_soft|temp	AUROC	acc@.5
ALL	5214	0.669	0.331	0.036	0.027	0.919	0.953

Headline: the λ term improves soft calibration from the control's 0.065 to 0.036 at its best step (9× better than the hard-confidence reading 0.331, at equal accuracy), and k=4 multi-slot scoring stays near-perfectly trustworthy (ece_soft 0.020, acc@.5 0.998). On severity the posterior is within L1 0.072 of the closed-form Bayes-optimal posterior (model acc 0.713 vs oracle 0.716).

Caveats: choice is at chance (AUROC ≈ 0.52) — a knowledge limitation of the 350M base; escalate/review gold labels disagree with their own Bayes oracle; see the report.

Scoring (one bidirectional pass)

# Only the answer positions are [MASK]ed. See code/eval_calibration.py (pinned at 65691f24)
# for the exact released path (loading, option-token map, T scaling).
logits = model(x_masked, t=...)          # (B, 512, 50304)
probs  = logits[:, slot_idx].softmax(-1) # the decision distribution

Control arm (λ=0): nanodiff-350m-typed-decisions-lam0.