Document final 64-sample evaluation

README.md

@@ -48,8 +48,35 @@ model = AutoModelForCausalLM.from_pretrained(
 
 Spectral-soft retained more correct-program AST coverage than the controls at round 5. This is the conditional expected coverage in four draws from the correct samples, not coverage from four total generations. Its paired coverage delta versus SPD-hard was +0.293 [0.233, 0.353] on 251 shared eligible tasks. Its pass@1 delta versus SPD-hard was -0.0118 [-0.0190, -0.0048], so the declared 1% noninferiority criterion was not met. All three methods lost implementation diversity relative to the base model over five rounds.
 
+## Final 64-sample evaluation
+
+The base model and all three round-5 checkpoints were subsequently evaluated
+with 64 samples on each of the same 500 held-out tasks (32,000 samples per
+model). The complete compact evidence is in [`eval64/`](eval64/REPORT.md).
+
+| Model | pass@1 | pass@64 | Total AST coverage@64 | Correct-matched coverage@4 (eligible) |
+| --- | ---: | ---: | ---: | ---: |
+| Base | 0.379 | 0.720 | 12.802 | 3.358 (322/500) |
+| Plain | 0.414 | 0.704 | 8.506 | 2.856 (321/500) |
+| SPD-hard | 0.418 | 0.702 | 8.388 | 2.810 (321/500) |
+| Spectral-soft | 0.403 | 0.726 | 11.510 | 3.143 (323/500) |
+
+Against SPD-hard, spectral-soft improves total coverage@64 by +3.122
+[2.686, 3.596], pass@64 by +0.024 [0.006, 0.044], and correct-matched
+coverage@4 by +0.334 [0.290, 0.379]. Pass@1 is lower by -0.0143
+[-0.0189, -0.0098], so the 1% correctness noninferiority gate remains unmet.
+
 ## Limitations
 
-This is a single-seed (`43`), 16-sample reduced-budget experiment. AST fingerprints are implementation proxies, not independently annotated algorithms. Correct-matched coverage is conditional on tasks producing at least four correct candidates. Training used raw self-generated candidates without correctness filtering. Candidate execution used the explicitly opted-in local evaluator rather than an isolated Docker evaluator. Generation-policy diagnostics were disabled. These checkpoints are research artifacts and should not be treated as production code-generation models.
+This is a single-training-seed (`43`) experiment. The per-round evaluation used
+16 samples per task and the final supplement used 64. AST fingerprints are
+implementation proxies, not independently annotated algorithms.
+Correct-matched coverage is conditional on tasks producing enough correct
+candidates; at budget 64 only 24–53 tasks are eligible per model. Training used
+raw self-generated candidates without correctness filtering. Candidate
+execution used the explicitly opted-in local evaluator rather than an isolated
+Docker evaluator. Generation-policy diagnostics were disabled. These
+checkpoints are research artifacts and should not be treated as production
+code-generation models.
 
 The repository includes the exact configuration, compact per-round metrics, paired final-round comparisons, run status, and SHA-256 checksums. Full reports and code belong to https://github.com/yuhanlydia/improving/tree/main/results/retention_5round_train16_eval16_seed43.