add MANIFEST.md

MANIFEST.md

@@ -0,0 +1,46 @@
+# Checkpoints and the numbers they reproduce
+
+All files are PyTorch weights of ELF-B (105M). Evaluate with `scripts/evaluate.sh TASK FILE` (deployment sampler
+gamma56@64, gamma28@32 on Sudoku; classifier-free guidance 2). Accuracies are on the test half of the held-out
+prompts (odd indices), the protocol of Table 1.
+
+Two numbers are given for every file. **Release eval** is what `scripts/evaluate.sh` prints: the mean over the 5
+default sampler seeds, with the standard deviation across seeds. **Paper** is the single evaluation of the same
+file in the research harness that went into the paper; it sits within the seed spread of the release eval.
+
+## Base models: `checkpoints/base/<task>.pt`
+Task-SFT ELF-B, the starting point of every RL run.
+
+| task | release eval (5 seeds) | paper |
+|---|---|---|
+| countdown | 41.8 ± 0.8 | 42.8 |
+| sudoku | 63.1 ± 0.7 | 64.0 |
+| kk | 67.6 ± 0.3 | 68.2 |
+| game24 | 44.5 ± 1.3 | 44.0 |
+
+## Critics: `checkpoints/critic/<task>_{1d,2d}.pt`
+Pre-trained critics (denoiser copy + head) that the SVG runs start from; the `mode` field inside the file selects
+CFRL-SVG-1D or CFRL-SVG-2D in `train_svg.py`.
+
+## RL checkpoints: `checkpoints/{grr,svg1d,svg2d}/<task>.pt`
+One checkpoint per task and variant: the best remaining checkpoint of the runs behind Table 1. Table 1 reports the
+mean over three runs; the shipped file is one run, chosen as the best of them, so its paper number sits at or
+above that mean.
+
+| task | variant | file | release eval (5 seeds) | paper (this file) | Table 1 (mean of 3 runs) |
+|---|---|---|---|---|---|
+| countdown | CFRL-GRR | grr/countdown.pt | 41.7 ± 1.0 | 43.0 | 42.6 |
+| countdown | CFRL-SVG-1D | svg1d/countdown.pt | 46.6 ± 1.0 | 48.4 | 47.4 |
+| countdown | CFRL-SVG-2D | svg2d/countdown.pt | 49.4 ± 0.7 | 50.5 | 49.7 |
+| sudoku | CFRL-GRR | grr/sudoku.pt | 69.9 ± 1.2 | 71.2 | 67.9 |
+| sudoku | CFRL-SVG-1D | svg1d/sudoku.pt | 66.3 ± 0.6 | 65.2 | 64.1 |
+| sudoku | CFRL-SVG-2D | svg2d/sudoku.pt | 69.0 ± 0.5 | 69.1 | 68.7 |
+| kk | CFRL-GRR | grr/kk.pt | 74.5 ± 0.5 | 73.9 | 71.4 |
+| kk | CFRL-SVG-1D | svg1d/kk.pt | 78.1 ± 0.8 | 77.5 | 76.5 |
+| kk | CFRL-SVG-2D | svg2d/kk.pt | 79.3 ± 0.6 | 79.8 | 77.5 |
+| game24 | CFRL-GRR | grr/game24.pt | 49.0 ± 0.5 | 48.8 | 47.4 |
+| game24 | CFRL-SVG-1D | svg1d/game24.pt | 53.3 ± 0.6 | 52.9 | 51.2 |
+| game24 | CFRL-SVG-2D | svg2d/game24.pt | 58.6 ± 0.9 | 59.6 | 58.2 |
+
+Training iterations of the shipped RL checkpoints (GRR / SVG-1D / SVG-2D): countdown 500 / 750 / 1500; sudoku 500 / 250 / 500;
+kk 600 / 1000 / 1000; game24 600 / 1250 / 1500.