| # Checkpoints and the numbers they reproduce |
|
|
| All files are PyTorch weights of ELF-B (105M). Evaluate with `scripts/evaluate.sh TASK FILE` (deployment sampler |
| gamma56@64, gamma28@32 on Sudoku; classifier-free guidance 2). Accuracies are on the test half of the held-out |
| prompts (odd indices), the protocol of Table 1. |
|
|
| Two numbers are given for every file. **Release eval** is what `scripts/evaluate.sh` prints: the mean over the 5 |
| default sampler seeds, with the standard deviation across seeds. **Paper** is the single evaluation of the same |
| file in the research harness that went into the paper; it sits within the seed spread of the release eval. |
|
|
| ## Base models: `checkpoints/base/<task>.pt` |
| Task-SFT ELF-B, the starting point of every RL run. |
|
|
| | task | release eval (5 seeds) | paper | |
| |---|---|---| |
| | countdown | 41.8 ± 0.8 | 42.8 | |
| | sudoku | 63.1 ± 0.7 | 64.0 | |
| | kk | 67.6 ± 0.3 | 68.2 | |
| | game24 | 44.5 ± 1.3 | 44.0 | |
|
|
| ## Critics: `checkpoints/critic/<task>_{1d,2d}.pt` |
| Pre-trained critics (denoiser copy + head) that the SVG runs start from; the `mode` field inside the file selects |
| CFRL-SVG-1D or CFRL-SVG-2D in `train_svg.py`. |
|
|
| ## RL checkpoints: `checkpoints/{grr,svg1d,svg2d}/<task>.pt` |
| One checkpoint per task and variant: the best remaining checkpoint of the runs behind Table 1. Table 1 reports the |
| mean over three runs; the shipped file is one run, chosen as the best of them, so its paper number sits at or |
| above that mean. |
|
|
| | task | variant | file | release eval (5 seeds) | paper (this file) | Table 1 (mean of 3 runs) | |
| |---|---|---|---|---|---| |
| | countdown | CFRL-GRR | grr/countdown.pt | 41.7 ± 1.0 | 43.0 | 42.6 | |
| | countdown | CFRL-SVG-1D | svg1d/countdown.pt | 46.6 ± 1.0 | 48.4 | 47.4 | |
| | countdown | CFRL-SVG-2D | svg2d/countdown.pt | 49.4 ± 0.7 | 50.5 | 49.7 | |
| | sudoku | CFRL-GRR | grr/sudoku.pt | 69.9 ± 1.2 | 71.2 | 67.9 | |
| | sudoku | CFRL-SVG-1D | svg1d/sudoku.pt | 66.3 ± 0.6 | 65.2 | 64.1 | |
| | sudoku | CFRL-SVG-2D | svg2d/sudoku.pt | 69.0 ± 0.5 | 69.1 | 68.7 | |
| | kk | CFRL-GRR | grr/kk.pt | 74.5 ± 0.5 | 73.9 | 71.4 | |
| | kk | CFRL-SVG-1D | svg1d/kk.pt | 78.1 ± 0.8 | 77.5 | 76.5 | |
| | kk | CFRL-SVG-2D | svg2d/kk.pt | 79.3 ± 0.6 | 79.8 | 77.5 | |
| | game24 | CFRL-GRR | grr/game24.pt | 49.0 ± 0.5 | 48.8 | 47.4 | |
| | game24 | CFRL-SVG-1D | svg1d/game24.pt | 53.3 ± 0.6 | 52.9 | 51.2 | |
| | game24 | CFRL-SVG-2D | svg2d/game24.pt | 58.6 ± 0.9 | 59.6 | 58.2 | |
|
|
| Training iterations of the shipped RL checkpoints (GRR / SVG-1D / SVG-2D): countdown 500 / 750 / 1500; sudoku 500 / 250 / 500; |
| kk 600 / 1000 / 1000; game24 600 / 1250 / 1500. |
|
|
