README: v2 family rows + dense-supervision finding in headline
Browse files
README.md
CHANGED
|
@@ -54,8 +54,12 @@ cosine decay, batch 256, 25k steps (100k for fixed4-long).
|
|
| 54 |
| rec-1×dynamic | 1 × R~U{2..8} | 234,880 | 2.73e15 |
|
| 55 |
| rec-4×4 | 4 × 4 | 829,696 | 8.68e15 |
|
| 56 |
| fixed-4-long | 4 × 1, 100k steps | 829,696 | 8.74e15 |
|
|
|
|
|
|
|
|
|
|
| 57 |
|
| 58 |
Seeds: 3 per family (2 for fixed-4-long). All checkpoints in `ckpts/`.
|
|
|
|
| 59 |
|
| 60 |
## Headline results (mean over seeds, exact-match accuracy)
|
| 61 |
|
|
@@ -67,6 +71,9 @@ Seeds: 3 per family (2 for fixed-4-long). All checkpoints in `ckpts/`.
|
|
| 67 |
| rec-1×dynamic | 0.899±0.008 | 0.103±0.036 | 0.195 |
|
| 68 |
| rec-4×4 | **1.000** | 0.251±0.099 | 0.269 |
|
| 69 |
| fixed-4-long (iso-FLOP) | 0.716±0.009 | 0.171±0.018 | 0.172 |
|
|
|
|
|
|
|
|
|
|
| 70 |
|
| 71 |
Key observations (empirical):
|
| 72 |
|
|
@@ -80,8 +87,18 @@ Key observations (empirical):
|
|
| 80 |
3. **More inference-time unrolls than trained sometimes help a lot** —
|
| 81 |
rec-4×4 seed2: 0.394→0.401 at R=5; rec-1×dynamic seed1: 0.088→0.319 at
|
| 82 |
R=5 — but the effect is seed-dependent, not systematic at this scale.
|
| 83 |
-
4. **Large seed variance in OOD** (rec-4×4 k=64: 0.18/0.18/0.39
|
| 84 |
-
single-seed comparisons of
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
See REPORT.md for the full analysis, figures, and limitations.
|
| 87 |
|
|
|
|
| 54 |
| rec-1×dynamic | 1 × R~U{2..8} | 234,880 | 2.73e15 |
|
| 55 |
| rec-4×4 | 4 × 4 | 829,696 | 8.68e15 |
|
| 56 |
| fixed-4-long | 4 × 1, 100k steps | 829,696 | 8.74e15 |
|
| 57 |
+
| rec-1×aligned | 1 × k (R = k) | 240,256 | 2.46e15 |
|
| 58 |
+
| rec-1×m4 | 1 × (k+4) | 242,048 | 4.62e15 |
|
| 59 |
+
| rec-1×m4+dense | 1 × (k+4), stem, dense loss | 440,320 | 4.62e15* |
|
| 60 |
|
| 61 |
Seeds: 3 per family (2 for fixed-4-long). All checkpoints in `ckpts/`.
|
| 62 |
+
(*The m4+dense row's stored FLOPs under-count its stem by ~12%.)
|
| 63 |
|
| 64 |
## Headline results (mean over seeds, exact-match accuracy)
|
| 65 |
|
|
|
|
| 71 |
| rec-1×dynamic | 0.899±0.008 | 0.103±0.036 | 0.195 |
|
| 72 |
| rec-4×4 | **1.000** | 0.251±0.099 | 0.269 |
|
| 73 |
| fixed-4-long (iso-FLOP) | 0.716±0.009 | 0.171±0.018 | 0.172 |
|
| 74 |
+
| rec-1×aligned (R=k) | 0.106±0.003 | 0.017±0.003 | 0.139 |
|
| 75 |
+
| rec-1×m4 (R=k+4) | 0.262±0.003 | 0.029±0.017 | 0.196 |
|
| 76 |
+
| rec-1×m4+dense | **1.000±0.000** | 0.075±0.032 | **0.437** |
|
| 77 |
|
| 78 |
Key observations (empirical):
|
| 79 |
|
|
|
|
| 87 |
3. **More inference-time unrolls than trained sometimes help a lot** —
|
| 88 |
rec-4×4 seed2: 0.394→0.401 at R=5; rec-1×dynamic seed1: 0.088→0.319 at
|
| 89 |
R=5 — but the effect is seed-dependent, not systematic at this scale.
|
| 90 |
+
4. **Large seed variance in OOD** (rec-4×4 k=64: 0.18/0.18/0.39;
|
| 91 |
+
m4+dense k=64: 0.116/0.038/0.070) means single-seed comparisons of
|
| 92 |
+
length generalisation are unreliable here.
|
| 93 |
+
5. **The v2 ablation (§4.7-4.8) decomposes what makes aligned training
|
| 94 |
+
work**: R=k with end-of-chain loss fails (0.106 — credit assignment
|
| 95 |
+
across depths); a +4 unroll margin alone only partially helps (0.262);
|
| 96 |
+
adding an untied stem block and dense per-iteration supervision with
|
| 97 |
+
hold-after-k targets f^min(t,k)(x) fully rescues it (1.000), and its
|
| 98 |
+
fixed-R sweep shows a staircase (+0.634) — the model genuinely does
|
| 99 |
+
one hop per unroll. **But even this model does not extend past k=8**
|
| 100 |
+
(best k=64 over R: 0.437) — the frontier tracks the training-data
|
| 101 |
+
limit, making curriculum/data coverage the most promising next lever.
|
| 102 |
|
| 103 |
See REPORT.md for the full analysis, figures, and limitations.
|
| 104 |
|