eac123 commited on
Commit
9e26d38
·
verified ·
1 Parent(s): 96d4e86

README: v2 family rows + dense-supervision finding in headline

Browse files
Files changed (1) hide show
  1. README.md +19 -2
README.md CHANGED
@@ -54,8 +54,12 @@ cosine decay, batch 256, 25k steps (100k for fixed4-long).
54
  | rec-1×dynamic | 1 × R~U{2..8} | 234,880 | 2.73e15 |
55
  | rec-4×4 | 4 × 4 | 829,696 | 8.68e15 |
56
  | fixed-4-long | 4 × 1, 100k steps | 829,696 | 8.74e15 |
 
 
 
57
 
58
  Seeds: 3 per family (2 for fixed-4-long). All checkpoints in `ckpts/`.
 
59
 
60
  ## Headline results (mean over seeds, exact-match accuracy)
61
 
@@ -67,6 +71,9 @@ Seeds: 3 per family (2 for fixed-4-long). All checkpoints in `ckpts/`.
67
  | rec-1×dynamic | 0.899±0.008 | 0.103±0.036 | 0.195 |
68
  | rec-4×4 | **1.000** | 0.251±0.099 | 0.269 |
69
  | fixed-4-long (iso-FLOP) | 0.716±0.009 | 0.171±0.018 | 0.172 |
 
 
 
70
 
71
  Key observations (empirical):
72
 
@@ -80,8 +87,18 @@ Key observations (empirical):
80
  3. **More inference-time unrolls than trained sometimes help a lot** —
81
  rec-4×4 seed2: 0.394→0.401 at R=5; rec-1×dynamic seed1: 0.088→0.319 at
82
  R=5 — but the effect is seed-dependent, not systematic at this scale.
83
- 4. **Large seed variance in OOD** (rec-4×4 k=64: 0.18/0.18/0.39) means
84
- single-seed comparisons of length generalisation are unreliable here.
 
 
 
 
 
 
 
 
 
 
85
 
86
  See REPORT.md for the full analysis, figures, and limitations.
87
 
 
54
  | rec-1×dynamic | 1 × R~U{2..8} | 234,880 | 2.73e15 |
55
  | rec-4×4 | 4 × 4 | 829,696 | 8.68e15 |
56
  | fixed-4-long | 4 × 1, 100k steps | 829,696 | 8.74e15 |
57
+ | rec-1×aligned | 1 × k (R = k) | 240,256 | 2.46e15 |
58
+ | rec-1×m4 | 1 × (k+4) | 242,048 | 4.62e15 |
59
+ | rec-1×m4+dense | 1 × (k+4), stem, dense loss | 440,320 | 4.62e15* |
60
 
61
  Seeds: 3 per family (2 for fixed-4-long). All checkpoints in `ckpts/`.
62
+ (*The m4+dense row's stored FLOPs under-count its stem by ~12%.)
63
 
64
  ## Headline results (mean over seeds, exact-match accuracy)
65
 
 
71
  | rec-1×dynamic | 0.899±0.008 | 0.103±0.036 | 0.195 |
72
  | rec-4×4 | **1.000** | 0.251±0.099 | 0.269 |
73
  | fixed-4-long (iso-FLOP) | 0.716±0.009 | 0.171±0.018 | 0.172 |
74
+ | rec-1×aligned (R=k) | 0.106±0.003 | 0.017±0.003 | 0.139 |
75
+ | rec-1×m4 (R=k+4) | 0.262±0.003 | 0.029±0.017 | 0.196 |
76
+ | rec-1×m4+dense | **1.000±0.000** | 0.075±0.032 | **0.437** |
77
 
78
  Key observations (empirical):
79
 
 
87
  3. **More inference-time unrolls than trained sometimes help a lot** —
88
  rec-4×4 seed2: 0.394→0.401 at R=5; rec-1×dynamic seed1: 0.088→0.319 at
89
  R=5 — but the effect is seed-dependent, not systematic at this scale.
90
+ 4. **Large seed variance in OOD** (rec-4×4 k=64: 0.18/0.18/0.39;
91
+ m4+dense k=64: 0.116/0.038/0.070) means single-seed comparisons of
92
+ length generalisation are unreliable here.
93
+ 5. **The v2 ablation (§4.7-4.8) decomposes what makes aligned training
94
+ work**: R=k with end-of-chain loss fails (0.106 — credit assignment
95
+ across depths); a +4 unroll margin alone only partially helps (0.262);
96
+ adding an untied stem block and dense per-iteration supervision with
97
+ hold-after-k targets f^min(t,k)(x) fully rescues it (1.000), and its
98
+ fixed-R sweep shows a staircase (+0.634) — the model genuinely does
99
+ one hop per unroll. **But even this model does not extend past k=8**
100
+ (best k=64 over R: 0.437) — the frontier tracks the training-data
101
+ limit, making curriculum/data coverage the most promising next lever.
102
 
103
  See REPORT.md for the full analysis, figures, and limitations.
104