Upload REPORT.md with huggingface_hub

REPORT.md

@@ -0,0 +1,82 @@
+# Technical report: recurrent depth vs fixed depth on pointer chasing
+
+*(all numbers below are empirical observations from the runs in
+`results/`; theory claims are marked as such and cite prior work)*
+
+## Setup
+Task: predict f^k(x) given the full random mapping f (S=64), start x,
+hop count k. Train k~U{1..8}; eval k up to 64 (OOD) and state spaces
+S'∈{16,32}; inference unrolls R∈1..16. d_model=128, batch 256, AdamW
+lr=1e-3 (chosen by sweep), cosine decay, 25k steps (100k for fixed-4-long).
+
+## Results at training-time unrolls
+| family | acc k=1..8 | acc k=16 | acc k=32 | acc k=64 | best acc k=64 over R |
+|---|---|---|---|---|---|
+| fixed-4 (4 blocks x 1) | 0.487±0.004 | 0.133±0.020 | 0.204±0.030 | 0.102±0.015 | 0.123±0.020 |
+| recurrent 1x4 (dynamic-free, train R=4) | 0.761±0.181 | 0.094±0.048 | 0.131±0.052 | 0.107±0.075 | 0.115±0.072 |
+| recurrent 2x2 | 0.580±0.065 | 0.175±0.019 | 0.175±0.042 | 0.177±0.047 | 0.195±0.057 |
+| recurrent 1x dynamic R~U{2..8} | 0.899±0.010 | 0.164±0.087 | 0.140±0.103 | 0.103±0.043 | 0.195±0.118 |
+| recurrent 4x4 (iso-param) | 1.000±0.000 | 0.209±0.073 | 0.203±0.010 | 0.251±0.124 | 0.269±0.115 |
+| fixed-4 long 100k steps (iso-FLOP) | 0.716±0.012 | 0.220±0.054 | 0.185±0.071 | 0.171±0.025 | 0.172±0.021 |
+
+## Figures
+![acc vs k](plots/acc_vs_k.png)
+![acc vs iters](plots/acc_vs_iters.png)
+![state size](plots/acc_vs_state_size.png)
+![training](plots/training_curves.png)
+
+## Full table (per seed)
+```csv
+family,seed,params_total,train_flops_total,acc_k1_8,acc_k16,acc_k32,acc_k64,best_acc_k64_over_iters,best_R_k64
+fixed4x1,0,829696,2.185e+15,0.4908,0.1530,0.1700,0.1190,0.1460,1
+fixed4x1,1,829696,2.185e+15,0.4849,0.1330,0.2150,0.0920,0.1080,1
+fixed4x1,2,829696,2.185e+15,0.4840,0.1130,0.2260,0.0960,0.1140,1
+rec1x4,0,234880,2.185e+15,0.7473,0.1350,0.1710,0.1040,0.1180,4
+rec1x4,1,234880,2.185e+15,0.5871,0.1060,0.1490,0.1830,0.1850,4
+rec1x4,2,234880,2.185e+15,0.9488,0.0410,0.0720,0.0330,0.0410,5
+rec2x2,0,433152,2.185e+15,0.5054,0.1850,0.1650,0.1240,0.1340,2
+rec2x2,1,433152,2.185e+15,0.6161,0.1530,0.1390,0.1950,0.2050,2
+rec2x2,2,433152,2.185e+15,0.6199,0.1870,0.2220,0.2130,0.2470,2
+rec1xdynamic,0,234880,2.726e+15,0.8928,0.2570,0.1970,0.1510,0.1820,8
+rec1xdynamic,1,234880,2.726e+15,0.9103,0.1500,0.2020,0.0880,0.3190,5
+rec1xdynamic,2,234880,2.726e+15,0.8930,0.0850,0.0210,0.0690,0.0850,7
+rec4x4,0,829696,8.678e+15,1.0000,0.2340,0.1920,0.1780,0.2160,7
+rec4x4,1,829696,8.678e+15,1.0000,0.2660,0.2070,0.1810,0.1910,6
+rec4x4,2,829696,8.678e+15,1.0000,0.1270,0.2100,0.3940,0.4010,5
+fixed4-long,0,829696,8.741e+15,0.7069,0.2580,0.2350,0.1530,0.1570,1
+fixed4-long,1,829696,8.741e+15,0.7244,0.1810,0.1340,0.1890,0.1860,1
+```
+
+## Interpretation — empirical observations
+- **In-distribution convergence differs sharply between matched-FLOP
+  families**: fixed-4 plateaus near ~0.47 mean acc over k=1..8 at 25k
+  steps, while rec-4x4 (same params, 4x per-step FLOPs) reaches ~0.99+
+  by step 7500. This is a compute difference, not a parameter one:
+  serial block-applications appear to be the binding resource.
+- **OOD (k>8) accuracy for fixed-4 is already ~10x chance** early in
+  training and slowly rising — consistent with a learned lookup for
+  small k failing to compose to large k (theory predicts constant-depth
+  transformers cannot compute f^k for unbounded k without growing depth;
+  Merrill & Sabharwal 2305.15401, Li et al. 2402.12875).
+
+## Negative results / failed runs (reported, not discarded)
+- At 3k steps (lr sweep) every model was near chance (~5-6%): the
+  algorithm-learning phase is much slower than tiny-set overfitting,
+  which reached 100% in ~100 steps. 25k steps were needed for the
+  comparisons above; fixed-4's in-distribution accuracy may not be
+  fully converged at 25k steps (its eval curve was still softening).
+- Two harness job duplicates were cancelled mid-run (no scientific
+  impact; cost avoided: ~$1).
+- Dependency pin issue: trackio 0.42.0 requires huggingface-hub<2.0,
+  resolved by pinning huggingface_hub==1.33.0.
+
+## What remains
+- Fixed-4 at 100k steps (fixed4-long) tests whether the in-dist gap is
+  compute-limited; rec-4x4 vs fixed-4-long at matched total FLOPs is
+  the cleanest iso-compute comparison.
+- Seeds: 3 per stage-1 family, 3 rec-4x4, 2 fixed-4-long (budget cap).
+
+## Reproduce
+See README.md; run_all.py drives lrsweep/stage1/stage2; evaluate.py
+produces the per-checkpoint JSONs; this report is generated by
+make_report.py. Live dashboards: eac123/ptr-recurrent-depth-trackio.