# Technical report: recurrent depth vs fixed depth on pointer chasing *(Everything below marked **observation** is measured from the runs in `results/`. Statements about what transformers *can* compute in principle are **theory** and cite prior work.)* ## TL;DR We train nine families of tiny transformers (3 seeds each, except fixed-4-long: 2) on a synthetic multi-hop task: given a random mapping f, a start state x and a hop count k, output f^k(x). Training only ever uses k ≤ 8; evaluation uses k up to 64 with freshly generated mappings. **Observations:** (1) recurrent depth (a weight-tied block applied R times) learns the trained task far better than a fixed-depth transformer at matched parameters and even at matched *total* training compute; (2) end-of-chain aligned supervision (R=k) fails on credit assignment, but adding a +4 unroll margin, an untied stem layer and dense per-iteration supervision fully rescues it (0.106 -> 1.000 in-distribution, sections 4.7-4.8); (3) **no model in this budget class generalises well to untrained hop counts** (best mean acc ~0.27 at k=64 vs 1.0 in-distribution): the capability frontier tracks the training-data limit in every family that learns, so the remaining bottleneck is data coverage, not architecture. ## 1. The task Each example is a fresh, uniformly random function f on a state space of size S=64, a uniformly random start state x, and a hop count k. The correct answer is f applied k times to x. During training k ~ Uniform{1..8}; at evaluation k ranges up to 64 (out of distribution) and every mapping is newly generated, so nothing can be memorised. Shortcut controls: f is uniform random (no exploitable regular structure, cf. arXiv:2210.10749); the marginal distribution of the answer is ~uniform for every k (verified: total-variation < 0.02, no majority-class shortcut); k is the only token that correlates with difficulty. A deterministic lookup solver scores 1.0 by construction and is re-run at every evaluation as a pipeline integrity check. **Example 1 (small state space S=8, so it fits on the page).** The mapping f is drawn uniformly at random; with this seed it is ```text f(0)=2 f(1)=5 f(2)=2 f(3)=7 f(4)=0 f(5)=3 f(6)=6 f(7)=2 ``` Start state x=1, hop count k=3. Applying f step by step: ```text f^1(1) = f(1) = 7 f^2(1) = f(7) = 2 f^3(1) = f(2) = 7 <- the required answer ``` So the answer is state **7**. Note how the chain entered a cycle (2 -> 7 -> 2 -> ...): random finite mappings always fall into cycles, which is exactly why a model that only looks up f(x) fails once k grows past the cycle structure. The model never sees the chain. It sees one flat token sequence (S+2 = 10 tokens here, 66 for S=64): ```text input : [2, 5, 2, 7, 0, 3, 6, 2, | 3, 3] f(0) f(1) ... f(7) x k target: the state id of f^k(x), predicted at the final position ``` The position of each f-token encodes *which* state it is the image of (state i is always at position i), so no 'arrow' tokens are needed. The final two tokens are the start state x and the hop count k; the model must combine all three pieces of information. **Example 2 (training-scale S=64).** Same construction, but only the first few mapping entries are shown: ```text f = [54, 11, 1, 40, 23, 29, 5, 23, 41, 22, 53, 50, ...] (64 entries) x = 61, k = 7 -> answer f^7(61) = 41 ``` Every training batch draws a *fresh* random f, a fresh x and a fresh k ~ Uniform{{1..8}} for every example, so there is nothing to memorise: the only way to score above chance is to learn the algorithm itself. A note on what 'length' means here: the input sequence is always S+2 = 66 tokens regardless of k, so the OOD axis is **the value of the hop-count token**, not the sequence length. Generalising from k ≤ 8 to k = 64 therefore requires the model to apply its learned procedure 8x more times than it ever saw during training. ## 2. The models All models share one implementation (`ptr/models.py`): a pre-LN transformer block (4 heads, d_ff = 4*d_model, GELU MLP, bidirectional attention, learned positional embeddings) applied **n_iters** times. Two knobs distinguish the families: - `depth` = number of distinct blocks in the stack. In the fixed model each block has its own weights and is applied once. - `n_iters` (R) = how many times the *same* stack is applied. Each unroll adds a learned step embedding so the block can tell pass t from pass t+1. At initialisation the blocks are the identity (zero-initialised output projections), which keeps gradients stable through long unroll chains (cf. arXiv:2604.07822). So 'fixed-4' = depth 4, R=1; 'rec-1x4' = one shared block unrolled R=4 times; 'rec-4x4' = a 4-block stack unrolled R=4 times. The input encoding, output head, optimizer and data pipeline are identical everywhere (d_model=128, batch 256, AdamW lr=1e-3 chosen by a 6-run sweep, cosine decay, 25k steps unless stated). ![architecture schematic](plots/schematic.png) ### The nine families and what is matched | family | blocks x unrolls | params | per-step block-applications | total training FLOPs | seeds | |---|---|---|---|---|---| | fixed-4 | 4 x 1 | 829,696 | 4 | 2.19e15 | 3 | | rec-2x2 | 2 x 2 | 433,152 | 4 | 2.19e15 | 3 | | rec-1x4 | 1 x 4 | 234,880 | 4 | 2.19e15 | 3 | | rec-1x-dynamic | 1 x R~U{2..8} | 234,880 | ~5 avg | 2.73e15 | 3 | | rec-4x4 | 4 x 4 | 829,696 | 16 | 8.68e15 | 3 | | fixed-4-long | 4 x 1, 100k steps | 829,696 | 4 | 8.74e15 | 2 | | rec-1x-aligned | 1 x k (R = k) | 240,256 | ~4.5 avg | 2.46e15 | 3 | | rec-1x-m4 | 1 x (k+4) | 242,048 | ~8.5 avg | 4.62e15 | 3 | | rec-1x-m4+dense | 1 x (k+4), stem, dense loss | 440,320 | ~9.5 avg | 4.62e15* | 3 | Three comparison axes follow from this table: 1. **Matched-FLOP family** (rows 1-3): every model executes exactly 4 block-applications per training step and is trained for the same number of steps, so total training FLOPs are identical while parameters differ 4:2:1. Question: *does the loop structure itself matter when compute is fixed?* 2. **Iso-parameter pair** (fixed-4 vs rec-4x4): identical parameter counts, but rec-4x4 does 4x the per-step work. Question: *given the same weights, does iterating them help?* 3. **Iso-FLOP pair** (rec-4x4 vs fixed-4-long): identical total training FLOPs - fixed-4-long simply trains 4x longer. Question: *can extra training compute substitute for recurrent depth?* ![budget map](plots/budget_map.png) Rows 7-9 (sections 4.7-4.8) test a fourth axis - *supervision and halting structure*: whether training the unroll count to match the task (R = k or k+4), with and without per-step supervision, changes what the recurrence learns. (*The m4+dense row's stored FLOPs under-counts its stem by ~12%; flagged rather than silently corrected.*) ## 3. What is evaluated Every evaluation cell uses freshly generated mappings (nothing from training is reused) and 1000 examples per cell; accuracy is exact match on the answer state (chance = 1/64 ≈ 0.016). The varied parameters are: - **Chain length k ∈ 1..64** at the model's training-time unroll count: the OOD length-generalisation curve. - **Inference-time unroll count R ∈ 1..16**, crossed with k ∈ {1,2,4,8,16,32,64}: 'more or fewer iterations than trained'. For the fixed model, R > 1 re-applies its own block stack (effectively deeper than trained) - also measured. - **State-space size S' ∈ {16, 32, 64}**: evaluation on subsets of the trained vocabulary/positions (larger S' would need new embedding rows the model never had; noted as a limitation). - **Training seed** (3 per family) and **training compute** (the three budget axes above). ## 4. Results ### 4.1 Headline (at training-time unrolls) | family | acc k=1..8 | acc k=16 | acc k=32 | acc k=64 | best acc k=64 over R | |---|---|---|---|---|---| | fixed-4 (4 blocks x 1) | 0.487±0.004 | 0.133±0.020 | 0.204±0.030 | 0.102±0.015 | 0.123±0.020 | | recurrent 1x4 (train R=4) | 0.761±0.181 | 0.094±0.048 | 0.131±0.052 | 0.107±0.075 | 0.115±0.072 | | recurrent 2x2 (train R=2) | 0.580±0.065 | 0.175±0.019 | 0.175±0.042 | 0.177±0.047 | 0.195±0.057 | | recurrent 1x dynamic R~U{2..8} | 0.899±0.010 | 0.164±0.087 | 0.140±0.103 | 0.103±0.043 | 0.195±0.118 | | recurrent 4x4 (train R=4) | 1.000±0.000 | 0.209±0.073 | 0.203±0.010 | 0.251±0.124 | 0.269±0.115 | | fixed-4 long (100k steps) | 0.716±0.012 | 0.220±0.054 | 0.185±0.071 | 0.171±0.025 | 0.172±0.021 | | rec-1x-aligned (train R=k) | 0.106±0.003 | 0.038±0.016 | 0.027±0.012 | 0.017±0.004 | 0.139±0.041 | | rec-1x m4 (R=k+4, end-of-chain loss) | 0.262±0.004 | 0.070±0.020 | 0.069±0.029 | 0.029±0.022 | 0.196±0.025 | | rec-1x m4+dense (R=k+4, stem, dense loss) | 1.000±0.000 | 0.190±0.060 | 0.114±0.041 | 0.075±0.039 | 0.437±0.002 | ![headline bars](plots/headline_bars.png) Reading the bar chart: blue = how well the model learned the task it was trained on; red = how far that skill stretches to 8x longer chains. The gap between them is the length-generalisation deficit, and it is large for every family. ### 4.2 Accuracy vs chain length ![acc vs k](plots/acc_vs_k.png) All curves bend downward after the training range (k ≤ 8, dashed line). **Observation:** no family learned a clean length-generalising algorithm in this budget class. ### 4.3 Accuracy vs inference-time unrolls (the k x R grid) Two different quantities are crossed in this figure, and keeping them apart is the whole point: - **k — task depth.** How many times f must be applied to answer the question. It is part of the *input* (the k token) and is fixed along each line of the figure. - **R — the recurrent loop parameter.** How many times the model *iterates its own block stack* before reading out an answer. It is a free choice made at inference time: the trained model is frozen and we simply run it for more or fewer unrolls. The dashed vertical line marks the R each family was trained with. **What is being tested.** If a recurrent model learned the abstraction 'apply one unroll per hop', accuracy should rise with R and peak near R = k. If instead it learned a fixed amount of computation (whatever R it was trained with), accuracy should peak at the trained R regardless of k and fall off on both sides. ![acc vs iters](plots/acc_vs_iters.png) **Observations.** 1. The R-accuracy curves are *sharp*, not flat. For rec-1x4 at k=4, mean accuracy jumps from 0.06 (R=3) to 0.94 (R=4) and falls back to 0.35 (R=8): the model tuned itself to its trained unroll count, not to the hop count in the input. 2. rec-4x4 peaks at R=4 for both k=4 (1.00) and k=8 (1.00): with 16 serial block-applications it solves the whole training range at its trained R, and *loses* accuracy when given more unrolls (k=8: 0.73 at R=8, 0.39 at R=16) - a mild 'overthinking' regime. 3. The fixed model is the opposite: its best R is 1 (0.14/0.20 at k=4/8) and re-applying its own stack at R>1 only hurts - extra depth without training the depth is useless. 4. Consequence: the length-generalisation deficit starts *inside* the training range - see 4.3b. ### 4.3b Within (and just beyond) the training range: accuracy per hop count The natural question - 'recurrent depth 4 should get hop count 4, right?' - is answered by per-k accuracy inside k=1..16 (1000 fresh mappings per cell, mean over seeds): | family | k=1 | k=2 | k=3 | k=4 | k=5 | k=6 | k=7 | k=8 | |---|---|---|---|---|---|---|---|---| | fixed-4 | 1.00 | 1.00 | 0.99 | 0.16 | 0.16 | 0.17 | 0.20 | 0.21 | | rec-1x4 | 1.00 | 1.00 | 1.00 | 0.95 | 0.71 | 0.56 | 0.50 | 0.37 | | rec-2x2 | 1.00 | 1.00 | 1.00 | 0.71 | 0.21 | 0.20 | 0.25 | 0.27 | | rec-1x-dyn | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.99 | 0.96 | 0.24 | | rec-4x4 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | | fixed-4-long | 1.00 | 1.00 | 1.00 | 0.99 | 0.93 | 0.23 | 0.28 | 0.29 | | rec-1x-aligned | 0.06 | 0.05 | 0.07 | 0.10 | 0.10 | 0.12 | 0.17 | 0.18 | | m4 | 0.25 | 0.24 | 0.24 | 0.27 | 0.25 | 0.25 | 0.29 | 0.31 | | m4+dense | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | | family | k=9 | k=10 | k=11 | k=12 | k=13 | k=14 | k=15 | k=16 | |---|---|---|---|---|---|---|---|---| | fixed-4 | 0.14 | 0.15 | 0.15 | 0.18 | 0.10 | 0.18 | 0.20 | 0.13 | | rec-1x4 | 0.14 | 0.13 | 0.14 | 0.13 | 0.14 | 0.11 | 0.16 | 0.09 | | rec-2x2 | 0.18 | 0.19 | 0.19 | 0.21 | 0.17 | 0.20 | 0.20 | 0.18 | | rec-1x-dyn | 0.18 | 0.15 | 0.18 | 0.24 | 0.19 | 0.22 | 0.22 | 0.16 | | rec-4x4 | 0.20 | 0.24 | 0.19 | 0.23 | 0.19 | 0.23 | 0.28 | 0.21 | | fixed-4-long | 0.21 | 0.19 | 0.22 | 0.21 | 0.22 | 0.21 | 0.25 | 0.22 | | rec-1x-aligned | 0.10 | 0.09 | 0.06 | 0.07 | 0.05 | 0.04 | 0.06 | 0.04 | | m4 | 0.14 | 0.10 | 0.09 | 0.12 | 0.08 | 0.07 | 0.08 | 0.07 | | m4+dense | 0.20 | 0.23 | 0.17 | 0.29 | 0.19 | 0.19 | 0.21 | 0.19 | ![per-k zoom](plots/perk_zoom.png) **Observations.** 1. k=1..3 are solved by every family (1.00): reading f(x) and very short chains are easy. 2. The capability frontier scales with serial block-applications: fixed-4 (4 applications) solves k<=3; rec-1x-dynamic (up to 8) solves k<=7 (0.96 at k=7); rec-4x4 (16) solves the entire training range k<=8 at 1.000; rec-1x4 (4 unrolls of 1 block) manages k=4 at 0.95 and decays to 0.37 by k=8. So recurrence with enough unrolls does buy hop count - roughly one hop per block-application, plus a constant. 3. But the frontier stops at the training-data limit (k<=8): even rec-4x4, with twice the serial depth k=8 needs, does not extend beyond it (0.20-0.28 at k=9..16). Capacity alone does not give length generalisation when the data never demanded it. 4. rec-1x4's k=8 accuracy (0.37) is low not because 8 hops exceed recurrence but because it was trained with R=4 and cannot turn 'k=8' into 'unroll 8 times' (at R=8 it scores only 0.14). This is exactly the gap the aligned-unroll experiment (4.7) targets. ### 4.4 Accuracy vs state-space size **S'** (S-prime) is the state-space size used *at evaluation time*. Training always used S=64: the state-embedding table has 64 rows and the positional table 66 positions. To ask whether the learned procedure depends on the specific vocabulary, we evaluate on smaller state spaces S' in {16, 32}: mappings are generated over states {0..S'-1} only, which reuses trained embedding rows (no new ones are introduced). S' > 64 is impossible without adding embedding rows the model never trained - a limitation, not a finding. ![state size](plots/acc_vs_state_size.png) In-distribution accuracy on smaller state spaces (mean over k = 1..8): - fixed-4 (4 blocks x 1): S'=16: 0.218±0.022, S'=32: 0.116±0.011, S'=64: 0.484±0.005 - recurrent 1x4 (train R=4): S'=16: 0.221±0.063, S'=32: 0.102±0.062, S'=64: 0.756±0.180 **Observation:** accuracy degrades sharply and non-monotonically: fixed-4 drops from 0.48 at S'=64 to 0.12 at S'=32 and 0.22 at S'=16; rec-1x4 from 0.76 to 0.10 and 0.22. Relative to chance (1/S'), smaller state spaces are far harder (roughly 3-4x chance, vs ~30x chance at S'=64), so the learned solution is strongly tied to the trained state space - it is not a clean, vocabulary- independent algorithm. (The S'=16 > S'=32 inversion is unexplained; random functions on smaller domains enter cycles sooner, which may change the task's structure.) ### 4.5 Training dynamics ![training](plots/training_curves.png) **Observation:** rec-4x4 reaches ~1.0 in-distribution by ~7.5k steps; fixed-4 plateaus near ~0.47 and is still softening at 25k; fixed-4-long reaches 0.72 by 100k. The dense v2 variant reaches 1.0 by ~12.5k despite its variable-depth readout - the dense targets act as a curriculum (unroll 1 learns the lookup first). The learning *rate* differs dramatically even when parameters are identical. ### 4.6 Full table (per seed) ```csv family,seed,params_total,train_flops_total,acc_k1_8,acc_k16,acc_k32,acc_k64,best_acc_k64_over_iters,best_R_k64 fixed4x1,0,829696,2.185e+15,0.4908,0.1530,0.1700,0.1190,0.1460,1 fixed4x1,1,829696,2.185e+15,0.4849,0.1330,0.2150,0.0920,0.1080,1 fixed4x1,2,829696,2.185e+15,0.4840,0.1130,0.2260,0.0960,0.1140,1 rec1x4,0,234880,2.185e+15,0.7473,0.1350,0.1710,0.1040,0.1180,4 rec1x4,1,234880,2.185e+15,0.5871,0.1060,0.1490,0.1830,0.1850,4 rec1x4,2,234880,2.185e+15,0.9488,0.0410,0.0720,0.0330,0.0410,5 rec2x2,0,433152,2.185e+15,0.5054,0.1850,0.1650,0.1240,0.1340,2 rec2x2,1,433152,2.185e+15,0.6161,0.1530,0.1390,0.1950,0.2050,2 rec2x2,2,433152,2.185e+15,0.6199,0.1870,0.2220,0.2130,0.2470,2 rec1xdynamic,0,234880,2.726e+15,0.8928,0.2570,0.1970,0.1510,0.1820,8 rec1xdynamic,1,234880,2.726e+15,0.9103,0.1500,0.2020,0.0880,0.3190,5 rec1xdynamic,2,234880,2.726e+15,0.8930,0.0850,0.0210,0.0690,0.0850,7 rec4x4,0,829696,8.678e+15,1.0000,0.2340,0.1920,0.1780,0.2160,7 rec4x4,1,829696,8.678e+15,1.0000,0.2660,0.2070,0.1810,0.1910,6 rec4x4,2,829696,8.678e+15,1.0000,0.1270,0.2100,0.3940,0.4010,5 fixed4-long,0,829696,8.741e+15,0.7069,0.2580,0.2350,0.1530,0.1570,1 fixed4-long,1,829696,8.741e+15,0.7244,0.1810,0.1340,0.1890,0.1860,1 rec1aligned,0,240256,2.456e+15,0.1024,0.0470,0.0180,0.0150,0.1260,5 rec1aligned,1,240256,2.456e+15,0.1093,0.0480,0.0400,0.0220,0.1850,7 rec1aligned,2,240256,2.456e+15,0.1059,0.0190,0.0220,0.0150,0.1070,4 rec1m4,0,242048,4.620e+15,0.2638,0.0790,0.0420,0.0130,0.1700,6 rec1m4,1,242048,4.620e+15,0.2649,0.0470,0.0650,0.0200,0.2200,5 rec1m4,2,242048,4.620e+15,0.2580,0.0850,0.0990,0.0540,0.1970,7 rec1m4dense,0,440320,4.620e+15,1.0000,0.2440,0.1550,0.1160,0.4380,4 rec1m4dense,1,440320,4.620e+15,1.0000,0.1250,0.1120,0.0380,0.4380,4 rec1m4dense,2,440320,4.620e+15,1.0000,0.2000,0.0740,0.0700,0.4340,4 ``` ### 4.7 Aligned unrolls: training with R = k (one unroll per hop) Motivated by 4.3b: the models implicitly tuned themselves to their trained unroll count, and rec-1x4 fails k=8 mainly because it cannot turn the k token into 'unroll 8 times'. The aligned variant removes that gap *by construction*: during training every batch uses a single k ~ U{1..8} and runs exactly R = k unrolls, so 'one unroll per hop' is the only consistent solution. Everything else (architecture, data, optimizer, lr, 25k steps) matches rec-1x4; 3 seeds. At evaluation the model reads out after R = k unrolls for every k - including k > 8, which is the real test. *Predictions registered before the runs:* (a) if alignment is the missing ingredient, rec-1x-aligned should reach ~1.0 across the full k=1..8 range (rec-1x4 manages only 0.37 at k=8) and extend beyond k=8 better than the other 1-block families, since R=k is in-distribution for every k; (b) its fixed-R sweep should look like a staircase (accuracy high iff k <= R) rather than peaking at one R. *Results:* - in-distribution accuracy (k=1..8 mean): 0.106±0.003 - capability frontier (last k with mean acc >= 0.9): **k=0** (rec-1x4: k=4; rec-4x4 solves the full training range) - beyond the training range: k=16 0.038, k=32 0.027, k=64 0.017 - staircase check on the fixed-R sweep: mean accuracy where k <= R minus where k > R = +0.059 (large positive = the model really does one-unroll-per-hop) - even k=1 (a single unroll: read f(x) and stop) is unsolved at 25k steps: mean acc 0.059 - the model has not even learned the easiest aligned case, so this is optimisation, not a capacity, failure - prediction (a) **not supported** - prediction (b) **not supported** - **100k-step probe (1 seed, 4x budget)**: id acc 0.235, k=16 0.035, k=64 0.015 -> **a slow learner that plateaus**: it climbed 0.10 -> 0.235 by 60k steps, then stayed flat for the last 40k (slope ~0), so the aligned objective is roughly 4-10x slower to optimise than fixed-R training and did not break out of its second plateau by 100k. Why: with supervision only at the end of the chain, the model faces a credit-assignment problem across depths - it must learn to do one hop per unroll AND to stop exactly at depth k, with no per-step signal telling it either. Even the easiest case (k=1: one unroll) stayed unsolved at 25k steps. ### 4.8 v2 aligned variants: margin R = k+4, stem layer, dense supervision The 4.7 diagnosis says the R=k objective fails because of (i) exact halting at depth k, and (ii) no per-step signal. The v2 variants (user design) target both, separately: - **A / m4** (3 seeds): R = k + 4 with end-of-chain supervision only. The margin removes exact halting - the model needs 'at least k unrolls', and can idle for the last 4. Everything else identical to rec-1x4. - **B / m4+dense** (3 seeds): R = k + 4, one untied stem block before the loop, and dense supervision - every unroll t gets an auxiliary head predicting f^{min(t,k)}(x), i.e. each unroll does its own hop and then holds. This is the per-step signal the R=k run lacked, and it acts as a curriculum: unroll 1 only has to learn the lookup, later unrolls inherit it. At evaluation both read out at R = k + 4 for every k - the trained inference mode - including k > 8. *Predictions registered before the runs:* - (P1) margin alone does **not** rescue learning (the credit- assignment problem, not exact halting, was the blocker); it should learn faster than R=k but plateau low. - (P2) dense + stem **does** rescue: >= 0.9 mean accuracy across k = 1..8, and a fixed-R sweep showing a staircase (accuracy high iff k <= R - 4) rather than a peak at one R. - (P3) if the fix is real, B extends beyond k=8 better than every existing family (R = k+4 is its trained mode at any k), bounded by OOD step embeddings (unroll indices > 11 were never trained). *Results:* **A / m4 (margin, end-of-chain loss):** - in-distribution (k=1..8 mean): 0.262±0.004 - frontier (last k with >=0.9 mean acc): **k=0** - k=16 0.070, k=32 0.069, k=64 0.029 - staircase check (acc high iff k <= R-4): +0.042 **B / m4+dense (margin + stem + dense):** - in-distribution (k=1..8 mean): 1.000±0.000 - frontier (last k with >=0.9 mean acc): **k=8** - k=16 0.190, k=32 0.114, k=64 0.075 - staircase check (acc high iff k <= R-4): +0.634 - (P1) **supported** - (P2) **supported** - (P3) **not supported** ## 5. Conclusions 1. **Recurrent depth buys learnability that neither parameters nor extra training compute provide** *(observation)*: at identical parameter counts rec-4x4 reaches 1.000 in-distribution vs 0.487 for fixed-4, and fixed-4-long still only 0.716 after 4x the total training FLOPs. 2. **The loop structure itself matters at fixed compute** *(observation, with caveats)*: rec-1x4 learns the trained task better than fixed-4 (0.761 vs 0.487) with 4x fewer parameters and identical training FLOPs - but seed variance is large (0.59-0.95), so this needs more seeds to be a firm claim. 3. **Serial depth converts to hop capability with sharply diminishing marginal returns at the data limit** *(observation)*: the per-k frontier scales at roughly one hop per block- application (4.3b) until it hits the training-data limit k<=8, after which returns collapse to ~zero for every family (k=9..16 accuracy 0.04-0.28 regardless of 4 vs 16 serial applications). 4. **Inference-time unrolls are tuned to the trained R, not to the task's hop count - unless the loss makes the alignment explicit** *(observation)*: fixed-R models peak sharply at their training-time R (rec-1x4 at k=4: 0.06 at R=3 -> 0.94 at R=4) and more unrolls usually hurt (rec-4x4 at k=8: 1.00 at R=4 -> 0.39 at R=16). Training with R=k alone does not remove this coupling (4.7), but dense per-iteration supervision with a +4 margin does: the m4+dense model's fixed-R sweep shows a clean staircase (+0.634) instead of a peak at one R (4.8). 5. **Dense per-iteration supervision - not margin, not architecture - is what makes aligned training work** *(observation)*: the ablation ladder R=k (0.106) -> +margin (0.262) -> +margin+stem+dense (1.000) isolates per-step signal as the decisive ingredient; margin alone leaves the model at 0.26 because the cross-depth credit-assignment problem remains. 6. **Length generalisation to untrained hop counts is weak everywhere** *(observation)*: best mean k=64 accuracy ~0.27 (rec-4x4), vs 1.0 in-distribution. Consistent with **theory**: constant-depth transformers sit in TC^0/AC^0 and cannot compute iterated composition f^k for unbounded k without growing depth or serial steps (Merrill & Sabharwal 2305.15401; Li et al. 2402.12875) - and our data-coverage result (conclusion 3) shows the binding constraint at this scale is the training distribution, not the architecture. 7. **Single-seed comparisons of OOD generalisation are unreliable at this scale** *(observation)*: rec-4x4's k=64 accuracy across seeds is 0.18 / 0.18 / 0.39; the dense variant's is 0.116 / 0.038 / 0.070. ### How could the OOD deficit be fixed? (discussion) Generalisation is the whole point, so it is worth being explicit about the levers - which this study tests and which it does not: 1. **Aligned / per-iteration supervision** - make the training computation match the task's serial structure. **Tested here (4.7-4.8) with a clear decomposition**: exact alignment (R=k) fails on credit assignment; a +4 margin helps a little; dense per-iteration supervision toward f^{min(t,k)}(x) with an untied stem fully rescues in-distribution learning. Supervising every unroll toward f^t(x) without the hold-rule (Fan et al. 2409.15647) remains untested. 2. **Adaptive halting** (ACT, Universal Transformer) - let the model decide how many unrolls to run per input, so R stops being a fixed hyperparameter. Untested here. 3. **Externalise the serial steps** - chain-of-thought/scratchpad: theory says serial *tokens* substitute for serial depth (Li et al. 2402.12875) and can grow with k at inference. Untested. 4. **Curriculum and data coverage** - the frontier in 4.3b tracks the training-data limit (k<=8) almost exactly - even for the fully-fixed m4+dense model (4.8) - so this is now the most promising lever: train on longer chains or a growing-k curriculum. Untested here. 5. **Scale** - the frontier also scales with serial depth (4.3b), so more parameters *and* more unrolls should extend it. Only partially tested (fixed-4-long). ## 6. Negative results, failures and caveats - At 3k steps (lr sweep) every model was near chance (~5-6%): the algorithm-learning phase is far slower than tiny-set overfitting (which reached 100% in ~100 steps). - fixed-4's in-distribution accuracy may not be fully converged at 25k steps (its curve was still softening); fixed-4-long at 100k partially addresses this but 0.72 < 1.0. - Evaluation state sizes are limited to S' ≤ 64 because the vocabulary/positional tables are fixed at training time. - The rec-1x-dynamic family trains at ~5 average block- applications per step (R ~ U{2..8}), slightly above the matched 4 - flagged rather than hidden. - The aligned-unroll variant (R=k) is a negative result at this budget: ~0.10 in-distribution across 3 seeds at 25k steps, and a 100k-step probe still only reaches 0.24 - see 4.7. - Two duplicate jobs created by interrupted tool calls were cancelled (~$0.30 wasted). - Dependency pin: trackio 0.42.0 requires huggingface-hub<2.0; resolved with huggingface_hub==1.33.0. ## 7. What remains - More seeds to tighten the OOD family means (esp. rec-4x4 and the m4+dense variant, whose k=64 seeds span 0.038-0.116). - Growing-k curriculum on the m4+dense recipe (train k up to 16, or a growing schedule): the frontier tracks the data limit in every family that learns, so this is the most promising lever - but the OOD step embeddings (>11) would need training coverage too. - Supervising every unroll toward f^t(x) without the hold-rule (Fan et al. 2409.15647): our dense variant uses hold-after-k targets f^{min(t,k)}(x); the no-hold variant is untested and would compose chains past k - possibly the strongest test of whether the recurrence can generalise at all. - Deeper aligned stacks (rec-4x m4+dense) and adaptive halting. - Larger-scale runs (see the discussion in section 5). ## 8. Reproduce See README.md. `run_all.py` drives lrsweep/stage1/stage2/stage3/ stage4/stage5/eval: stage1-2 are the matched-FLOP and iso-param families, stage3-4 the R=k aligned variant (25k and 100k steps), stage5 the v2 margin/stem/dense variants. `evaluate.py` produces the per-checkpoint JSONs; `make_report.py` generates this report and every figure. Live dashboards: eac123/ptr-recurrent-depth-trackio.