REPORT.md

Technical report: recurrent depth vs fixed depth on pointer chasing

(Everything below marked observation is measured from the runs in results/. Statements about what transformers can compute in principle are theory and cite prior work.)

TL;DR

We train nine families of tiny transformers (3 seeds each, except fixed-4-long: 2) on a synthetic multi-hop task: given a random mapping f, a start state x and a hop count k, output f^k(x). Training only ever uses k ≤ 8; evaluation uses k up to 64 with freshly generated mappings. Observations: (1) recurrent depth (a weight-tied block applied R times) learns the trained task far better than a fixed-depth transformer at matched parameters and even at matched total training compute; (2) end-of-chain aligned supervision (R=k) fails on credit assignment, but adding a +4 unroll margin, an untied stem layer and dense per-iteration supervision fully rescues it (0.106 -> 1.000 in-distribution, sections 4.7-4.8); (3) no model in this budget class generalises well to untrained hop counts (best mean acc ~0.27 at k=64 vs 1.0 in-distribution): the capability frontier tracks the training-data limit in every family that learns, so the remaining bottleneck is data coverage, not architecture.

1. The task

Each example is a fresh, uniformly random function f on a state space of size S=64, a uniformly random start state x, and a hop count k. The correct answer is f applied k times to x. During training k ~ Uniform{1..8}; at evaluation k ranges up to 64 (out of distribution) and every mapping is newly generated, so nothing can be memorised. Shortcut controls: f is uniform random (no exploitable regular structure, cf. arXiv:2210.10749); the marginal distribution of the answer is ~uniform for every k (verified: total-variation < 0.02, no majority-class shortcut); k is the only token that correlates with difficulty. A deterministic lookup solver scores 1.0 by construction and is re-run at every evaluation as a pipeline integrity check.

Example 1 (small state space S=8, so it fits on the page).

The mapping f is drawn uniformly at random; with this seed it is

f(0)=2  f(1)=5  f(2)=2  f(3)=7  f(4)=0  f(5)=3  f(6)=6  f(7)=2

Start state x=1, hop count k=3. Applying f step by step:

f^1(1) = f(1)  = 7
f^2(1) = f(7)  = 2
f^3(1) = f(2)  = 7   <- the required answer

So the answer is state 7. Note how the chain entered a cycle (2 -> 7 -> 2 -> ...): random finite mappings always fall into cycles, which is exactly why a model that only looks up f(x) fails once k grows past the cycle structure.

The model never sees the chain. It sees one flat token sequence (S+2 = 10 tokens here, 66 for S=64):

input : [2, 5, 2, 7, 0, 3, 6, 2, | 3, 3]
          f(0) f(1) ...  f(7)      x  k
target: the state id of f^k(x), predicted at the final position

The position of each f-token encodes which state it is the image of (state i is always at position i), so no 'arrow' tokens are needed. The final two tokens are the start state x and the hop count k; the model must combine all three pieces of information.

Example 2 (training-scale S=64). Same construction, but only the first few mapping entries are shown:

f = [54, 11, 1, 40, 23, 29, 5, 23, 41, 22, 53, 50, ...]  (64 entries)
x = 61, k = 7  ->  answer f^7(61) = 41

Every training batch draws a fresh random f, a fresh x and a fresh k ~ Uniform{{1..8}} for every example, so there is nothing to memorise: the only way to score above chance is to learn the algorithm itself.

A note on what 'length' means here: the input sequence is always S+2 = 66 tokens regardless of k, so the OOD axis is the value of the hop-count token, not the sequence length. Generalising from k ≤ 8 to k = 64 therefore requires the model to apply its learned procedure 8x more times than it ever saw during training.

2. The models

All models share one implementation (ptr/models.py): a pre-LN transformer block (4 heads, d_ff = 4*d_model, GELU MLP, bidirectional attention, learned positional embeddings) applied n_iters times. Two knobs distinguish the families:

• depth = number of distinct blocks in the stack. In the fixed model each block has its own weights and is applied once.
• n_iters (R) = how many times the same stack is applied. Each unroll adds a learned step embedding so the block can tell pass t from pass t+1. At initialisation the blocks are the identity (zero-initialised output projections), which keeps gradients stable through long unroll chains (cf. arXiv:2604.07822).

So 'fixed-4' = depth 4, R=1; 'rec-1x4' = one shared block unrolled R=4 times; 'rec-4x4' = a 4-block stack unrolled R=4 times. The input encoding, output head, optimizer and data pipeline are identical everywhere (d_model=128, batch 256, AdamW lr=1e-3 chosen by a 6-run sweep, cosine decay, 25k steps unless stated).

The nine families and what is matched

family	blocks x unrolls	params	per-step block-applications	total training FLOPs	seeds
fixed-4	4 x 1	829,696	4	2.19e15	3
rec-2x2	2 x 2	433,152	4	2.19e15	3
rec-1x4	1 x 4	234,880	4	2.19e15	3
rec-1x-dynamic	1 x R~U{2..8}	234,880	~5 avg	2.73e15	3
rec-4x4	4 x 4	829,696	16	8.68e15	3
fixed-4-long	4 x 1, 100k steps	829,696	4	8.74e15	2
rec-1x-aligned	1 x k (R = k)	240,256	~4.5 avg	2.46e15	3
rec-1x-m4	1 x (k+4)	242,048	~8.5 avg	4.62e15	3
rec-1x-m4+dense	1 x (k+4), stem, dense loss	440,320	~9.5 avg	4.62e15*	3

Three comparison axes follow from this table:

1. Matched-FLOP family (rows 1-3): every model executes exactly 4 block-applications per training step and is trained for the same number of steps, so total training FLOPs are identical while parameters differ 4:2:1. Question: does the loop structure itself matter when compute is fixed?
2. Iso-parameter pair (fixed-4 vs rec-4x4): identical parameter counts, but rec-4x4 does 4x the per-step work. Question: given the same weights, does iterating them help?
3. Iso-FLOP pair (rec-4x4 vs fixed-4-long): identical total training FLOPs - fixed-4-long simply trains 4x longer. Question: can extra training compute substitute for recurrent depth?

Rows 7-9 (sections 4.7-4.8) test a fourth axis - supervision and halting structure: whether training the unroll count to match the task (R = k or k+4), with and without per-step supervision, changes what the recurrence learns. (The m4+dense row's stored FLOPs under-counts its stem by ~12%; flagged rather than silently corrected.)

3. What is evaluated

Every evaluation cell uses freshly generated mappings (nothing from training is reused) and 1000 examples per cell; accuracy is exact match on the answer state (chance = 1/64 ≈ 0.016). The varied parameters are:

• Chain length k ∈ 1..64 at the model's training-time unroll count: the OOD length-generalisation curve.
• Inference-time unroll count R ∈ 1..16, crossed with k ∈ {1,2,4,8,16,32,64}: 'more or fewer iterations than trained'. For the fixed model, R > 1 re-applies its own block stack (effectively deeper than trained) - also measured.
• State-space size S' ∈ {16, 32, 64}: evaluation on subsets of the trained vocabulary/positions (larger S' would need new embedding rows the model never had; noted as a limitation).
• Training seed (3 per family) and training compute (the three budget axes above).

4. Results

4.1 Headline (at training-time unrolls)

family	acc k=1..8	acc k=16	acc k=32	acc k=64	best acc k=64 over R
fixed-4 (4 blocks x 1)	0.487±0.004	0.133±0.020	0.204±0.030	0.102±0.015	0.123±0.020
recurrent 1x4 (train R=4)	0.761±0.181	0.094±0.048	0.131±0.052	0.107±0.075	0.115±0.072
recurrent 2x2 (train R=2)	0.580±0.065	0.175±0.019	0.175±0.042	0.177±0.047	0.195±0.057
recurrent 1x dynamic R~U{2..8}	0.899±0.010	0.164±0.087	0.140±0.103	0.103±0.043	0.195±0.118
recurrent 4x4 (train R=4)	1.000±0.000	0.209±0.073	0.203±0.010	0.251±0.124	0.269±0.115
fixed-4 long (100k steps)	0.716±0.012	0.220±0.054	0.185±0.071	0.171±0.025	0.172±0.021
rec-1x-aligned (train R=k)	0.106±0.003	0.038±0.016	0.027±0.012	0.017±0.004	0.139±0.041
rec-1x m4 (R=k+4, end-of-chain loss)	0.262±0.004	0.070±0.020	0.069±0.029	0.029±0.022	0.196±0.025
rec-1x m4+dense (R=k+4, stem, dense loss)	1.000±0.000	0.190±0.060	0.114±0.041	0.075±0.039	0.437±0.002

Reading the bar chart: blue = how well the model learned the task it was trained on; red = how far that skill stretches to 8x longer chains. The gap between them is the length-generalisation deficit, and it is large for every family.

4.2 Accuracy vs chain length

All curves bend downward after the training range (k ≤ 8, dashed line). Observation: no family learned a clean length-generalising algorithm in this budget class.

4.3 Accuracy vs inference-time unrolls (the k x R grid)

Two different quantities are crossed in this figure, and keeping them apart is the whole point:

• k — task depth. How many times f must be applied to answer the question. It is part of the input (the k token) and is fixed along each line of the figure.
• R — the recurrent loop parameter. How many times the model iterates its own block stack before reading out an answer. It is a free choice made at inference time: the trained model is frozen and we simply run it for more or fewer unrolls. The dashed vertical line marks the R each family was trained with.

What is being tested. If a recurrent model learned the abstraction 'apply one unroll per hop', accuracy should rise with R and peak near R = k. If instead it learned a fixed amount of computation (whatever R it was trained with), accuracy should peak at the trained R regardless of k and fall off on both sides.

Observations.

1. The R-accuracy curves are sharp, not flat. For rec-1x4 at k=4, mean accuracy jumps from 0.06 (R=3) to 0.94 (R=4) and falls back to 0.35 (R=8): the model tuned itself to its trained unroll count, not to the hop count in the input.
2. rec-4x4 peaks at R=4 for both k=4 (1.00) and k=8 (1.00): with 16 serial block-applications it solves the whole training range at its trained R, and loses accuracy when given more unrolls (k=8: 0.73 at R=8, 0.39 at R=16) - a mild 'overthinking' regime.
3. The fixed model is the opposite: its best R is 1 (0.14/0.20 at k=4/8) and re-applying its own stack at R>1 only hurts - extra depth without training the depth is useless.
4. Consequence: the length-generalisation deficit starts inside the training range - see 4.3b.

4.3b Within (and just beyond) the training range: accuracy per hop count

The natural question - 'recurrent depth 4 should get hop count 4, right?' - is answered by per-k accuracy inside k=1..16 (1000 fresh mappings per cell, mean over seeds):

family	k=1	k=2	k=3	k=4	k=5	k=6	k=7	k=8
fixed-4	1.00	1.00	0.99	0.16	0.16	0.17	0.20	0.21
rec-1x4	1.00	1.00	1.00	0.95	0.71	0.56	0.50	0.37
rec-2x2	1.00	1.00	1.00	0.71	0.21	0.20	0.25	0.27
rec-1x-dyn	1.00	1.00	1.00	1.00	1.00	0.99	0.96	0.24
rec-4x4	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
fixed-4-long	1.00	1.00	1.00	0.99	0.93	0.23	0.28	0.29
rec-1x-aligned	0.06	0.05	0.07	0.10	0.10	0.12	0.17	0.18
m4	0.25	0.24	0.24	0.27	0.25	0.25	0.29	0.31
m4+dense	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00

family	k=9	k=10	k=11	k=12	k=13	k=14	k=15	k=16
fixed-4	0.14	0.15	0.15	0.18	0.10	0.18	0.20	0.13
rec-1x4	0.14	0.13	0.14	0.13	0.14	0.11	0.16	0.09
rec-2x2	0.18	0.19	0.19	0.21	0.17	0.20	0.20	0.18
rec-1x-dyn	0.18	0.15	0.18	0.24	0.19	0.22	0.22	0.16
rec-4x4	0.20	0.24	0.19	0.23	0.19	0.23	0.28	0.21
fixed-4-long	0.21	0.19	0.22	0.21	0.22	0.21	0.25	0.22
rec-1x-aligned	0.10	0.09	0.06	0.07	0.05	0.04	0.06	0.04
m4	0.14	0.10	0.09	0.12	0.08	0.07	0.08	0.07
m4+dense	0.20	0.23	0.17	0.29	0.19	0.19	0.21	0.19

Observations.

1. k=1..3 are solved by every family (1.00): reading f(x) and very short chains are easy.
2. The capability frontier scales with serial block-applications: fixed-4 (4 applications) solves k<=3; rec-1x-dynamic (up to 8) solves k<=7 (0.96 at k=7); rec-4x4 (16) solves the entire training range k<=8 at 1.000; rec-1x4 (4 unrolls of 1 block) manages k=4 at 0.95 and decays to 0.37 by k=8. So recurrence with enough unrolls does buy hop count - roughly one hop per block-application, plus a constant.
3. But the frontier stops at the training-data limit (k<=8): even rec-4x4, with twice the serial depth k=8 needs, does not extend beyond it (0.20-0.28 at k=9..16). Capacity alone does not give length generalisation when the data never demanded it.
4. rec-1x4's k=8 accuracy (0.37) is low not because 8 hops exceed recurrence but because it was trained with R=4 and cannot turn 'k=8' into 'unroll 8 times' (at R=8 it scores only 0.14). This is exactly the gap the aligned-unroll experiment (4.7) targets.

4.4 Accuracy vs state-space size

S' (S-prime) is the state-space size used at evaluation time. Training always used S=64: the state-embedding table has 64 rows and the positional table 66 positions. To ask whether the learned procedure depends on the specific vocabulary, we evaluate on smaller state spaces S' in {16, 32}: mappings are generated over states {0..S'-1} only, which reuses trained embedding rows (no new ones are introduced). S' > 64 is impossible without adding embedding rows the model never trained - a limitation, not a finding.

In-distribution accuracy on smaller state spaces (mean over k = 1..8):

• fixed-4 (4 blocks x 1): S'=16: 0.218±0.022, S'=32: 0.116±0.011, S'=64: 0.484±0.005
• recurrent 1x4 (train R=4): S'=16: 0.221±0.063, S'=32: 0.102±0.062, S'=64: 0.756±0.180

Observation: accuracy degrades sharply and non-monotonically: fixed-4 drops from 0.48 at S'=64 to 0.12 at S'=32 and 0.22 at S'=16; rec-1x4 from 0.76 to 0.10 and 0.22. Relative to chance (1/S'), smaller state spaces are far harder (roughly 3-4x chance, vs ~30x chance at S'=64), so the learned solution is strongly tied to the trained state space - it is not a clean, vocabulary- independent algorithm. (The S'=16 > S'=32 inversion is unexplained; random functions on smaller domains enter cycles sooner, which may change the task's structure.)

4.5 Training dynamics

Observation: rec-4x4 reaches ~1.0 in-distribution by ~7.5k steps; fixed-4 plateaus near ~0.47 and is still softening at 25k; fixed-4-long reaches 0.72 by 100k. The dense v2 variant reaches 1.0 by ~12.5k despite its variable-depth readout - the dense targets act as a curriculum (unroll 1 learns the lookup first). The learning rate differs dramatically even when parameters are identical.

4.6 Full table (per seed)

family,seed,params_total,train_flops_total,acc_k1_8,acc_k16,acc_k32,acc_k64,best_acc_k64_over_iters,best_R_k64
fixed4x1,0,829696,2.185e+15,0.4908,0.1530,0.1700,0.1190,0.1460,1
fixed4x1,1,829696,2.185e+15,0.4849,0.1330,0.2150,0.0920,0.1080,1
fixed4x1,2,829696,2.185e+15,0.4840,0.1130,0.2260,0.0960,0.1140,1
rec1x4,0,234880,2.185e+15,0.7473,0.1350,0.1710,0.1040,0.1180,4
rec1x4,1,234880,2.185e+15,0.5871,0.1060,0.1490,0.1830,0.1850,4
rec1x4,2,234880,2.185e+15,0.9488,0.0410,0.0720,0.0330,0.0410,5
rec2x2,0,433152,2.185e+15,0.5054,0.1850,0.1650,0.1240,0.1340,2
rec2x2,1,433152,2.185e+15,0.6161,0.1530,0.1390,0.1950,0.2050,2
rec2x2,2,433152,2.185e+15,0.6199,0.1870,0.2220,0.2130,0.2470,2
rec1xdynamic,0,234880,2.726e+15,0.8928,0.2570,0.1970,0.1510,0.1820,8
rec1xdynamic,1,234880,2.726e+15,0.9103,0.1500,0.2020,0.0880,0.3190,5
rec1xdynamic,2,234880,2.726e+15,0.8930,0.0850,0.0210,0.0690,0.0850,7
rec4x4,0,829696,8.678e+15,1.0000,0.2340,0.1920,0.1780,0.2160,7
rec4x4,1,829696,8.678e+15,1.0000,0.2660,0.2070,0.1810,0.1910,6
rec4x4,2,829696,8.678e+15,1.0000,0.1270,0.2100,0.3940,0.4010,5
fixed4-long,0,829696,8.741e+15,0.7069,0.2580,0.2350,0.1530,0.1570,1
fixed4-long,1,829696,8.741e+15,0.7244,0.1810,0.1340,0.1890,0.1860,1
rec1aligned,0,240256,2.456e+15,0.1024,0.0470,0.0180,0.0150,0.1260,5
rec1aligned,1,240256,2.456e+15,0.1093,0.0480,0.0400,0.0220,0.1850,7
rec1aligned,2,240256,2.456e+15,0.1059,0.0190,0.0220,0.0150,0.1070,4
rec1m4,0,242048,4.620e+15,0.2638,0.0790,0.0420,0.0130,0.1700,6
rec1m4,1,242048,4.620e+15,0.2649,0.0470,0.0650,0.0200,0.2200,5
rec1m4,2,242048,4.620e+15,0.2580,0.0850,0.0990,0.0540,0.1970,7
rec1m4dense,0,440320,4.620e+15,1.0000,0.2440,0.1550,0.1160,0.4380,4
rec1m4dense,1,440320,4.620e+15,1.0000,0.1250,0.1120,0.0380,0.4380,4
rec1m4dense,2,440320,4.620e+15,1.0000,0.2000,0.0740,0.0700,0.4340,4

4.7 Aligned unrolls: training with R = k (one unroll per hop)

Motivated by 4.3b: the models implicitly tuned themselves to their trained unroll count, and rec-1x4 fails k=8 mainly because it cannot turn the k token into 'unroll 8 times'. The aligned variant removes that gap by construction: during training every batch uses a single k ~ U{1..8} and runs exactly R = k unrolls, so 'one unroll per hop' is the only consistent solution. Everything else (architecture, data, optimizer, lr, 25k steps) matches rec-1x4; 3 seeds. At evaluation the model reads out after R = k unrolls for every k - including k > 8, which is the real test.

Predictions registered before the runs: (a) if alignment is the missing ingredient, rec-1x-aligned should reach ~1.0 across the full k=1..8 range (rec-1x4 manages only 0.37 at k=8) and extend beyond k=8 better than the other 1-block families, since R=k is in-distribution for every k; (b) its fixed-R sweep should look like a staircase (accuracy high iff k <= R) rather than peaking at one R.

Results:

• in-distribution accuracy (k=1..8 mean): 0.106±0.003
• capability frontier (last k with mean acc >= 0.9): k=0 (rec-1x4: k=4; rec-4x4 solves the full training range)
• beyond the training range: k=16 0.038, k=32 0.027, k=64 0.017
• staircase check on the fixed-R sweep: mean accuracy where k <= R minus where k > R = +0.059 (large positive = the model really does one-unroll-per-hop)
• even k=1 (a single unroll: read f(x) and stop) is unsolved at 25k steps: mean acc 0.059 - the model has not even learned the easiest aligned case, so this is optimisation, not a capacity, failure
• prediction (a) not supported
• prediction (b) not supported
• 100k-step probe (1 seed, 4x budget): id acc 0.235, k=16 0.035, k=64 0.015 -> a slow learner that plateaus: it climbed 0.10 -> 0.235 by 60k steps, then stayed flat for the last 40k (slope ~0), so the aligned objective is roughly 4-10x slower to optimise than fixed-R training and did not break out of its second plateau by 100k. Why: with supervision only at the end of the chain, the model faces a credit-assignment problem across depths - it must learn to do one hop per unroll AND to stop exactly at depth k, with no per-step signal telling it either. Even the easiest case (k=1: one unroll) stayed unsolved at 25k steps.

4.8 v2 aligned variants: margin R = k+4, stem layer, dense supervision

The 4.7 diagnosis says the R=k objective fails because of (i) exact halting at depth k, and (ii) no per-step signal. The v2 variants (user design) target both, separately:

• A / m4 (3 seeds): R = k + 4 with end-of-chain supervision only. The margin removes exact halting - the model needs 'at least k unrolls', and can idle for the last 4. Everything else identical to rec-1x4.
• B / m4+dense (3 seeds): R = k + 4, one untied stem block before the loop, and dense supervision - every unroll t gets an auxiliary head predicting f^{min(t,k)}(x), i.e. each unroll does its own hop and then holds. This is the per-step signal the R=k run lacked, and it acts as a curriculum: unroll 1 only has to learn the lookup, later unrolls inherit it.

At evaluation both read out at R = k + 4 for every k - the trained inference mode - including k > 8.

Predictions registered before the runs:

• (P1) margin alone does not rescue learning (the credit- assignment problem, not exact halting, was the blocker); it should learn faster than R=k but plateau low.
• (P2) dense + stem does rescue: >= 0.9 mean accuracy across k = 1..8, and a fixed-R sweep showing a staircase (accuracy high iff k <= R - 4) rather than a peak at one R.
• (P3) if the fix is real, B extends beyond k=8 better than every existing family (R = k+4 is its trained mode at any k), bounded by OOD step embeddings (unroll indices > 11 were never trained).

Results:

A / m4 (margin, end-of-chain loss):

• in-distribution (k=1..8 mean): 0.262±0.004
• frontier (last k with >=0.9 mean acc): k=0
• k=16 0.070, k=32 0.069, k=64 0.029
• staircase check (acc high iff k <= R-4): +0.042

B / m4+dense (margin + stem + dense):

• in-distribution (k=1..8 mean): 1.000±0.000
• frontier (last k with >=0.9 mean acc): k=8
• k=16 0.190, k=32 0.114, k=64 0.075
• staircase check (acc high iff k <= R-4): +0.634
• (P1) supported
• (P2) supported
• (P3) not supported

5. Conclusions

1. Recurrent depth buys learnability that neither parameters nor extra training compute provide (observation): at identical parameter counts rec-4x4 reaches 1.000 in-distribution vs 0.487 for fixed-4, and fixed-4-long still only 0.716 after 4x the total training FLOPs.
2. The loop structure itself matters at fixed compute (observation, with caveats): rec-1x4 learns the trained task better than fixed-4 (0.761 vs 0.487) with 4x fewer parameters and identical training FLOPs - but seed variance is large (0.59-0.95), so this needs more seeds to be a firm claim.
3. Serial depth converts to hop capability with sharply diminishing marginal returns at the data limit (observation): the per-k frontier scales at roughly one hop per block- application (4.3b) until it hits the training-data limit k<=8, after which returns collapse to ~zero for every family (k=9..16 accuracy 0.04-0.28 regardless of 4 vs 16 serial applications).
4. Inference-time unrolls are tuned to the trained R, not to the task's hop count - unless the loss makes the alignment explicit (observation): fixed-R models peak sharply at their training-time R (rec-1x4 at k=4: 0.06 at R=3 -> 0.94 at R=4) and more unrolls usually hurt (rec-4x4 at k=8: 1.00 at R=4 -> 0.39 at R=16). Training with R=k alone does not remove this coupling (4.7), but dense per-iteration supervision with a +4 margin does: the m4+dense model's fixed-R sweep shows a clean staircase (+0.634) instead of a peak at one R (4.8).
5. Dense per-iteration supervision - not margin, not architecture - is what makes aligned training work (observation): the ablation ladder R=k (0.106) -> +margin (0.262) -> +margin+stem+dense (1.000) isolates per-step signal as the decisive ingredient; margin alone leaves the model at 0.26 because the cross-depth credit-assignment problem remains.
6. Length generalisation to untrained hop counts is weak everywhere (observation): best mean k=64 accuracy ~0.27 (rec-4x4), vs 1.0 in-distribution. Consistent with theory: constant-depth transformers sit in TC^0/AC^0 and cannot compute iterated composition f^k for unbounded k without growing depth or serial steps (Merrill & Sabharwal 2305.15401; Li et al. 2402.12875) - and our data-coverage result (conclusion 3) shows the binding constraint at this scale is the training distribution, not the architecture.
7. Single-seed comparisons of OOD generalisation are unreliable at this scale (observation): rec-4x4's k=64 accuracy across seeds is 0.18 / 0.18 / 0.39; the dense variant's is 0.116 / 0.038 / 0.070.

How could the OOD deficit be fixed? (discussion)

Generalisation is the whole point, so it is worth being explicit about the levers - which this study tests and which it does not:

1. Aligned / per-iteration supervision - make the training computation match the task's serial structure. Tested here (4.7-4.8) with a clear decomposition: exact alignment (R=k) fails on credit assignment; a +4 margin helps a little; dense per-iteration supervision toward f^{min(t,k)}(x) with an untied stem fully rescues in-distribution learning. Supervising every unroll toward f^t(x) without the hold-rule (Fan et al. 2409.15647) remains untested.
2. Adaptive halting (ACT, Universal Transformer) - let the model decide how many unrolls to run per input, so R stops being a fixed hyperparameter. Untested here.
3. Externalise the serial steps - chain-of-thought/scratchpad: theory says serial tokens substitute for serial depth (Li et al. 2402.12875) and can grow with k at inference. Untested.
4. Curriculum and data coverage - the frontier in 4.3b tracks the training-data limit (k<=8) almost exactly - even for the fully-fixed m4+dense model (4.8) - so this is now the most promising lever: train on longer chains or a growing-k curriculum. Untested here.
5. Scale - the frontier also scales with serial depth (4.3b), so more parameters and more unrolls should extend it. Only partially tested (fixed-4-long).

6. Negative results, failures and caveats

• At 3k steps (lr sweep) every model was near chance (~5-6%): the algorithm-learning phase is far slower than tiny-set overfitting (which reached 100% in ~100 steps).
• fixed-4's in-distribution accuracy may not be fully converged at 25k steps (its curve was still softening); fixed-4-long at 100k partially addresses this but 0.72 < 1.0.
• Evaluation state sizes are limited to S' ≤ 64 because the vocabulary/positional tables are fixed at training time.
• The rec-1x-dynamic family trains at ~5 average block- applications per step (R ~ U{2..8}), slightly above the matched 4 - flagged rather than hidden.
• The aligned-unroll variant (R=k) is a negative result at this budget: ~0.10 in-distribution across 3 seeds at 25k steps, and a 100k-step probe still only reaches 0.24 - see 4.7.
• Two duplicate jobs created by interrupted tool calls were cancelled (~$0.30 wasted).
• Dependency pin: trackio 0.42.0 requires huggingface-hub<2.0; resolved with huggingface_hub==1.33.0.

7. What remains

• More seeds to tighten the OOD family means (esp. rec-4x4 and the m4+dense variant, whose k=64 seeds span 0.038-0.116).
• Growing-k curriculum on the m4+dense recipe (train k up to 16, or a growing schedule): the frontier tracks the data limit in every family that learns, so this is the most promising lever - but the OOD step embeddings (>11) would need training coverage too.
• Supervising every unroll toward f^t(x) without the hold-rule (Fan et al. 2409.15647): our dense variant uses hold-after-k targets f^{min(t,k)}(x); the no-hold variant is untested and would compose chains past k - possibly the strongest test of whether the recurrence can generalise at all.
• Deeper aligned stacks (rec-4x m4+dense) and adaptive halting.
• Larger-scale runs (see the discussion in section 5).

8. Reproduce

See README.md. run_all.py drives lrsweep/stage1/stage2/stage3/ stage4/stage5/eval: stage1-2 are the matched-FLOP and iso-param families, stage3-4 the R=k aligned variant (25k and 100k steps), stage5 the v2 margin/stem/dense variants. evaluate.py produces the per-checkpoint JSONs; make_report.py generates this report and every figure. Live dashboards: eac123/ptr-recurrent-depth-trackio.