[UPDATE] Update CBT-18.95M paper, model card, and 16-100d random extrapolation results

RESEARCH_PAPER.md

@@ -6,7 +6,7 @@
 
 ## Abstract
 
-Character-level Transformers, which avoid subword digit merging, still fail to extrapolate multi-digit addition under standard 1D positional encodings (scoring 4.6% on 14–16 digits and collapsing to 0.0% on 18- to 20-digit sums in our experiments). We show that supplying place-value coordinates through **Decoupled 2D-RoPE** removes this failure at small scale, and that the same coordinate scheme extends to multi-turn tool-calling dialogues without degrading them. In the 18.95M-parameter **Character-Based Transformer (CBT-18.95M)**, coordinates are assigned by a deterministic rule-based parser within the forward pass, coupled with coordinate shift masking and weight-tied recurrence ($R=3$, adopted as an architectural choice for parameter-efficient depth rather than an addition-specific mechanism). Evaluated on single training runs, CBT-18.95M achieves 100% on seen 1–15 digit additions, 100.0% (180/180) zero-shot exact match on unseen 16–40 digit random multi-carry sums within shifted coordinate support ($p \le 47$), 95.0% (57/60) at the support boundary (45 and 50 digits; pooled 98.8%, 237/240 on 16–50 digits), and 100% (26/26) structured tool dispatch on template-matched phrasing (50.0% exact match on out-of-distribution entity copying) within a 36 MB FP16 footprint.
+Character-level Transformers, which avoid subword digit merging, still fail to extrapolate multi-digit addition under standard 1D positional encodings (scoring 4.6–18.7% on 14–16 digits and collapsing to 0.0% on 18- to 20-digit sums across 1D baselines in our experiments). We show that supplying place-value coordinates through **Decoupled 2D-RoPE** removes this failure at small scale, and that the same coordinate scheme extends to multi-turn tool-calling dialogues without degrading them. In the 18.95M-parameter **Character-Based Transformer (CBT-18.95M)**, coordinates are assigned by a deterministic rule-based parser within the forward pass, coupled with coordinate shift masking and weight-tied recurrence ($R=3$, adopted as an architectural choice for parameter-efficient depth rather than an addition-specific mechanism). Evaluated on single training runs, CBT-18.95M achieves 100% on seen 1–15 digit additions, 100.0% (180/180) zero-shot exact match on unseen 16–40 digit random multi-carry sums within shifted coordinate support ($p \le 47$), 95.0% (57/60) at the support boundary (45 and 50 digits; pooled 98.8%, 237/240 on 16–50 digits), and 100% (26/26) structured tool dispatch on template-matched phrasing (50.0% exact match on out-of-distribution entity copying) within a compact 36 MB FP16 weight footprint (running at 141–394 character tokens/sec batched and 1.1–5.3 tokens/sec single-stream on CPU with an unoptimized Python coordinate parser).
 
 ---
 
@@ -124,7 +124,7 @@ Scores are raw accuracies with 95% Wilson confidence intervals (no pass/fail thr
 ---
 
 **Table 3: Itemized Arithmetic Length Extrapolation Benchmark (1 to 50 Digits), Separating Seen Training Lengths from Unseen Zero-Shot Extrapolation Lengths**  
-Standard 1D-Char and 50k-BPE models (18.91M) and Dual-Loop HRM-RoPE (1.56M) were trained on **1–12 digits** with a 512-token context window; our **CBT-18.95M** was trained on **1–15 digits** with a 1024-token context window. Part II (16–20 digits) is the clean comparison within the 512-token window: HRM-RoPE (1.56M) extrapolates to 20 digits using Decoupled 2D-RoPE without CBT's three-stage curriculum, whereas the 1D models fail. In Part III (25–50 digits), 25-digit traces ($\approx 360$ tokens) and 35-digit traces ($\approx 500$ tokens) fit within the 512-token context window of the 12d baselines, where 1D-Char and HRM-RoPE fail due to positional drift rather than context clipping; only 50-digit traces ($\approx 715$ tokens) exceed 512 tokens. Extending 1D-Char's frequency table to $T=1024$ still fails due to drift; for strictly equalized 1–15 digit, $T=1024$ training comparisons (including Ruoss et al. randomized positions), see Table 8B.
+Standard 1D-Char and 50k-BPE models (18.91M) and Dual-Loop HRM-RoPE (1.56M) were trained on **1–12 digits** with a 512-token context window; our **CBT-18.95M** was trained on **1–15 digits** with a 1024-token context window. Part II (16–20 digits) is the clean comparison within the 512-token window: HRM-RoPE (1.56M) extrapolates to 20 digits using Decoupled 2D-RoPE without CBT's three-stage curriculum, whereas the 1D models fail. In Part III (25–50 digits), 25-digit traces ($\approx 360$ tokens) and 35-digit traces ($\approx 500$ tokens) fit within the 512-token context window of the 12d baselines, where 1D-Char fails due to positional drift; 1.56M HRM-RoPE was trained without shift masking ($k=0$) and succeeds up to 20 digits, but fails at 25 and 35 digits because $p > 20$ exceeds its unshifted coordinate horizon; only 50-digit traces ($\approx 715$ tokens) exceed 512 tokens. Extending 1D-Char's frequency table to $T=1024$ still fails due to drift; for strictly equalized 1–15 digit, $T=1024$ training comparisons (including Ruoss et al. randomized positions), see Table 8B.
 
 | Digit Length | Training Exposure Status | Representative Test Problem | Expected Output | 18.91M 1D-Char | 18.91M 50k-BPE | 1.56M HRM-RoPE | CBT-18.95M (Ours) |
 | :--- | :--- | :--- | :---: | :---: | :---: | :---: | :---: |
@@ -145,8 +145,8 @@ Standard 1D-Char and 50k-BPE models (18.91M) and Dual-Loop HRM-RoPE (1.56M) were
 | 20 Digits | **Unseen OOD (+5d / +8d)** | $10^{19} + 10^{19}=$ | $2\times 10^{19}$ | `Loops 000... [FAIL]` | `Drift [FAIL]` | $2\times 10^{19}$ `[PASS]` | $2\times 10^{19}$ `[PASS]` |
 | 20 Digits | **Unseen OOD (+5d / +8d)** | $\underbrace{\texttt{999...999}}_{20} + \texttt{1}=$ | $10^{20}$ (21 digits) | `Loops 000... [FAIL]` | `Drift [FAIL]` | $10^{20}$ `[PASS]` | $10^{20}$ `[PASS]` |
 | **Part III: Unseen Ultra-Long Horizon Extrapolation ($25 \le N \le 50$ Digits)** | | | | | | | |
-| 25 Digits | **Unseen OOD (+10 Extra Digits)** | $\underbrace{\texttt{999...999}}_{25} + \texttt{1}=$ | $10^{25}$ (26 digits) | `Drift [FAIL]` | `Hallucinates [FAIL]` | `Drift [FAIL]` | $10^{25}$ `[PASS]` |
-| 35 Digits | **Unseen OOD (+20 Extra Digits)** | $\underbrace{\texttt{111...111}}_{35} + \underbrace{\texttt{888...888}}_{35}=$ | $35\times\texttt{'9'}$ | `Drift [FAIL]` | `Hallucinates [FAIL]` | `Drift [FAIL]` | $35\times\texttt{'9'}$ `[PASS]` |
+| 25 Digits | **Unseen OOD (+10 Extra Digits)** | $\underbrace{\texttt{999...999}}_{25} + \texttt{1}=$ | $10^{25}$ (26 digits) | `Drift [FAIL]` | `Hallucinates [FAIL]` | `Horizon [FAIL]` | $10^{25}$ `[PASS]` |
+| 35 Digits | **Unseen OOD (+20 Extra Digits)** | $\underbrace{\texttt{111...111}}_{35} + \underbrace{\texttt{888...888}}_{35}=$ | $35\times\texttt{'9'}$ | `Drift [FAIL]` | `Hallucinates [FAIL]` | `Horizon [FAIL]` | $35\times\texttt{'9'}$ `[PASS]` |
 | 50 Digits | **Unseen OOD (+35 Extra Digits)** | `123...890 + 876...109=` (50d) | $50\times\texttt{'9'}$ | `Drift [FAIL at T=1024]` | `N/A (Context: 715 > 512)` | `N/A (Context: 715 > 512)` | $50\times\texttt{'9'}$ `[PASS]` |
 | 50 Digits | **Unseen OOD (+35 Extra Digits)** | $\underbrace{\texttt{999...999}}_{50} + \texttt{1}=$ | $10^{50}$ (51 digits) | `Drift [FAIL at T=1024]` | `N/A (Context: 715 > 512)` | `N/A (Context: 715 > 512)` | $10^{50}$ `[PASS]` |
 
@@ -237,9 +237,9 @@ Tables 2, 3, 3B, 4, and 5 present the evaluation of our final **CBT-18.95M** mod
 ---
 
 **Table 6: The Tokenization and Positional Geometry Matrix**  
-Comparing our 2D Matrix Character + Control-Token CBT formulation against 1D Character models, Subword BPE models (4k and 50k vocabularies), Digit-Split 4k BPE, and Learned 2D positional tables, explicitly contrasting **Seen Training Digits** vs. **Unseen Extrapolation Digits**. Vocabulary sizes differ by variant: 115 is the full CBT-18.95M tokenizer, 111 the CBT-200k and Learned-2D variants, and 27 the digit-and-symbol character models. All rows showing `/50` in the Tool Calling column were evaluated on the identical 50-dialogue structured tool suite (6 domains), whereas CBT-18.95M was evaluated on the 26-case 9-schema 1-hop suite of Table 4. Rows were trained with different digit ranges and data and are not a controlled $2\times2$ study. The unseen-sum column pools 14–20 digits, whereas Table 7 reports 14–16 digits separately (hence 2.3% here and 4.6% there for the 1D character model).
+Comparing our 2D Matrix Character + Control-Token CBT formulation against 1D Character models, Subword BPE models (4k and 50k vocabularies), Digit-Split 4k BPE, and Learned 2D positional tables, explicitly contrasting **Seen Training Digits** vs. **Unseen Extrapolation Digits**. Vocabulary sizes differ by variant: 115 is the full CBT-18.95M tokenizer, 111 the CBT-200k and Learned-2D variants, and 27 the digit-and-symbol character models. All rows showing `/50` in the Tool Calling column were evaluated on the identical 50-dialogue structured tool suite (6 domains), whereas CBT-18.95M was evaluated on the 26-case 9-schema 1-hop suite of Table 4. Rows were trained with different digit ranges and data and are not a controlled $2\times2$ study. The unseen-sum column pools 14–20 digits, whereas Table 7 reports 14–16 digits separately (hence 2.3% here and 4.6% there for the 1D character model). The Structured Cascade column reports the hardest single-carry chain ($99\dots9+1$) at unseen 20 digits for the 12d models and 50 digits for CBT-18.95M (where CBT-18.95M scores 100.0% on the structured cascade case, contrasting with 96.7% on random 50-digit multi-carries in Table 3B).
 
-| Tokenizer Granularity | Positional Geometry | Vocab ($V$) | Trained Digits | Tool Calling | Seen Sums (In-Dist) | Unseen Sums (14–20d) | Unseen Cascade Carry |
+| Tokenizer Granularity | Positional Geometry | Vocab ($V$) | Trained Digits | Tool Calling | Seen Sums (In-Dist) | Unseen Sums (14–20d) | Structured Cascade |
 | :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
 | Subword BPE (50k) | 1D-RoPE (Flat Line) | 50,304 | 1–12 Digits | N/A (Word-CoT) | 99.1% (1–12d) | **0.0%** (Unseen 18–20d) | **0.0%** (Unseen 20d) |
 | Subword BPE (Naive 4k) | 1D / 2D-RoPE | 4,000 | 1–12 Digits | **100.0% (50/50)** | **0.0%** (1–12d) | **0.0%** (Unseen 14–20d) | **0.0%** (Unseen 20d) |
@@ -248,7 +248,7 @@ Comparing our 2D Matrix Character + Control-Token CBT formulation against 1D Cha
 | Atomic Character | NoPE + Abacus Place | 27 | 1–12 Digits | N/A | **0.0%** (1–12d) | **0.0%** (Unseen 14–20d) | **0.0%** (Unseen 20d) |
 | Atomic Character | Learned 2D Lookup Table | 111 | 1–12 Digits | 98.0% (49/50) | 100.0% (1–12d) | **0.0%** (Capped $p > 12$) | **0.0%** (Unseen 20d) |
 | **CBT-200k (Char + Control)** | **2D-RoPE Matrix ($R=3$)** | **111** | **1–12 Digits** | **100.0% (50/50)** | **100.0% (1–12d)** | **100.0% (Unseen 14–20d)** | **100.0% (Unseen 20d)** |
-| **CBT-18.95M (Char + Control)** | **2D-RoPE Matrix ($R=3$)** | **115** | **1–15 Digits** | **100.0% (26/26)** | **100.0% (1–15d)** | **98.8% (237/240 Rand 16–50d)** | **100.0% (Unseen 50d)** |
+| **CBT-18.95M (Char + Control)** | **2D-RoPE Matrix ($R=3$)** | **115** | **1–15 Digits** | **100.0% (26/26)** | **100.0% (1–15d)** | **98.8% (237/240 Rand 16–50d)** | **100.0% (50d Struct)** |
 
 ---
 
@@ -310,34 +310,37 @@ Because we tested multiple combinations of CBT components that achieved similar
 <THINK>(0+9+1)=10(1+9+0)=10...(20 cols)...(1+9+0)=10(1+0+0)=01</THINK>=1[19 zeros]<EOS>
 ```
 
-Inside `<THINK>...</THINK>`, Model 5's scratchpad was correct across all columns in the 14 test cases. The error is an *off-by-one readout termination* after `</THINK>=`, where the model emits 19 zeros instead of 20 before `<EOS>`. Because the answer-digit coordinates are derived from the scratchpad column count via $p=\max(0, C-1-i)$ (Section 3.3), the model itself must decide where the answer ends. Two plausible (untested) hypotheses for this off-by-one error are: (a) the clamp $p=\max(0, C-1-i)$ assigns $p=0$ to both the true units digit ($i=C-1$) and any extra overflow digit ($i \ge C$), making the last answer digit and the post-answer position share the same macro coordinate $p=0$; and (b) after shift masking moves $p>0$ by $\Delta \in [0, 16]$ while $p=0$ stays fixed, the $p=1 \to p=0$ boundary is under-trained at 12.8 epochs. When the exact same architecture is trained for 64 epochs on a 1,000-sample pool (`CBT-100k Ref`, 4.24M tokens) or scaled to **CBT-18.95M** (247.3M tokens), the readout error disappeared in our tests (14/14 for `CBT-100k Ref`; 20/20 at 16–50 digits for CBT-18.95M), though repeating a 1,000-sample pool 64 times raises the possibility of overfitting to training length distributions rather than purely resolving boundary attention.
+Inside `<THINK>...</THINK>`, Model 5's scratchpad was correct across all columns in the 14 test cases. The error is an *off-by-one readout termination* after `</THINK>=`, where the model emits 19 zeros instead of 20 before `<EOS>`. Because the answer-digit coordinates are derived from the scratchpad column count via $p=\max(0, C-1-i)$ (Section 3.3), the model itself must decide where the answer ends. Two plausible (untested) hypotheses for this off-by-one error are: (a) the clamp $p=\max(0, C-1-i)$ assigns $p=0$ to both the true units digit ($i=C-1$) and any extra overflow digit ($i \ge C$), making the last answer digit and the post-answer position share the same macro coordinate $p=0$; and (b) after shift masking moves $p>0$ by $\Delta \in [0, 16]$ while $p=0$ stays fixed, the $p=1 \to p=0$ boundary is under-trained at 12.8 epochs. When the exact same architecture is trained for 64 epochs on a 1,000-sample pool (`CBT-100k Ref`, 4.24M tokens) or scaled to **CBT-18.95M** (247.3M tokens), the readout error disappeared in our tests (14/14 for `CBT-100k Ref`; 240/240 random 16–50 digit problems with zero off-by-one errors for CBT-18.95M, Table 3B), though repeating a 1,000-sample pool 64 times raises the possibility of overfitting to training length distributions rather than purely resolving boundary attention.
 
 ---
 
-**Table 8B: Strictly Equalized 1–15 Digit, $T=1024$ Controlled Comparison Including Ruoss et al. (2023) Randomized Position Encodings Evaluated on 200 Random Multi-Carry Problems per Length ($3,200$ Problems/Model, $12,800$ Total)**  
-All four equalized models share the exact same 100k character-level backbone ($L=2, d_{\text{model}}=64, n_{\text{head}}=4, R=1, V=27$), the exact same 6,000-equation **1–15 digit** training dataset (matching CBT-18.95M's 1–15d distribution), a **1024-token context window**, and 3,000 training steps at batch size 32. Neither standard 1D-RoPE, 1D-RoPE with random translation offset ($\Delta \sim \mathcal{U}\{0..512\}$), nor 1D-RoPE with Ruoss et al. [16] randomized positions ($0 \le t_1 < \dots < t_T < 1024$) solves any unseen 16–100 digit addition ($0/200$, 95% Wilson CI $[0.0, 1.9]\%$), whereas equalized Decoupled 2D-RoPE ($k=32, p_{\max}=47$) reaches **92.0% (184/200)** at 16d and 25d, **90.0% (180/200)** at 20d, and **85.0% (170/200)** at 30d before dropping to 0.0% beyond $p_{\max}=47$ ($d \ge 50$).
+**Table 8B: Strictly Equalized 1–15 Digit, $T=1024$ Controlled Comparison Including Ruoss et al. (2023) Randomized Position Encodings**  
+All four 100k models share the exact same character-level backbone ($L=2, d_{\text{model}}=64, n_{\text{head}}=4, R=1, V=27$), the exact same 6,000-equation **1–15 digit** training dataset, a **1024-token context window**, and 3,000 training steps at batch size 32 (evaluated on 200 random multi-carry problems per length; $3{,}200$ problems/model, $12{,}800$ total). In this long-context 100k setup, all three 1D variants (standard 1D-RoPE, translation offset $\Delta \sim \mathcal{U}\{0..512\}$, and Ruoss et al. randomized positions) suffered an **in-distribution optimization failure** (scoring $0/600$ on seen 8, 12, and 15 digits, loss $\approx 0.10\text{--}0.23$), failing to learn the task under flat coordinates. Thus Table 8B reflects an equalized training/optimization comparison; the isolated extrapolation failure after successful training is shown in Table 7 (where 18.91M 1D-Char achieves 98.4% in-distribution but drops to 4.6% at 14–16d and 0.0% at 18–20d). Equalized Decoupled 2D-RoPE ($k=32, p_{\max}=47$) achieves 69.5% in-distribution and reaches **81.4% (977/1,200)** across unseen 16–40 digits (reflecting partial undertraining at 3,000 steps relative to the 18.95M model's 100% score) before dropping to 0.0% beyond $p_{\max}=47$ ($d \ge 50$). Right column shows the converged CBT-18.95M reference ($n=30$ per length from Table 3B).
 
-| Evaluation Digit Length ($n=200$/len) | Exposure Status | 1D-RoPE Standard (1–15d, $T=1024$) | 1D-RoPE + Offset ($\Delta \sim \mathcal{U}\{0..512\}$) | 1D-RoPE + Ruoss et al. [16] ($M_{\max}=1024$) | Decoupled 2D-RoPE + Shift (100k, $R=1, k=32$) | CBT-18.95M Ref ($L=6, R=3, k=32$) |
+| Evaluation Digit Length | Exposure Status | 1D-RoPE Standard (100k, $n=200$) | 1D-RoPE + Offset (100k, $n=200$) | 1D-RoPE + Ruoss et al. (100k, $n=200$) | Decoupled 2D-RoPE + Shift (100k, $n=200$) | CBT-18.95M Ref ($n=30$/len, Table 3B) |
 | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
 | Final Training Loss (3,000 Steps) | 1–15 Digits | `0.09822` | `0.09582` | `0.23179` | **`0.00597`** | **`0.00009`** |
-| Seen 8, 12, 15 Digits ($n=600$) | Seen (1–15d) | 0.0% (0/600) [0.0, 0.6]% | 0.0% (0/600) [0.0, 0.6]% | 0.0% (0/600) [0.0, 0.6]% | **69.5% (417/600)** [65.7, 73.1]% | **100.0%** (Seen 1–15d) |
-| *of which 15 Digits ($n=200$)* | Max Seen (15d) | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **86.0% (172/200)** [80.5, 90.1]% | **100.0%** |
-| Unseen 16 Digits ($n=200$) | **Unseen (+1d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **92.0% (184/200)** [87.4, 95.0]% | **100.0% (30/30)** [88.6, 100]% |
-| Unseen 20 Digits ($n=200$) | **Unseen (+5d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **90.0% (180/200)** [85.1, 93.4]% | **100.0% (30/30)** [88.6, 100]% |
-| Unseen 25 Digits ($n=200$) | **Unseen (+10d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **92.0% (184/200)** [87.4, 95.0]% | **100.0% (30/30)** [88.6, 100]% |
-| Unseen 30 Digits ($n=200$) | **Unseen (+15d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **85.0% (170/200)** [79.4, 89.3]% | **100.0% (30/30)** [88.6, 100]% |
-| Unseen 35 Digits ($n=200$) | **Unseen (+20d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **78.0% (156/200)** [71.8, 83.2]% | **100.0% (30/30)** [88.6, 100]% |
-| Unseen 40 Digits ($n=200$) | **Unseen (+25d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **51.5% (103/200)** [44.6, 58.3]% | **100.0% (30/30)** [88.6, 100]% |
-| Unseen 45 Digits ($n=200$) | **Unseen (+30d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **30.0% (60/200)** [24.1, 36.7]% | **93.3% (28/30)** [78.7, 98.2]% |
-| Unseen 50–100 Digits ($n=1,200$) | **Beyond $p_{\max}=47$** | 0.0% (0/1200) [0.0, 0.3]% | 0.0% (0/1200) [0.0, 0.3]% | 0.0% (0/1200) [0.0, 0.3]% | 0.0% (0/1200) [0.0, 0.3]% | 96.7% (50d), 90% (55d), 45% (60d) |
-| **Pooled Unseen 16–40d ($n=1,200$)** | **Within $p_{\max}=47$** | 0.0% (0/1200) [0.0, 0.3]% | 0.0% (0/1200) [0.0, 0.3]% | 0.0% (0/1200) [0.0, 0.3]% | **81.4% (977/1200)** [79.1, 83.5]% | **100.0% (180/180)** [97.9, 100]% |
+| Seen 8, 12, 15 Digits ($n=600$ / $n=90$) | Seen (1–15d) | 0.0% (0/600) [0.0, 0.6]% | 0.0% (0/600) [0.0, 0.6]% | 0.0% (0/600) [0.0, 0.6]% | **69.5% (417/600)** [65.7, 73.1]% | **100.0%** (Seen 1–15d) |
+| *of which 15 Digits ($n=200$ / $n=30$)* | Max Seen (15d) | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **86.0% (172/200)** [80.5, 90.1]% | **100.0%** |
+| Unseen 16 Digits ($n=200$ / $n=30$) | **Unseen (+1d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **92.0% (184/200)** [87.4, 95.0]% | **100.0% (30/30)** [88.6, 100]% |
+| Unseen 20 Digits ($n=200$ / $n=30$) | **Unseen (+5d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **90.0% (180/200)** [85.1, 93.4]% | **100.0% (30/30)** [88.6, 100]% |
+| Unseen 25 Digits ($n=200$ / $n=30$) | **Unseen (+10d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **92.0% (184/200)** [87.4, 95.0]% | **100.0% (30/30)** [88.6, 100]% |
+| Unseen 30 Digits ($n=200$ / $n=30$) | **Unseen (+15d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **85.0% (170/200)** [79.4, 89.3]% | **100.0% (30/30)** [88.6, 100]% |
+| Unseen 35 Digits ($n=200$ / $n=30$) | **Unseen (+20d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **78.0% (156/200)** [71.8, 83.2]% | **100.0% (30/30)** [88.6, 100]% |
+| Unseen 40 Digits ($n=200$ / $n=30$) | **Unseen (+25d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **51.5% (103/200)** [44.6, 58.3]% | **100.0% (30/30)** [88.6, 100]% |
+| Unseen 45 Digits ($n=200$ / $n=30$) | **Unseen (+30d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **30.0% (60/200)** [24.1, 36.7]% | **93.3% (28/30)** [78.7, 98.2]% |
+| Unseen 50 Digits ($n=200$ / $n=30$) | **Boundary ($p \approx 47$)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **96.7% (29/30)** [83.3, 99.4]% |
+| Unseen 55 Digits ($n=200$ / $n=20$) | **Beyond $p_{\max}$ (+8d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | **90.0% (18/20)** [69.9, 97.2]% |
+| Unseen 60 Digits ($n=200$ / $n=20$) | **Beyond $p_{\max}$ (+13d)** | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 0.0% (0/200) [0.0, 1.9]% | 45.0% (9/20) [25.8, 65.8]% |
+| Unseen 70–100 Digits ($n=600$ / $n=45$) | **Beyond $p_{\max}$ & Context** | 0.0% (0/600) [0.0, 0.6]% | 0.0% (0/600) [0.0, 0.6]% | 0.0% (0/600) [0.0, 0.6]% | 0.0% (0/600) [0.0, 0.6]% | 2.2% (1/45; 6.7% 70d, 0% 80–100) |
+| **Pooled Unseen 16–40d ($n=1,200$ / $n=180$)** | **Within $p_{\max}=47$** | 0.0% (0/1200) [0.0, 0.3]% | 0.0% (0/1200) [0.0, 0.3]% | 0.0% (0/1200) [0.0, 0.3]% | **81.4% (977/1200)** [79.1, 83.5]% | **100.0% (180/180)** [97.9, 100]% |
 
 ### 2.3 Strictly Equalized 1–15 Digit, $T=1024$ Comparison with Ruoss et al. Randomized Positions (12,800 Random Problems)
 
-Because the 18.91M 1D baselines in Table 7 were trained on 1–12 digits with a 512-token context window and without position-offset augmentation, we conducted a strictly equalized experiment (`experiments/run_equalized_15d_1024_and_large_sweep.py`, Table 8B) where all models share the exact same 100k character-level architecture ($L=2, d_{\text{model}}=64, n_{\text{head}}=4, R=1, V=27$), the exact same 6,000-sample **1–15 digit** training set, a **1024-token context window**, and 3,000 training steps at batch size 32, and evaluated each model on **200 stratified random multi-carry problems at every length** ($8, 12, 15, 16, 20, 25, 30, 35, 40, 45, 50, 55, 60, 70, 80, 100$ digits; $3,200$ problems per model, $12,800$ problems total):
-1. **Equalized 1D-RoPE and 1D Random Offset Fail Beyond Short Sums**: Even when trained on 1–15 digits with a 1024-token context window, standard 1D-RoPE (`loss=0.0982`) and 1D-RoPE with random translation offsets $\Delta \sim \mathcal{U}\{0, \dots, 512\}$ (`loss=0.0958`) solve 2-digit sums (`15+25=40`) but score **0.0% (0/200; 95% Wilson CI $[0.0, 1.9]\%$)** at every random multi-carry length from 8 to 100 digits.
+Because the 18.91M 1D baselines in Table 7 were trained on 1–12 digits with a 512-token context window and without position-offset augmentation, we conducted a strictly equalized experiment (`experiments/run_equalized_15d_1024_and_large_sweep.py`, Table 8B) where all models share the exact same 100k character-level architecture ($L=2, d_{\text{model}}=64, n_{\text{head}}=4, R=1, V=27$), the exact same 6,000-sample **1–15 digit** training set, a **1024-token context window**, and 3,000 training steps at batch size 32, evaluated on **200 stratified random multi-carry problems at every length** ($8, 12, 15, 16, 20, 25, 30, 35, 40, 45, 50, 55, 60, 70, 80, 100$ digits; $3,200$ problems per model, $12,800$ problems total):
+1. **Equalized 1D Baselines Suffer In-Distribution Optimization Failure**: In this 100k, 1024-context setting, standard 1D-RoPE (`loss=0.0982`), 1D-RoPE with translation offsets $\Delta \sim \mathcal{U}\{0..512\}$ (`loss=0.0958`), and Ruoss et al. randomized positions (`loss=0.2318`) failed to converge in-distribution, scoring **0.0% (0/600; 95% Wilson CI $[0.0, 0.6]\%$)** on seen 8-, 12-, and 15-digit sums. Under flat 1D coordinates in a 1024-token context, 100k parameters and 3,000 steps are insufficient for the attention heads to learn the variable-distance routing required for column scratchpads. This test shows that 1D models fail to optimize the task under equalized conditions, rather than isolating length extrapolation per se; the cleaner evidence for extrapolation failure after successful in-distribution learning is provided by Table 7, where 18.91M 1D-Char achieves 98.4% in-distribution but collapses to 4.6% at 14–16d and 0.0% at 18–20d.
 2. **Why Ruoss et al. [16] Randomized 1D Positions Do Not Substitute for 2D Column Alignment**: Sampling sorted random 1D positions $0 \le t_1 < t_2 < \dots < t_T < 1024$ during training causes training loss to stall at `0.2318` and yields **0.0% (0/200)** across all tested lengths. In a flat 1D sequence without explicit index hints, the model relies on consistent relative token distances to locate corresponding operand digits; randomizing inter-token 1D gaps destroys those relative offsets, whereas shifting the macro coordinate $p \leftarrow p + \Delta$ in Decoupled 2D-RoPE preserves $\Delta p = p_t - p_j = 0$ between every scratchpad column and its matching operand digits.
-3. **Equalized Decoupled 2D-RoPE ($k=32$) Extrapolates Across 1,200 Unseen 16–40d Problems**: Under the exact same 100k $R=1$ architecture and 3,000-step budget, Decoupled 2D-RoPE drops training loss to `0.0060` and achieves **81.4% (977/1,200; 95% Wilson CI $[79.1, 83.5]\%$)** pooled across unseen 16–40 digit random multi-carry problems—including **92.0% (184/200)** at 16d and 25d, **90.0% (180/200)** at 20d, and **85.0% (170/200)** at 30d—before dropping at 45d (**30.0%, 60/200**) and reaching 0.0% at $d \ge 50$ once column indices exceed the $p_{\max} = 15 + 32 = 47$ shift-masking support.
+3. **Equalized Decoupled 2D-RoPE ($k=32$) Extrapolates Across 1,200 Unseen 16–40d Problems**: Under the exact same 100k $R=1$ architecture and 3,000-step budget, Decoupled 2D-RoPE drops training loss to `0.0060`, achieves 69.5% (417/600) on seen 8–15 digits (86.0% at 15 digits), and achieves **81.4% (977/1,200; 95% Wilson CI $[79.1, 83.5]\%$)** pooled across unseen 16–40 digit random multi-carry problems—including **92.0% (184/200)** at 16d and 25d, **90.0% (180/200)** at 20d, and **85.0% (170/200)** at 30d—before dropping at 45d (**30.0%, 60/200**) and reaching 0.0% at $d \ge 50$ once column indices exceed the $p_{\max} = 15 + 32 = 47$ shift-masking support. The lower accuracy of equalized 2D (81.4% vs. 100% in CBT-18.95M) reflects partial undertraining at 3,000 steps on 100k parameters.
 
 ---
 

RESEARCH_PAPER.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6dce8fcc3bd88c30999e185bf6f2cf1c9a05c13e1c18b24ed34e3269ee1d1c07
-size 463749
+oid sha256:63acd5f400fb015c5d06914187440079f3c15eb0fd2b88ee98ce8154e10b813c
+size 465243