| ---
|
| license: mit
|
| tags:
|
| - arithmetic
|
| - length-generalization
|
| - grokking
|
| - interpretability
|
| ---
|
|
|
| # carrybit checkpoints
|
|
|
| Trained weights from [carrybit](https://github.com/Supergoatscriptguy/carrybit),
|
| a small research project on tiny transformers learning exact integer
|
| arithmetic. The code, configs, figures and the full write-up live in that
|
| repo. This repo holds the final checkpoint of every run in the write-up so the
|
| analysis experiments can be run without retraining.
|
|
|
| Every checkpoint is a plain PyTorch `state_dict` for `carrybit.model.Transformer`
|
| (or `TwoHotMLP` for `modular_mlp`). Each folder has the exact `config.json` it
|
| was trained with and the `metrics.csv` logged during training. Folder names
|
| match `runs/` in the GitHub repo, so every experiment script there works on
|
| these files as they are.
|
|
|
| ```python
|
| import json, torch
|
| from carrybit.config import load_config
|
| from carrybit.model import Transformer
|
|
|
| run = "addition_big/position_coupling_s1"
|
| raw = json.load(open(f"{run}/config.json"))
|
| cfg = load_config("configs/addition_big.yaml",
|
| [f"{s}.{k}={json.dumps(v)}" for s in ("task", "model")
|
| for k, v in raw[s].items() if k != "kind"])
|
| model = Transformer(16, cfg.model)
|
| model.load_state_dict(torch.load(f"{run}/step_60000.pt"))
|
| ```
|
|
|
| ## Contents
|
|
|
| | folder | what | runs |
|
| |---|---|---|
|
| | `modular_add`, `modular_add_wd0.1`, `modular_add_wd0` | one-layer transformer, a + b mod 113, weight decay 1 / 0.1 / 0 | 1 each |
|
| | `modular_mlp` | ReLU MLP on two-hot inputs, p = 97 (Swaroop 2026 setup) | 1 |
|
| | `addition` | the length ladder: 3.4M params, trained on 1 to 20 digits, nine formats | 3 seeds each, 6 for position coupling |
|
| | `addition_big` | 11M params, abacus and position coupling, trained on 1 to 30 digits | 2 abacus seeds, 4 coupling seeds |
|
| | `addition_sharp`, `addition_big_sharp` | position coupling trained with attention logits scaled by 2 (`model.attn_scale`) | 3 and 2 seeds |
|
| | `blankspace_big` | 11M params, fixed-width aligned blankspace, trained on 1 to 20 digits | 2 seeds |
|
| | `addition_constant_lr`, `addition_no_wd` | position coupling ablations | 3 and 6 seeds |
|
| | `addition_carry_heavy`, `addition_fixed_length` | position coupling with carry-heavy or single-length training data | 3 seeds each |
|
| | `subtraction` | position coupling and fixed blankspace on a - b | 3 seeds each |
|
|
|
| The one to try first is `addition_big/position_coupling_s1`. Trained on up to
|
| 30 digits, it scores 0% exact match at 100 digits as is. Set
|
| `block.attn.scale = 2.0` on every block before decoding and it scores 100% at
|
| 200. The GitHub README explains why.
|
|
|
