---
license: mit
tags:
  - arithmetic
  - length-generalization
  - grokking
  - interpretability
---

# carrybit checkpoints

Trained weights from [carrybit](https://github.com/Supergoatscriptguy/carrybit),
a small research project on tiny transformers learning exact integer
arithmetic. The code, configs, figures and the full write-up live in that
repo. This repo holds the final checkpoint of every run in the write-up so the
analysis experiments can be run without retraining.

Every checkpoint is a plain PyTorch `state_dict` for `carrybit.model.Transformer`
(or `TwoHotMLP` for `modular_mlp`). Each folder has the exact `config.json` it
was trained with and the `metrics.csv` logged during training. Folder names
match `runs/` in the GitHub repo, so every experiment script there works on
these files as they are.

```python
import json, torch
from carrybit.config import load_config
from carrybit.model import Transformer

run = "addition_big/position_coupling_s1"
raw = json.load(open(f"{run}/config.json"))
cfg = load_config("configs/addition_big.yaml",
                  [f"{s}.{k}={json.dumps(v)}" for s in ("task", "model")
                   for k, v in raw[s].items() if k != "kind"])
model = Transformer(16, cfg.model)
model.load_state_dict(torch.load(f"{run}/step_60000.pt"))
```

## Contents

| folder | what | runs |
|---|---|---|
| `modular_add`, `modular_add_wd0.1`, `modular_add_wd0` | one-layer transformer, a + b mod 113, weight decay 1 / 0.1 / 0 | 1 each |
| `modular_mlp` | ReLU MLP on two-hot inputs, p = 97 (Swaroop 2026 setup) | 1 |
| `addition` | the length ladder: 3.4M params, trained on 1 to 20 digits, nine formats | 3 seeds each, 6 for position coupling |
| `addition_big` | 11M params, abacus and position coupling, trained on 1 to 30 digits | 2 abacus seeds, 4 coupling seeds |
| `addition_sharp`, `addition_big_sharp` | position coupling trained with attention logits scaled by 2 (`model.attn_scale`) | 3 and 2 seeds |
| `blankspace_big` | 11M params, fixed-width aligned blankspace, trained on 1 to 20 digits | 2 seeds |
| `addition_constant_lr`, `addition_no_wd` | position coupling ablations | 3 and 6 seeds |
| `addition_carry_heavy`, `addition_fixed_length` | position coupling with carry-heavy or single-length training data | 3 seeds each |
| `subtraction` | position coupling and fixed blankspace on a - b | 3 seeds each |

The one to try first is `addition_big/position_coupling_s1`. Trained on up to
30 digits, it scores 0% exact match at 100 digits as is. Set
`block.attn.scale = 2.0` on every block before decoding and it scores 100% at
200. The GitHub README explains why.