license: mit
tags:
- arithmetic
- length-generalization
- grokking
- interpretability
carrybit checkpoints
Trained weights from carrybit, a small research project on tiny transformers learning exact integer arithmetic. The code, configs, figures and the full write-up live in that repo. This repo holds the final checkpoint of every run in the write-up so the analysis experiments can be run without retraining.
Every checkpoint is a plain PyTorch state_dict for carrybit.model.Transformer
(or TwoHotMLP for modular_mlp). Each folder has the exact config.json it
was trained with and the metrics.csv logged during training. Folder names
match runs/ in the GitHub repo, so every experiment script there works on
these files as they are.
import json, torch
from carrybit.config import load_config
from carrybit.model import Transformer
run = "addition_big/position_coupling_s1"
raw = json.load(open(f"{run}/config.json"))
cfg = load_config("configs/addition_big.yaml",
[f"{s}.{k}={json.dumps(v)}" for s in ("task", "model")
for k, v in raw[s].items() if k != "kind"])
model = Transformer(16, cfg.model)
model.load_state_dict(torch.load(f"{run}/step_60000.pt"))
Contents
| folder | what | runs |
|---|---|---|
modular_add, modular_add_wd0.1, modular_add_wd0 |
one-layer transformer, a + b mod 113, weight decay 1 / 0.1 / 0 | 1 each |
modular_mlp |
ReLU MLP on two-hot inputs, p = 97 (Swaroop 2026 setup) | 1 |
addition |
the length ladder: 3.4M params, trained on 1 to 20 digits, nine formats | 3 seeds each, 6 for position coupling |
addition_big |
11M params, abacus and position coupling, trained on 1 to 30 digits | 2 abacus seeds, 4 coupling seeds |
addition_sharp, addition_big_sharp |
position coupling trained with attention logits scaled by 2 (model.attn_scale) |
3 and 2 seeds |
blankspace_big |
11M params, fixed-width aligned blankspace, trained on 1 to 20 digits | 2 seeds |
addition_constant_lr, addition_no_wd |
position coupling ablations | 3 and 6 seeds |
addition_carry_heavy, addition_fixed_length |
position coupling with carry-heavy or single-length training data | 3 seeds each |
subtraction |
position coupling and fixed blankspace on a - b | 3 seeds each |
The one to try first is addition_big/position_coupling_s1. Trained on up to
30 digits, it scores 0% exact match at 100 digits as is. Set
block.attn.scale = 2.0 on every block before decoding and it scores 100% at
200. The GitHub README explains why.