carrybit checkpoints

Trained weights from carrybit, a small research project on tiny transformers learning exact integer arithmetic. The code, configs, figures and the full write-up live in that repo. This repo holds the final checkpoint of every run in the write-up so the analysis experiments can be run without retraining.

Every checkpoint is a plain PyTorch state_dict for carrybit.model.Transformer (or TwoHotMLP for modular_mlp). Each folder has the exact config.json it was trained with and the metrics.csv logged during training. Folder names match runs/ in the GitHub repo, so every experiment script there works on these files as they are.

import json, torch
from carrybit.config import load_config
from carrybit.model import Transformer

run = "addition_big/position_coupling_s1"
raw = json.load(open(f"{run}/config.json"))
cfg = load_config("configs/addition_big.yaml",
                  [f"{s}.{k}={json.dumps(v)}" for s in ("task", "model")
                   for k, v in raw[s].items() if k != "kind"])
model = Transformer(16, cfg.model)
model.load_state_dict(torch.load(f"{run}/step_60000.pt"))

Contents

folder	what	runs
modular_add, modular_add_wd0.1, modular_add_wd0	one-layer transformer, a + b mod 113, weight decay 1 / 0.1 / 0	1 each
modular_mlp	ReLU MLP on two-hot inputs, p = 97 (Swaroop 2026 setup)	1
addition	the length ladder: 3.4M params, trained on 1 to 20 digits, nine formats	3 seeds each, 6 for position coupling
addition_big	11M params, abacus and position coupling, trained on 1 to 30 digits	2 abacus seeds, 4 coupling seeds
addition_sharp, addition_big_sharp	position coupling trained with attention logits scaled by 2 (model.attn_scale)	3 and 2 seeds
blankspace_big	11M params, fixed-width aligned blankspace, trained on 1 to 20 digits	2 seeds
addition_constant_lr, addition_no_wd	position coupling ablations	3 and 6 seeds
addition_carry_heavy, addition_fixed_length	position coupling with carry-heavy or single-length training data	3 seeds each
subtraction	position coupling and fixed blankspace on a - b	3 seeds each

The one to try first is addition_big/position_coupling_s1. Trained on up to 30 digits, it scores 0% exact match at 100 digits as is. Set block.attn.scale = 2.0 on every block before decoding and it scores 100% at 200. The GitHub README explains why.