README.md

---
license: mit
language:
- en
tags:
- decoderstack
- nanochat
- pretraining
---

# DecoderStack d24

A 1.38B-parameter d24 model pre-trained from scratch with
[**DecoderStack**](https://github.com/chrisjmccormick/stacks) — a single-file training
stack with a handwritten forward/backward and a written-out optimizer: no autograd, no
`torch.optim`, no `nn.Module`. bf16, varlen FlashAttention, 8xH100, 1h51m.

The architecture is the nanochat d24 architecture, so these weights load into
[nanochat](https://github.com/karpathy/nanochat) once converted — see
[Loading into nanochat](#loading-into-nanochat).

## Results

Full horizon: 5,568 steps x 2^20 tokens = 5.84B tokens (data:param ratio 8).

```
step 05567/05568 (99.98%) | loss: 2.388180 | lrm: 0.05 | dt: 1198.08ms | tok/sec: 875,210 | bf16_mfu: 53.76 | total time: 110.74m
step:5568/5568 val_bpb:0.719042
CORE metric: 0.2517 | total CORE eval time: 80.46s
peak memory allocated: 71569 MiB reserved: 72736 MiB
total training time: 110.74m
minimum validation bpb: 0.719042
```

| | value |
|---|---|
| parameters | 1,384,122,122 |
| min val bpb | 0.719042 |
| CORE | 0.2517 |
| train time | 110.74 min on 8xH100 80GB |
| throughput | 1,212 ms/step, 864,856 tok/sec (steps 20–99) |
| precision | bf16 (no FP8) |

val bpb curve: 0:3.175835 / 250:0.997472 / 500:0.904081 / 1000:0.846058 / 2000:0.809199 /
3000:0.773139 / 4000:0.747155 / 5000:0.726908 / 5568:0.719042

**Against upstream nanochat**, run on the same box on the same rebuilt validation shard,
DecoderStack tracks 0.2–1.4% behind through step 1000 (where the upstream baseline was
stopped) while running ~6% faster per step (1,212 vs 1,291 ms/step). Full head-to-head in
[`logs/baseline_report.md`](logs/baseline_report.md).

`ChrisMcCormick/nanochat-varlen-d24-2026-03-22` is the closest sibling release
(0.724772 val bpb, 0.2614 CORE, 97.4 min) but it is **not** a controlled comparison: it
trained with FP8 and a different 32k tokenizer.

## ⚠️ Tokenizer

These weights are tied to the 32k vocabulary in
[`ChrisMcCormick/climbmix_32k_8_170`](https://huggingface.co/datasets/ChrisMcCormick/climbmix_32k_8_170),
mirrored here under [`tokenizer/`](tokenizer). It is a **different vocabulary** from the
one in `ChrisMcCormick/nanochat-varlen-d24-2026-03-22` — 31,474 of 32,759 merge ids
differ. Both are 32,768 tokens and both load without complaint, so nothing will warn you.
Pairing these weights with the wrong tokenizer produces garbage, not slightly-worse text.

## Contents

```
checkpoints/                        native DecoderStack captures (the source of record)
  model_step001950.pt   2.8 GB      {step, code, weights} -- bf16 live weights + fp32 scalars
  optim_step001950.pt  11.1 GB      {step, t_step, state} -- mantissas + Muon/AdamW moments
  model_step005568.pt   2.8 GB
  optim_step005568.pt  11.1 GB
base_checkpoints/d24_decoderstack/  nanochat-format target dir
  meta_001950.json                  model_config + run metadata
  meta_005568.json                  (the converter writes model_NNNNNN.pt in here)
code/run_full_d24_w8.py             the exact training script this run executed
logs/
  d24_decoderstack_20260801_023219.log   full training log
  run_full_d24_w8.sh                     launcher, with the config rationale
  baseline_report.md                     throughput + val bpb vs upstream nanochat
tokenizer/                          the 32k vocab these weights were trained on
convert_ckpt_to_nanochat.py         DecoderStack capture -> nanochat state_dict
```

Two capture points: **5568** is the end of the run, **1950** is the last uncooled state —
the LR/momentum hold ends after update 1949 (`N - round(0.65*N)`), so 1950 is the one to
resume from if you want to train the horizon longer.

DecoderStack writes `code` (the full training script, `open(sys.argv[0]).read()`) into
every model capture, so each `.pt` carries its own exact source. `code/run_full_d24_w8.py`
is that string extracted from `model_step005568.pt`.

## Loading into nanochat

`convert_ckpt_to_nanochat.py` unbanks the per-layer matrix banks, renames tensors to
nanochat's module paths, maps value-embedding bank slots back to their layers, and
rebuilds the fp32 masters from the bf16 live weights plus the uint16 mantissa held in the
optimizer file.

```bash
pip install huggingface_hub torch
python - <<'PY'
from huggingface_hub import hf_hub_download
R = "ChrisMcCormick/decoderstack-d24"
for f in ["checkpoints/model_step005568.pt",          # 2.8 GB
          "checkpoints/optim_step005568.pt",          # 11.1 GB -- optional, see below
          "base_checkpoints/d24_decoderstack/meta_005568.json",
          "convert_ckpt_to_nanochat.py",
          "tokenizer/tokenizer.pkl", "tokenizer/token_bytes.pt"]:
    hf_hub_download(R, f, local_dir="decoderstack-d24")
PY

cd decoderstack-d24
python convert_ckpt_to_nanochat.py \
    --model checkpoints/model_step005568.pt \
    --optim checkpoints/optim_step005568.pt \
    --meta  base_checkpoints/d24_decoderstack/meta_005568.json \
    --out   ~/.cache/nanochat/base_checkpoints/d24_decoderstack
mkdir -p ~/.cache/nanochat/tokenizer && cp tokenizer/* ~/.cache/nanochat/tokenizer/
```

(`~/.cache/nanochat` is nanochat's default base dir; override with `NANOCHAT_BASE_DIR`.)

`--optim` is optional. It only supplies the mantissas — the lower 16 bits of the fp32
masters. Without it the fp32 parameters carry bf16 precision, which costs about 0.0005
nats/token (2.4833 vs 2.4828 on a held-out paragraph); skip the 11 GB download unless you
want the exact master.

Then, on the [`fa-varlen`](https://github.com/chrisjmccormick/nanochat/tree/fa-varlen)
branch (varlen, matching how these weights were trained):

```python
import os, torch
from nanochat.checkpoint_manager import build_model
ckpt_dir = os.path.expanduser("~/.cache/nanochat/base_checkpoints/d24_decoderstack")
model, tokenizer, meta = build_model(ckpt_dir, 5568, torch.device("cuda"), "eval")
```

The conversion is verified: 175 tensors, shapes and dtypes identical to a freshly built
nanochat `GPT` at this config, loads with `strict=True`, and scores 2.4828 nats/token
(0.725 bpb) on held-out English — in line with the run's own 0.719 val bpb — with
coherent greedy continuations.

**DecoderStack itself cannot load these back yet.** The capture path is write-only by
design; a load/resume path is future work.

### Optimizer state

Not converted, and nanochat could not resume from it if it were. Every buffer does have a
counterpart, and the precisions line up on everything except the embedding tables:

| DecoderStack | nanochat | precision |
|---|---|---|
| `.frst_mntm` | Muon `momentum_buffer` | fp32 both |
| `.scnd_mntm` | Muon `second_momentum_buffer` | fp32 both |
| `.exp_avg` / `.exp_avg_sq` | AdamW `exp_avg` / `exp_avg_sq` | fp32 both, **except** `wte` and `value_embeds`, which nanochat allocates `zeros_like(p)` on a bf16 param — so bf16 there, fp32 here |
| `.mantissa` (uint16) | — | nanochat's fp32 param *is* the master; consumed by the converter to rebuild it |

Two things block a faithful resume regardless: `W_O`'s NorMuon second moment reduces along
a different axis in each codebase (nanochat infers it from the shape and lands on `-1` for
a square `c_proj`; DecoderStack sets `residual_dim = -2` deliberately, because W_O's heads
are stored transposed relative to QKV), and nanochat's resume needs a
`dataloader_state_dict` that DecoderStack's pre-tokenized binary loader has no equivalent
of. The remaining mapping — param ordering and per-rank sharding — is written out at the
bottom of `convert_ckpt_to_nanochat.py`.

## Provenance

Trained by [`chrisjmccormick/stacks`](https://github.com/chrisjmccormick/stacks) — the
converter is committed at
[`e664830`](https://github.com/chrisjmccormick/stacks/commit/e6648306442e0479433b287ede02d271997365c0)
(`utils/convert_ckpt_to_nanochat.py`). The training script that produced these weights is
`code/run_full_d24_w8.py` in this repo, which is the run copy of the single-file d24
trainer with three launcher overrides (`micro_batch_tokens` 32768→65536,
`max_num_docs` 96→192, `NUM_TRAIN_SHARDS` 20→80); the rationale for each is in
`logs/run_full_d24_w8.sh`.

Training data: [`ChrisMcCormick/climbmix_32k_8_170`](https://huggingface.co/datasets/ChrisMcCormick/climbmix_32k_8_170)
(ClimbMix, pre-tokenized to 32k binary shards). Validation is a rebuild of shard 06542 to
match nanochat's pinned val split.