Slayer1B 1.09 — training checkpoints
Arkadiusz Słota, Kacper Wikieł, Anna Sieczka, Piotr Styła, Aleksander Ogrodzki, Paweł Puzio, Marcin Kuśmierski / Fabryka AI (SlayerLab)
Native training checkpoints of the run that will produce Slayer1B 1.09: continued pretraining from the final weights of Slayer1B 1.08 with a fresh optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a Transformers/safetensors package and not a finished model.
Status: training in progress (not finished). It was paused from 12 to 14 November 2026, resumed on 14 November from update 40,000, moved to 8 × NVIDIA B300 on 15 November, continued on NVIDIA RTX 4090 from update 47,200 and on 8 × NVIDIA B200 from update 47,700 on 16 November, and resumed on 8 × NVIDIA B200 from update 48,600 on 17 November (see Hardware). This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 55,000 and one full-state checkpoint at update 35,000 (update 40,000 added on 13 November 2026, update 45,000 on 15 November 2026, updates 50,000 and 55,000 on 17 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on Track. Every file listed below was compared after upload with the checksum written when it was copied from the training machine: the LFS object id (sha256) and the size match.
| Start | final weights of Slayer1B 1.08 (revision 5f854a0c), fresh optimizer state, update 0 |
| Updates so far | 1 → 55,000 in this repository (55,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
| Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then constant; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
| Data | see the data card; new data 20% Polish / 80% English by target allocation |
| Hardware | updates 1–40,000: 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once. Updates 40,001–42,800: 2 × NVIDIA RTX 4090 with the same 256 × 4,096 tokens per update (gradient accumulation); a 100-update check (updates 35,001–35,100) matched the earlier hardware's training loss within 0.0002 on all 10 logged points (mean difference 0.00005). Updates 42,801–47,200: 8 × NVIDIA B300, spot instance, resumed from the full checkpoint at update 42,800; the RTX 4090 had already computed updates 42,801–42,880, and on the 8 logged points in that range the B300 training loss matched it within 0.0003 (mean difference −0.0002). The public Track curve keeps the RTX 4090 points up to update 42,880 and shows B300 points after it. Updates 47,201–47,700: 2 × NVIDIA RTX 4090 (same 256 × 4,096 tokens per update), resumed from the full checkpoint at update 47,200. Updates from 47,701: 8 × NVIDIA B200, spot instances, resumed from the full checkpoint at update 47,700; 4 × NVIDIA RTX 4090 had already computed updates 47,701–47,730, and on the 3 logged points in that range the absolute training-loss difference was below 0.00014 (a local sanity check, not a trajectory guarantee). The public Track curve keeps the RTX 4090 points up to update 47,730 and shows B200 points after it |
| Trainer | train_gpt_ref r24.2 (sha256 46f83b51…); model definition train_gpt_ref_r23.py (sha256 db36c150…) |
| Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
| Tokenizer | v6, vocabulary 32,768 (sha256 0640d3bd…), as in Slayer1B 1.08 |
Files
| file | contents | size |
|---|---|---|
wagi_<update>.pt |
model weights (float32), config, step |
4.3 GB |
ckpt_35000.pt |
model weights, optimizer state (Muon and AdamW), data-queue position, config, step |
8.9 GB |
wagi is Polish for “weights”. Load with torch.load(path, map_location="cpu", weights_only=True); the model definition is in
train_gpt_ref_r23.py in the 1.08 repository.
Tokens per checkpoint
Two different counts, kept apart on purpose. Positions in this run = update × 1,048,576 (training positions processed by the 1.09 run itself, counting each update once). Lineage lower bound = positions inherited from Slayer1B 1.00–1.08 (18,863,257,682, source- and file-disjoint) + positions in this run − 114,627,128 positions identified as copies in the first data segment. The lineage figure is a lower bound on source- and file-disjoint positions seen by the model line, not a count of globally deduplicated unique tokens.
| update | positions in this run (billions) | lineage lower bound (billions) |
|---|---|---|
| 5000 | 5.24 | 23.99 |
| 10000 | 10.49 | 29.23 |
| 15000 | 15.73 | 34.48 |
| 20000 | 20.97 | 39.72 |
| 25000 | 26.21 | 44.96 |
| 30000 | 31.46 | 50.21 |
| 35000 | 36.70 | 55.45 |
| 40000 | 41.94 | 60.69 |
| 45000 | 47.19 | 65.93 |
| 50000 | 52.43 | 71.18 |
| 55000 | 57.67 | 76.42 |
SHA-256
| update | file | sha256 |
|---|---|---|
| 5000 | wagi_5000.pt |
bc289e2afe7e40d37c60a23c75bd10480ac903e29a0f5d8d184be6bcdf640e91 |
| 10000 | wagi_10000.pt |
fcfd2d3044d504706519fb6e1bd8d6dea959b162dac62e0f5967bbd6807df643 |
| 15000 | wagi_15000.pt |
1cc151d20cd1abf3b02a9dfbedb5c5b458861105307498acccfc15283d951ae4 |
| 20000 | wagi_20000.pt |
41a457dcdf1d60011fe576fe938d2d61e80ce71028c8f1b484face9ee3dc73d5 |
| 25000 | wagi_25000.pt |
3bd8e411635eb1a18fc6e96fcfee38b68850dbd20171f9cc73897187277ceda3 |
| 30000 | wagi_30000.pt |
74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d |
| 35000 | wagi_35000.pt |
75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43 |
| 40000 | wagi_40000.pt |
f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881 |
| 45000 | wagi_45000.pt |
e6dd131a1e7b6a25d0421260bb8a24a52a95f48b01cfecb183987c65bb97d827 |
| 50000 | wagi_50000.pt |
4a913162116e78e497399b20cccaf896aa79663f7bbca31a6a7964b72001b20f |
| 55000 | wagi_55000.pt |
d398310c12cc5b71e5e09fcf63c99153dc516e496002d4c6a713512fd6fa6da6 |
| 35000 | ckpt_35000.pt |
f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e |
The same list is in SHA256SUMS.
Limitations
These checkpoints are intermediate states of a constant-learning-rate run. Training loss and evaluation results at a given update are on Track; they are not a quality verdict on a finished model. Changes of the data segment during the run (for example at update 10,000) change the training-loss distribution, so a drop across such a boundary does not by itself show a quality gain.
Checkpoint recovery can recompute updates from the last saved state. Public history retains first observations at each step; replay does not add distinct updates or token counts. Numerical trajectories need not be bit-identical across hardware.