Maggio33's picture
Card and SHA256SUMS for update 55000: README f6d5c3e9, SHA256SUMS 2369d538
32a7304 verified
|
Raw History Blame Contribute Delete
7.43 kB
---
license: apache-2.0
library_name: pytorch
tags:
- polish
- checkpoints
- training-history
- training-in-progress
---
# Slayer1B 1.09 — training checkpoints
**Arkadiusz Słota, Kacper Wikieł, Anna Sieczka, Piotr Styła, Aleksander Ogrodzki, Paweł Puzio, Marcin Kuśmierski / Fabryka AI (SlayerLab)**
Native training checkpoints of the run that will produce [Slayer1B 1.09](https://huggingface.co/SlayerLab/Slayer1B-1.09):
continued pretraining from the final weights of [Slayer1B 1.08](https://huggingface.co/SlayerLab/Slayer1B-1.08) with a fresh
optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a
Transformers/safetensors package and not a finished model.
**Status: training in progress (not finished).** It was paused from 12 to 14 November 2026, resumed on 14 November from update 40,000, moved to 8 × NVIDIA B300 on 15 November, continued on NVIDIA RTX 4090 from update 47,200 and on 8 × NVIDIA B200 from update 47,700 on 16 November, and resumed on 8 × NVIDIA B200 from update 48,600 on 17 November (see Hardware).
This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 55,000 and one full-state
checkpoint at update 35,000 (update 40,000 added on 13 November 2026, update 45,000 on 15 November 2026, updates 50,000 and 55,000 on 17 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
the LFS object id (sha256) and the size match.
| | |
|---|---|
| Start | final weights of Slayer1B 1.08 (revision `5f854a0c`), fresh optimizer state, update 0 |
| Updates so far | 1 → 55,000 in this repository (55,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
| Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
| Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
| Hardware | updates 1–40,000: 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once. Updates 40,001–42,800: 2 × NVIDIA RTX 4090 with the same 256 × 4,096 tokens per update (gradient accumulation); a 100-update check (updates 35,001–35,100) matched the earlier hardware's training loss within 0.0002 on all 10 logged points (mean difference 0.00005). Updates 42,801–47,200: 8 × NVIDIA B300, spot instance, resumed from the full checkpoint at update 42,800; the RTX 4090 had already computed updates 42,801–42,880, and on the 8 logged points in that range the B300 training loss matched it within 0.0003 (mean difference −0.0002). The public Track curve keeps the RTX 4090 points up to update 42,880 and shows B300 points after it. Updates 47,201–47,700: 2 × NVIDIA RTX 4090 (same 256 × 4,096 tokens per update), resumed from the full checkpoint at update 47,200. Updates from 47,701: 8 × NVIDIA B200, spot instances, resumed from the full checkpoint at update 47,700; 4 × NVIDIA RTX 4090 had already computed updates 47,701–47,730, and on the 3 logged points in that range the absolute training-loss difference was below 0.00014 (a local sanity check, not a trajectory guarantee). The public Track curve keeps the RTX 4090 points up to update 47,730 and shows B200 points after it |
| Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (sha256 `db36c150…`) |
| Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
| Tokenizer | v6, vocabulary 32,768 (sha256 `0640d3bd…`), as in Slayer1B 1.08 |
## Files
| file | contents | size |
|---|---|---:|
| `wagi_<update>.pt` | model weights (float32), `config`, `step` | 4.3 GB |
| `ckpt_35000.pt` | model weights, optimizer state (Muon and AdamW), data-queue position, `config`, `step` | 8.9 GB |
`wagi` is Polish for “weights”. Load with `torch.load(path, map_location="cpu", weights_only=True)`; the model definition is in
`train_gpt_ref_r23.py` in the [1.08 repository](https://huggingface.co/SlayerLab/Slayer1B-1.08).
## Tokens per checkpoint
Two different counts, kept apart on purpose. **Positions in this run** = update × 1,048,576 (training positions processed by
the 1.09 run itself, counting each update once). **Lineage lower bound** = positions inherited from Slayer1B 1.00–1.08
(18,863,257,682, source- and file-disjoint) + positions in this run − 114,627,128 positions identified as copies in the
first data segment. The lineage figure is a lower bound on source- and file-disjoint positions seen by the model line, not a
count of globally deduplicated unique tokens.
| update | positions in this run (billions) | lineage lower bound (billions) |
|---:|---:|---:|
| 5000 | 5.24 | 23.99 |
| 10000 | 10.49 | 29.23 |
| 15000 | 15.73 | 34.48 |
| 20000 | 20.97 | 39.72 |
| 25000 | 26.21 | 44.96 |
| 30000 | 31.46 | 50.21 |
| 35000 | 36.70 | 55.45 |
| 40000 | 41.94 | 60.69 |
| 45000 | 47.19 | 65.93 |
| 50000 | 52.43 | 71.18 |
| 55000 | 57.67 | 76.42 |
## SHA-256
| update | file | sha256 |
|---:|---|---|
| 5000 | `wagi_5000.pt` | `bc289e2afe7e40d37c60a23c75bd10480ac903e29a0f5d8d184be6bcdf640e91` |
| 10000 | `wagi_10000.pt` | `fcfd2d3044d504706519fb6e1bd8d6dea959b162dac62e0f5967bbd6807df643` |
| 15000 | `wagi_15000.pt` | `1cc151d20cd1abf3b02a9dfbedb5c5b458861105307498acccfc15283d951ae4` |
| 20000 | `wagi_20000.pt` | `41a457dcdf1d60011fe576fe938d2d61e80ce71028c8f1b484face9ee3dc73d5` |
| 25000 | `wagi_25000.pt` | `3bd8e411635eb1a18fc6e96fcfee38b68850dbd20171f9cc73897187277ceda3` |
| 30000 | `wagi_30000.pt` | `74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d` |
| 35000 | `wagi_35000.pt` | `75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43` |
| 40000 | `wagi_40000.pt` | `f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881` |
| 45000 | `wagi_45000.pt` | `e6dd131a1e7b6a25d0421260bb8a24a52a95f48b01cfecb183987c65bb97d827` |
| 50000 | `wagi_50000.pt` | `4a913162116e78e497399b20cccaf896aa79663f7bbca31a6a7964b72001b20f` |
| 55000 | `wagi_55000.pt` | `d398310c12cc5b71e5e09fcf63c99153dc516e496002d4c6a713512fd6fa6da6` |
| 35000 | `ckpt_35000.pt` | `f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e` |
The same list is in `SHA256SUMS`.
## Limitations
These checkpoints are intermediate states of a constant-learning-rate run. Training loss and evaluation results at a given
update are on Track; they are not a quality verdict on a finished model. Changes of the data segment during the run (for
example at update 10,000) change the training-loss distribution, so a drop across such a boundary does not by itself show a
quality gain.
Checkpoint recovery can recompute updates from the last saved state. Public history retains first observations at each step;
replay does not add distinct updates or token counts. Numerical trajectories need not be bit-identical across hardware.