Add model card (sha256 c40087bd)
Browse files
README.md
ADDED
|
@@ -0,0 +1,85 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
tags:
|
| 5 |
+
- polish
|
| 6 |
+
- checkpoints
|
| 7 |
+
- training-history
|
| 8 |
+
- training-in-progress
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
# Slayer1B 1.09 — training checkpoints
|
| 12 |
+
|
| 13 |
+
**Arkadiusz Słota, Anna Sieczka, Piotr Styła / Fabryka AI (SlayerLab)**
|
| 14 |
+
|
| 15 |
+
Native training checkpoints of the run that will produce [Slayer1B 1.09](https://huggingface.co/SlayerLab/Slayer1B-1.09):
|
| 16 |
+
continued pretraining from the final weights of [Slayer1B 1.08](https://huggingface.co/SlayerLab/Slayer1B-1.08) with a fresh
|
| 17 |
+
optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a
|
| 18 |
+
Transformers/safetensors package and not a finished model.
|
| 19 |
+
|
| 20 |
+
**Status: training in progress.** This repository holds every weights-only checkpoint saved at a 5,000-update boundary up
|
| 21 |
+
to update 35,000 and one full-state checkpoint at update 35,000. Later checkpoints will be added as the run continues. The
|
| 22 |
+
training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a). Every file
|
| 23 |
+
listed below was compared after upload with the checksum written when it was copied from the training machine: the LFS
|
| 24 |
+
object id (sha256) and the size match.
|
| 25 |
+
|
| 26 |
+
| | |
|
| 27 |
+
|---|---|
|
| 28 |
+
| Start | final weights of Slayer1B 1.08 (revision `5f854a0c`), fresh optimizer state, update 0 |
|
| 29 |
+
| Updates so far | 1 → 35,000 in this repository (35,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
|
| 30 |
+
| Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
|
| 31 |
+
| Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
|
| 32 |
+
| Hardware | 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once |
|
| 33 |
+
| Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (sha256 `db36c150…`) |
|
| 34 |
+
| Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
|
| 35 |
+
| Tokenizer | v6, vocabulary 32,768 (sha256 `0640d3bd…`), as in Slayer1B 1.08 |
|
| 36 |
+
|
| 37 |
+
## Files
|
| 38 |
+
|
| 39 |
+
| file | contents | size |
|
| 40 |
+
|---|---|---:|
|
| 41 |
+
| `wagi_<update>.pt` | model weights (float32), `config`, `step` | 4.3 GB |
|
| 42 |
+
| `ckpt_35000.pt` | model weights, optimizer state (Muon and AdamW), data-queue position, `config`, `step` | 8.9 GB |
|
| 43 |
+
|
| 44 |
+
`wagi` is Polish for “weights”. Load with `torch.load(path, map_location="cpu")`; the model definition is in
|
| 45 |
+
`train_gpt_ref_r23.py` in the [1.08 repository](https://huggingface.co/SlayerLab/Slayer1B-1.08).
|
| 46 |
+
|
| 47 |
+
## Tokens per checkpoint
|
| 48 |
+
|
| 49 |
+
Two different counts, kept apart on purpose. **Positions in this run** = update × 1,048,576 (training positions processed by
|
| 50 |
+
the 1.09 run itself, counting each update once). **Lineage lower bound** = positions inherited from Slayer1B 1.00–1.08
|
| 51 |
+
(18,863,257,682, source- and file-disjoint) + positions in this run − 114,627,128 positions identified as copies in the
|
| 52 |
+
first data segment. The lineage figure is a lower bound on source- and file-disjoint positions seen by the model line, not a
|
| 53 |
+
count of globally deduplicated unique tokens.
|
| 54 |
+
|
| 55 |
+
| update | positions in this run (billions) | lineage lower bound (billions) |
|
| 56 |
+
|---:|---:|---:|
|
| 57 |
+
| 5000 | 5.24 | 23.99 |
|
| 58 |
+
| 10000 | 10.49 | 29.23 |
|
| 59 |
+
| 15000 | 15.73 | 34.48 |
|
| 60 |
+
| 20000 | 20.97 | 39.72 |
|
| 61 |
+
| 25000 | 26.21 | 44.96 |
|
| 62 |
+
| 30000 | 31.46 | 50.21 |
|
| 63 |
+
| 35000 | 36.70 | 55.45 |
|
| 64 |
+
|
| 65 |
+
## SHA-256
|
| 66 |
+
|
| 67 |
+
| update | file | sha256 |
|
| 68 |
+
|---:|---|---|
|
| 69 |
+
| 5000 | `wagi_5000.pt` | `bc289e2afe7e40d37c60a23c75bd10480ac903e29a0f5d8d184be6bcdf640e91` |
|
| 70 |
+
| 10000 | `wagi_10000.pt` | `fcfd2d3044d504706519fb6e1bd8d6dea959b162dac62e0f5967bbd6807df643` |
|
| 71 |
+
| 15000 | `wagi_15000.pt` | `1cc151d20cd1abf3b02a9dfbedb5c5b458861105307498acccfc15283d951ae4` |
|
| 72 |
+
| 20000 | `wagi_20000.pt` | `41a457dcdf1d60011fe576fe938d2d61e80ce71028c8f1b484face9ee3dc73d5` |
|
| 73 |
+
| 25000 | `wagi_25000.pt` | `3bd8e411635eb1a18fc6e96fcfee38b68850dbd20171f9cc73897187277ceda3` |
|
| 74 |
+
| 30000 | `wagi_30000.pt` | `74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d` |
|
| 75 |
+
| 35000 | `wagi_35000.pt` | `75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43` |
|
| 76 |
+
| 35000 | `ckpt_35000.pt` | `f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e` |
|
| 77 |
+
|
| 78 |
+
The same list is in `SHA256SUMS`.
|
| 79 |
+
|
| 80 |
+
## Limitations
|
| 81 |
+
|
| 82 |
+
These checkpoints are intermediate states of a constant-learning-rate run. Training loss and evaluation results at a given
|
| 83 |
+
update are on Track; they are not a quality verdict on a finished model. Changes of the data segment during the run (for
|
| 84 |
+
example at update 10,000) change the training-loss distribution, so a drop across such a boundary does not by itself show a
|
| 85 |
+
quality gain.
|