Maggio33 commited on
Commit
ae19d21
·
verified ·
1 Parent(s): 4054346

Add model card (sha256 c40087bd)

Browse files
Files changed (1) hide show
  1. README.md +85 -0
README.md ADDED
@@ -0,0 +1,85 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: pytorch
4
+ tags:
5
+ - polish
6
+ - checkpoints
7
+ - training-history
8
+ - training-in-progress
9
+ ---
10
+
11
+ # Slayer1B 1.09 — training checkpoints
12
+
13
+ **Arkadiusz Słota, Anna Sieczka, Piotr Styła / Fabryka AI (SlayerLab)**
14
+
15
+ Native training checkpoints of the run that will produce [Slayer1B 1.09](https://huggingface.co/SlayerLab/Slayer1B-1.09):
16
+ continued pretraining from the final weights of [Slayer1B 1.08](https://huggingface.co/SlayerLab/Slayer1B-1.08) with a fresh
17
+ optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a
18
+ Transformers/safetensors package and not a finished model.
19
+
20
+ **Status: training in progress.** This repository holds every weights-only checkpoint saved at a 5,000-update boundary up
21
+ to update 35,000 and one full-state checkpoint at update 35,000. Later checkpoints will be added as the run continues. The
22
+ training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a). Every file
23
+ listed below was compared after upload with the checksum written when it was copied from the training machine: the LFS
24
+ object id (sha256) and the size match.
25
+
26
+ | | |
27
+ |---|---|
28
+ | Start | final weights of Slayer1B 1.08 (revision `5f854a0c`), fresh optimizer state, update 0 |
29
+ | Updates so far | 1 → 35,000 in this repository (35,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
30
+ | Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
31
+ | Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
32
+ | Hardware | 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once |
33
+ | Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (sha256 `db36c150…`) |
34
+ | Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
35
+ | Tokenizer | v6, vocabulary 32,768 (sha256 `0640d3bd…`), as in Slayer1B 1.08 |
36
+
37
+ ## Files
38
+
39
+ | file | contents | size |
40
+ |---|---|---:|
41
+ | `wagi_<update>.pt` | model weights (float32), `config`, `step` | 4.3 GB |
42
+ | `ckpt_35000.pt` | model weights, optimizer state (Muon and AdamW), data-queue position, `config`, `step` | 8.9 GB |
43
+
44
+ `wagi` is Polish for “weights”. Load with `torch.load(path, map_location="cpu")`; the model definition is in
45
+ `train_gpt_ref_r23.py` in the [1.08 repository](https://huggingface.co/SlayerLab/Slayer1B-1.08).
46
+
47
+ ## Tokens per checkpoint
48
+
49
+ Two different counts, kept apart on purpose. **Positions in this run** = update × 1,048,576 (training positions processed by
50
+ the 1.09 run itself, counting each update once). **Lineage lower bound** = positions inherited from Slayer1B 1.00–1.08
51
+ (18,863,257,682, source- and file-disjoint) + positions in this run − 114,627,128 positions identified as copies in the
52
+ first data segment. The lineage figure is a lower bound on source- and file-disjoint positions seen by the model line, not a
53
+ count of globally deduplicated unique tokens.
54
+
55
+ | update | positions in this run (billions) | lineage lower bound (billions) |
56
+ |---:|---:|---:|
57
+ | 5000 | 5.24 | 23.99 |
58
+ | 10000 | 10.49 | 29.23 |
59
+ | 15000 | 15.73 | 34.48 |
60
+ | 20000 | 20.97 | 39.72 |
61
+ | 25000 | 26.21 | 44.96 |
62
+ | 30000 | 31.46 | 50.21 |
63
+ | 35000 | 36.70 | 55.45 |
64
+
65
+ ## SHA-256
66
+
67
+ | update | file | sha256 |
68
+ |---:|---|---|
69
+ | 5000 | `wagi_5000.pt` | `bc289e2afe7e40d37c60a23c75bd10480ac903e29a0f5d8d184be6bcdf640e91` |
70
+ | 10000 | `wagi_10000.pt` | `fcfd2d3044d504706519fb6e1bd8d6dea959b162dac62e0f5967bbd6807df643` |
71
+ | 15000 | `wagi_15000.pt` | `1cc151d20cd1abf3b02a9dfbedb5c5b458861105307498acccfc15283d951ae4` |
72
+ | 20000 | `wagi_20000.pt` | `41a457dcdf1d60011fe576fe938d2d61e80ce71028c8f1b484face9ee3dc73d5` |
73
+ | 25000 | `wagi_25000.pt` | `3bd8e411635eb1a18fc6e96fcfee38b68850dbd20171f9cc73897187277ceda3` |
74
+ | 30000 | `wagi_30000.pt` | `74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d` |
75
+ | 35000 | `wagi_35000.pt` | `75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43` |
76
+ | 35000 | `ckpt_35000.pt` | `f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e` |
77
+
78
+ The same list is in `SHA256SUMS`.
79
+
80
+ ## Limitations
81
+
82
+ These checkpoints are intermediate states of a constant-learning-rate run. Training loss and evaluation results at a given
83
+ update are on Track; they are not a quality verdict on a finished model. Changes of the data segment during the run (for
84
+ example at update 10,000) change the training-loss distribution, so a drop across such a boundary does not by itself show a
85
+ quality gain.