Maggio33 commited on
Commit
33b6f10
Β·
verified Β·
1 Parent(s): b7f09b8

Card and SHA256SUMS for update 50000: README 9cddfedf, SHA256SUMS 6b6deee4

Browse files
Files changed (2) hide show
  1. README.md +10 -5
  2. SHA256SUMS +1 -0
README.md CHANGED
@@ -17,19 +17,19 @@ continued pretraining from the final weights of [Slayer1B 1.08](https://huggingf
17
  optimizer state. These are PyTorch files written by the trainer, for resuming or studying training β€” not a
18
  Transformers/safetensors package and not a finished model.
19
 
20
- **Status: training in progress (not finished).** It was paused from 12 to 14 November 2026, resumed on 14 November from update 40,000, and moved to 8 Γ— NVIDIA B300 on 15 November (see Hardware).
21
- This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 45,000 and one full-state
22
- checkpoint at update 35,000 (update 40,000 added on 13 November 2026, update 45,000 on 15 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
23
  Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
24
  the LFS object id (sha256) and the size match.
25
 
26
  | | |
27
  |---|---|
28
  | Start | final weights of Slayer1B 1.08 (revision `5f854a0c`), fresh optimizer state, update 0 |
29
- | Updates so far | 1 β†’ 45,000 in this repository (45,000 Γ— 1,048,576 tokens), 256 sequences Γ— 4,096 tokens per update |
30
  | Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
31
  | Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
32
- | Hardware | updates 1–40,000: 8 Γ— NVIDIA B200 or 8 Γ— H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once. Updates 40,001–42,800: 2 Γ— NVIDIA RTX 4090 with the same 256 Γ— 4,096 tokens per update (gradient accumulation); a 100-update check (updates 35,001–35,100) matched the earlier hardware's training loss within 0.0002 on all 10 logged points (mean difference 0.00005). Updates from 42,801: 8 Γ— NVIDIA B300, spot instance, resumed from the full checkpoint at update 42,800; the RTX 4090 had already computed updates 42,801–42,880, and on the 8 logged points in that range the B300 training loss matched it within 0.0003 (mean difference βˆ’0.0002). The public Track curve keeps the RTX 4090 points up to update 42,880 and shows B300 points after it |
33
  | Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (sha256 `db36c150…`) |
34
  | Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
35
  | Tokenizer | v6, vocabulary 32,768 (sha256 `0640d3bd…`), as in Slayer1B 1.08 |
@@ -63,6 +63,7 @@ count of globally deduplicated unique tokens.
63
  | 35000 | 36.70 | 55.45 |
64
  | 40000 | 41.94 | 60.69 |
65
  | 45000 | 47.19 | 65.93 |
 
66
 
67
  ## SHA-256
68
 
@@ -77,6 +78,7 @@ count of globally deduplicated unique tokens.
77
  | 35000 | `wagi_35000.pt` | `75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43` |
78
  | 40000 | `wagi_40000.pt` | `f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881` |
79
  | 45000 | `wagi_45000.pt` | `e6dd131a1e7b6a25d0421260bb8a24a52a95f48b01cfecb183987c65bb97d827` |
 
80
  | 35000 | `ckpt_35000.pt` | `f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e` |
81
 
82
  The same list is in `SHA256SUMS`.
@@ -87,3 +89,6 @@ These checkpoints are intermediate states of a constant-learning-rate run. Train
87
  update are on Track; they are not a quality verdict on a finished model. Changes of the data segment during the run (for
88
  example at update 10,000) change the training-loss distribution, so a drop across such a boundary does not by itself show a
89
  quality gain.
 
 
 
 
17
  optimizer state. These are PyTorch files written by the trainer, for resuming or studying training β€” not a
18
  Transformers/safetensors package and not a finished model.
19
 
20
+ **Status: training in progress (not finished).** It was paused from 12 to 14 November 2026, resumed on 14 November from update 40,000, moved to 8 Γ— NVIDIA B300 on 15 November, continued on NVIDIA RTX 4090 from update 47,200 and on 8 Γ— NVIDIA B200 from update 47,700 on 16 November, and resumed on 8 Γ— NVIDIA B200 from update 48,600 on 17 November (see Hardware).
21
+ This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 50,000 and one full-state
22
+ checkpoint at update 35,000 (update 40,000 added on 13 November 2026, update 45,000 on 15 November 2026, update 50,000 on 17 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
23
  Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
24
  the LFS object id (sha256) and the size match.
25
 
26
  | | |
27
  |---|---|
28
  | Start | final weights of Slayer1B 1.08 (revision `5f854a0c`), fresh optimizer state, update 0 |
29
+ | Updates so far | 1 β†’ 50,000 in this repository (50,000 Γ— 1,048,576 tokens), 256 sequences Γ— 4,096 tokens per update |
30
  | Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
31
  | Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
32
+ | Hardware | updates 1–40,000: 8 Γ— NVIDIA B200 or 8 Γ— H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once. Updates 40,001–42,800: 2 Γ— NVIDIA RTX 4090 with the same 256 Γ— 4,096 tokens per update (gradient accumulation); a 100-update check (updates 35,001–35,100) matched the earlier hardware's training loss within 0.0002 on all 10 logged points (mean difference 0.00005). Updates 42,801–47,200: 8 Γ— NVIDIA B300, spot instance, resumed from the full checkpoint at update 42,800; the RTX 4090 had already computed updates 42,801–42,880, and on the 8 logged points in that range the B300 training loss matched it within 0.0003 (mean difference βˆ’0.0002). The public Track curve keeps the RTX 4090 points up to update 42,880 and shows B300 points after it. Updates 47,201–47,700: 2 Γ— NVIDIA RTX 4090 (same 256 Γ— 4,096 tokens per update), resumed from the full checkpoint at update 47,200. Updates from 47,701: 8 Γ— NVIDIA B200, spot instances, resumed from the full checkpoint at update 47,700; 4 Γ— NVIDIA RTX 4090 had already computed updates 47,701–47,730, and on the 3 logged points in that range the absolute training-loss difference was below 0.00014 (a local sanity check, not a trajectory guarantee). The public Track curve keeps the RTX 4090 points up to update 47,730 and shows B200 points after it |
33
  | Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (sha256 `db36c150…`) |
34
  | Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
35
  | Tokenizer | v6, vocabulary 32,768 (sha256 `0640d3bd…`), as in Slayer1B 1.08 |
 
63
  | 35000 | 36.70 | 55.45 |
64
  | 40000 | 41.94 | 60.69 |
65
  | 45000 | 47.19 | 65.93 |
66
+ | 50000 | 52.43 | 71.18 |
67
 
68
  ## SHA-256
69
 
 
78
  | 35000 | `wagi_35000.pt` | `75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43` |
79
  | 40000 | `wagi_40000.pt` | `f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881` |
80
  | 45000 | `wagi_45000.pt` | `e6dd131a1e7b6a25d0421260bb8a24a52a95f48b01cfecb183987c65bb97d827` |
81
+ | 50000 | `wagi_50000.pt` | `4a913162116e78e497399b20cccaf896aa79663f7bbca31a6a7964b72001b20f` |
82
  | 35000 | `ckpt_35000.pt` | `f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e` |
83
 
84
  The same list is in `SHA256SUMS`.
 
89
  update are on Track; they are not a quality verdict on a finished model. Changes of the data segment during the run (for
90
  example at update 10,000) change the training-loss distribution, so a drop across such a boundary does not by itself show a
91
  quality gain.
92
+
93
+ Checkpoint recovery can recompute updates from the last saved state. Public history retains first observations at each step;
94
+ replay does not add distinct updates or token counts. Numerical trajectories need not be bit-identical across hardware.
SHA256SUMS CHANGED
@@ -7,4 +7,5 @@ fcfd2d3044d504706519fb6e1bd8d6dea959b162dac62e0f5967bbd6807df643 wagi_10000.pt
7
  75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43 wagi_35000.pt
8
  f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881 wagi_40000.pt
9
  e6dd131a1e7b6a25d0421260bb8a24a52a95f48b01cfecb183987c65bb97d827 wagi_45000.pt
 
10
  f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e ckpt_35000.pt
 
7
  75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43 wagi_35000.pt
8
  f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881 wagi_40000.pt
9
  e6dd131a1e7b6a25d0421260bb8a24a52a95f48b01cfecb183987c65bb97d827 wagi_45000.pt
10
+ 4a913162116e78e497399b20cccaf896aa79663f7bbca31a6a7964b72001b20f wagi_50000.pt
11
  f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e ckpt_35000.pt