Maggio33 commited on
Commit
e9806e5
·
verified ·
1 Parent(s): 297927a

Card and SHA256SUMS: README b7cbdaf1, SHA256SUMS d6e7d47b

Browse files
Files changed (2) hide show
  1. README.md +6 -4
  2. SHA256SUMS +1 -0
README.md CHANGED
@@ -17,16 +17,16 @@ continued pretraining from the final weights of [Slayer1B 1.08](https://huggingf
17
  optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a
18
  Transformers/safetensors package and not a finished model.
19
 
20
- **Status: training in progress.** This repository holds every weights-only checkpoint saved at a 5,000-update boundary up
21
- to update 35,000 and one full-state checkpoint at update 35,000 (uploaded 12 November 2026). Later checkpoints will be added
22
- as the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
23
  Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
24
  the LFS object id (sha256) and the size match.
25
 
26
  | | |
27
  |---|---|
28
  | Start | final weights of Slayer1B 1.08 (revision `5f854a0c`), fresh optimizer state, update 0 |
29
- | Updates so far | 1 → 35,000 in this repository (35,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
30
  | Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
31
  | Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
32
  | Hardware | 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once |
@@ -61,6 +61,7 @@ count of globally deduplicated unique tokens.
61
  | 25000 | 26.21 | 44.96 |
62
  | 30000 | 31.46 | 50.21 |
63
  | 35000 | 36.70 | 55.45 |
 
64
 
65
  ## SHA-256
66
 
@@ -73,6 +74,7 @@ count of globally deduplicated unique tokens.
73
  | 25000 | `wagi_25000.pt` | `3bd8e411635eb1a18fc6e96fcfee38b68850dbd20171f9cc73897187277ceda3` |
74
  | 30000 | `wagi_30000.pt` | `74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d` |
75
  | 35000 | `wagi_35000.pt` | `75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43` |
 
76
  | 35000 | `ckpt_35000.pt` | `f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e` |
77
 
78
  The same list is in `SHA256SUMS`.
 
17
  optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a
18
  Transformers/safetensors package and not a finished model.
19
 
20
+ **Status: training paused since 12 November 2026 (not finished).** This repository holds every weights-only checkpoint saved
21
+ at a 5,000-update boundary up to update 40,000 and one full-state checkpoint at update 35,000 (update 40,000 added on
22
+ 13 November 2026). Later checkpoints will be added if the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
23
  Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
24
  the LFS object id (sha256) and the size match.
25
 
26
  | | |
27
  |---|---|
28
  | Start | final weights of Slayer1B 1.08 (revision `5f854a0c`), fresh optimizer state, update 0 |
29
+ | Updates so far | 1 → 40,000 in this repository (40,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
30
  | Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
31
  | Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
32
  | Hardware | 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once |
 
61
  | 25000 | 26.21 | 44.96 |
62
  | 30000 | 31.46 | 50.21 |
63
  | 35000 | 36.70 | 55.45 |
64
+ | 40000 | 41.94 | 60.69 |
65
 
66
  ## SHA-256
67
 
 
74
  | 25000 | `wagi_25000.pt` | `3bd8e411635eb1a18fc6e96fcfee38b68850dbd20171f9cc73897187277ceda3` |
75
  | 30000 | `wagi_30000.pt` | `74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d` |
76
  | 35000 | `wagi_35000.pt` | `75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43` |
77
+ | 40000 | `wagi_40000.pt` | `f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881` |
78
  | 35000 | `ckpt_35000.pt` | `f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e` |
79
 
80
  The same list is in `SHA256SUMS`.
SHA256SUMS CHANGED
@@ -5,4 +5,5 @@ fcfd2d3044d504706519fb6e1bd8d6dea959b162dac62e0f5967bbd6807df643 wagi_10000.pt
5
  3bd8e411635eb1a18fc6e96fcfee38b68850dbd20171f9cc73897187277ceda3 wagi_25000.pt
6
  74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d wagi_30000.pt
7
  75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43 wagi_35000.pt
 
8
  f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e ckpt_35000.pt
 
5
  3bd8e411635eb1a18fc6e96fcfee38b68850dbd20171f9cc73897187277ceda3 wagi_25000.pt
6
  74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d wagi_30000.pt
7
  75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43 wagi_35000.pt
8
+ f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881 wagi_40000.pt
9
  f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e ckpt_35000.pt