Maggio33 commited on
Commit
33d2fae
Β·
verified Β·
1 Parent(s): 389dc28

Card and SHA256SUMS for update 45000: README a6ac0346, SHA256SUMS f78eef79

Browse files
Files changed (2) hide show
  1. README.md +5 -3
  2. SHA256SUMS +1 -0
README.md CHANGED
@@ -18,15 +18,15 @@ optimizer state. These are PyTorch files written by the trainer, for resuming or
18
  Transformers/safetensors package and not a finished model.
19
 
20
  **Status: training in progress (not finished).** It was paused from 12 to 14 November 2026, resumed on 14 November from update 40,000, and moved to 8 Γ— NVIDIA B300 on 15 November (see Hardware).
21
- This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 40,000 and one full-state
22
- checkpoint at update 35,000 (update 40,000 added on 13 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
23
  Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
24
  the LFS object id (sha256) and the size match.
25
 
26
  | | |
27
  |---|---|
28
  | Start | final weights of Slayer1B 1.08 (revision `5f854a0c`), fresh optimizer state, update 0 |
29
- | Updates so far | 1 β†’ 40,000 in this repository (40,000 Γ— 1,048,576 tokens), 256 sequences Γ— 4,096 tokens per update |
30
  | Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
31
  | Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
32
  | Hardware | updates 1–40,000: 8 Γ— NVIDIA B200 or 8 Γ— H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once. Updates 40,001–42,800: 2 Γ— NVIDIA RTX 4090 with the same 256 Γ— 4,096 tokens per update (gradient accumulation); a 100-update check (updates 35,001–35,100) matched the earlier hardware's training loss within 0.0002 on all 10 logged points (mean difference 0.00005). Updates from 42,801: 8 Γ— NVIDIA B300, spot instance, resumed from the full checkpoint at update 42,800; the RTX 4090 had already computed updates 42,801–42,880, and on the 8 logged points in that range the B300 training loss matched it within 0.0003 (mean difference βˆ’0.0002). The public Track curve keeps the RTX 4090 points up to update 42,880 and shows B300 points after it |
@@ -62,6 +62,7 @@ count of globally deduplicated unique tokens.
62
  | 30000 | 31.46 | 50.21 |
63
  | 35000 | 36.70 | 55.45 |
64
  | 40000 | 41.94 | 60.69 |
 
65
 
66
  ## SHA-256
67
 
@@ -75,6 +76,7 @@ count of globally deduplicated unique tokens.
75
  | 30000 | `wagi_30000.pt` | `74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d` |
76
  | 35000 | `wagi_35000.pt` | `75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43` |
77
  | 40000 | `wagi_40000.pt` | `f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881` |
 
78
  | 35000 | `ckpt_35000.pt` | `f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e` |
79
 
80
  The same list is in `SHA256SUMS`.
 
18
  Transformers/safetensors package and not a finished model.
19
 
20
  **Status: training in progress (not finished).** It was paused from 12 to 14 November 2026, resumed on 14 November from update 40,000, and moved to 8 Γ— NVIDIA B300 on 15 November (see Hardware).
21
+ This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 45,000 and one full-state
22
+ checkpoint at update 35,000 (update 40,000 added on 13 November 2026, update 45,000 on 15 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
23
  Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
24
  the LFS object id (sha256) and the size match.
25
 
26
  | | |
27
  |---|---|
28
  | Start | final weights of Slayer1B 1.08 (revision `5f854a0c`), fresh optimizer state, update 0 |
29
+ | Updates so far | 1 β†’ 45,000 in this repository (45,000 Γ— 1,048,576 tokens), 256 sequences Γ— 4,096 tokens per update |
30
  | Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
31
  | Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
32
  | Hardware | updates 1–40,000: 8 Γ— NVIDIA B200 or 8 Γ— H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once. Updates 40,001–42,800: 2 Γ— NVIDIA RTX 4090 with the same 256 Γ— 4,096 tokens per update (gradient accumulation); a 100-update check (updates 35,001–35,100) matched the earlier hardware's training loss within 0.0002 on all 10 logged points (mean difference 0.00005). Updates from 42,801: 8 Γ— NVIDIA B300, spot instance, resumed from the full checkpoint at update 42,800; the RTX 4090 had already computed updates 42,801–42,880, and on the 8 logged points in that range the B300 training loss matched it within 0.0003 (mean difference βˆ’0.0002). The public Track curve keeps the RTX 4090 points up to update 42,880 and shows B300 points after it |
 
62
  | 30000 | 31.46 | 50.21 |
63
  | 35000 | 36.70 | 55.45 |
64
  | 40000 | 41.94 | 60.69 |
65
+ | 45000 | 47.19 | 65.93 |
66
 
67
  ## SHA-256
68
 
 
76
  | 30000 | `wagi_30000.pt` | `74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d` |
77
  | 35000 | `wagi_35000.pt` | `75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43` |
78
  | 40000 | `wagi_40000.pt` | `f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881` |
79
+ | 45000 | `wagi_45000.pt` | `e6dd131a1e7b6a25d0421260bb8a24a52a95f48b01cfecb183987c65bb97d827` |
80
  | 35000 | `ckpt_35000.pt` | `f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e` |
81
 
82
  The same list is in `SHA256SUMS`.
SHA256SUMS CHANGED
@@ -6,4 +6,5 @@ fcfd2d3044d504706519fb6e1bd8d6dea959b162dac62e0f5967bbd6807df643 wagi_10000.pt
6
  74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d wagi_30000.pt
7
  75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43 wagi_35000.pt
8
  f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881 wagi_40000.pt
 
9
  f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e ckpt_35000.pt
 
6
  74a222d941381c80c0f80a566b5351c823f2c5c246470591ec62ce2ca6e6d35d wagi_30000.pt
7
  75a5a95ece86cdb54e99ce4d66f3f392245f75780eef8b8c1ba6ed4e4a27de43 wagi_35000.pt
8
  f83f7017872f7121f002bc77499c625304bd28e2c87fb4afaa0e3d445bd31881 wagi_40000.pt
9
+ e6dd131a1e7b6a25d0421260bb8a24a52a95f48b01cfecb183987c65bb97d827 wagi_45000.pt
10
  f5de8f3123387634c53213110343c637cb1c77e3ed2d5f3a4ac44c7b569fd62e ckpt_35000.pt