Card and SHA256SUMS: README 6cb909e8, SHA256SUMS d6e7d47b
Browse files
README.md
CHANGED
|
@@ -17,7 +17,7 @@ continued pretraining from the final weights of [Slayer1B 1.08](https://huggingf
|
|
| 17 |
optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a
|
| 18 |
Transformers/safetensors package and not a finished model.
|
| 19 |
|
| 20 |
-
**Status: training
|
| 21 |
This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 40,000 and one full-state
|
| 22 |
checkpoint at update 35,000 (update 40,000 added on 13 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
|
| 23 |
Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
|
|
@@ -29,7 +29,7 @@ the LFS object id (sha256) and the size match.
|
|
| 29 |
| Updates so far | 1 → 40,000 in this repository (40,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
|
| 30 |
| Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
|
| 31 |
| Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
|
| 32 |
-
| Hardware | updates 1–40,000: 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once.
|
| 33 |
| Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (sha256 `db36c150…`) |
|
| 34 |
| Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
|
| 35 |
| Tokenizer | v6, vocabulary 32,768 (sha256 `0640d3bd…`), as in Slayer1B 1.08 |
|
|
|
|
| 17 |
optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a
|
| 18 |
Transformers/safetensors package and not a finished model.
|
| 19 |
|
| 20 |
+
**Status: training in progress (not finished).** It was paused from 12 to 14 November 2026, resumed on 14 November from update 40,000, and moved to 8 × NVIDIA B300 on 15 November (see Hardware).
|
| 21 |
This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 40,000 and one full-state
|
| 22 |
checkpoint at update 35,000 (update 40,000 added on 13 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
|
| 23 |
Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
|
|
|
|
| 29 |
| Updates so far | 1 → 40,000 in this repository (40,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
|
| 30 |
| Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
|
| 31 |
| Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
|
| 32 |
+
| Hardware | updates 1–40,000: 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once. Updates 40,001–42,800: 2 × NVIDIA RTX 4090 with the same 256 × 4,096 tokens per update (gradient accumulation); a 100-update check (updates 35,001–35,100) matched the earlier hardware's training loss within 0.0002 on all 10 logged points (mean difference 0.00005). Updates from 42,801: 8 × NVIDIA B300, spot instance, resumed from the full checkpoint at update 42,800; the RTX 4090 had already computed updates 42,801–42,880, and on the 8 logged points in that range the B300 training loss matched it within 0.0003 (mean difference −0.0002). The public Track curve keeps the RTX 4090 points up to update 42,880 and shows B300 points after it |
|
| 33 |
| Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (sha256 `db36c150…`) |
|
| 34 |
| Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
|
| 35 |
| Tokenizer | v6, vocabulary 32,768 (sha256 `0640d3bd…`), as in Slayer1B 1.08 |
|