Maggio33 commited on
Commit
067a78c
·
verified ·
1 Parent(s): cff81fd

Card and SHA256SUMS: README 6cb909e8, SHA256SUMS d6e7d47b

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -17,7 +17,7 @@ continued pretraining from the final weights of [Slayer1B 1.08](https://huggingf
17
  optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a
18
  Transformers/safetensors package and not a finished model.
19
 
20
- **Status: training resumed on 14 November 2026 from update 40,000 (not finished).** It was paused from 12 to 14 November.
21
  This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 40,000 and one full-state
22
  checkpoint at update 35,000 (update 40,000 added on 13 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
23
  Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
@@ -29,7 +29,7 @@ the LFS object id (sha256) and the size match.
29
  | Updates so far | 1 → 40,000 in this repository (40,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
30
  | Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
31
  | Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
32
- | Hardware | updates 1–40,000: 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once. Training updates from 40,001: 2 × NVIDIA RTX 4090 with the same 256 × 4,096 tokens per update (gradient accumulation); a 100-update check (updates 35,001–35,100) matched the earlier hardware's training loss within 0.0002 on all 10 logged points (mean difference 0.00005) |
33
  | Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (sha256 `db36c150…`) |
34
  | Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
35
  | Tokenizer | v6, vocabulary 32,768 (sha256 `0640d3bd…`), as in Slayer1B 1.08 |
 
17
  optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a
18
  Transformers/safetensors package and not a finished model.
19
 
20
+ **Status: training in progress (not finished).** It was paused from 12 to 14 November 2026, resumed on 14 November from update 40,000, and moved to 8 × NVIDIA B300 on 15 November (see Hardware).
21
  This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 40,000 and one full-state
22
  checkpoint at update 35,000 (update 40,000 added on 13 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
23
  Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
 
29
  | Updates so far | 1 → 40,000 in this repository (40,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
30
  | Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
31
  | Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
32
+ | Hardware | updates 1–40,000: 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once. Updates 40,001–42,800: 2 × NVIDIA RTX 4090 with the same 256 × 4,096 tokens per update (gradient accumulation); a 100-update check (updates 35,001–35,100) matched the earlier hardware's training loss within 0.0002 on all 10 logged points (mean difference 0.00005). Updates from 42,801: 8 × NVIDIA B300, spot instance, resumed from the full checkpoint at update 42,800; the RTX 4090 had already computed updates 42,801–42,880, and on the 8 logged points in that range the B300 training loss matched it within 0.0003 (mean difference −0.0002). The public Track curve keeps the RTX 4090 points up to update 42,880 and shows B300 points after it |
33
  | Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (sha256 `db36c150…`) |
34
  | Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
35
  | Tokenizer | v6, vocabulary 32,768 (sha256 `0640d3bd…`), as in Slayer1B 1.08 |