Maggio33 commited on
Commit
cff81fd
·
verified ·
1 Parent(s): e9806e5

Card and SHA256SUMS: README c3401c32, SHA256SUMS d6e7d47b

Browse files
Files changed (1) hide show
  1. README.md +4 -4
README.md CHANGED
@@ -17,9 +17,9 @@ continued pretraining from the final weights of [Slayer1B 1.08](https://huggingf
17
  optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a
18
  Transformers/safetensors package and not a finished model.
19
 
20
- **Status: training paused since 12 November 2026 (not finished).** This repository holds every weights-only checkpoint saved
21
- at a 5,000-update boundary up to update 40,000 and one full-state checkpoint at update 35,000 (update 40,000 added on
22
- 13 November 2026). Later checkpoints will be added if the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
23
  Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
24
  the LFS object id (sha256) and the size match.
25
 
@@ -29,7 +29,7 @@ the LFS object id (sha256) and the size match.
29
  | Updates so far | 1 → 40,000 in this repository (40,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
30
  | Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
31
  | Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
32
- | Hardware | 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once |
33
  | Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (sha256 `db36c150…`) |
34
  | Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
35
  | Tokenizer | v6, vocabulary 32,768 (sha256 `0640d3bd…`), as in Slayer1B 1.08 |
 
17
  optimizer state. These are PyTorch files written by the trainer, for resuming or studying training — not a
18
  Transformers/safetensors package and not a finished model.
19
 
20
+ **Status: training resumed on 14 November 2026 from update 40,000 (not finished).** It was paused from 12 to 14 November.
21
+ This repository holds every weights-only checkpoint saved at a 5,000-update boundary up to update 40,000 and one full-state
22
+ checkpoint at update 35,000 (update 40,000 added on 13 November 2026). Later checkpoints will be added as the run continues. The training curve and evaluations are on [Track](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac47-0ea7a23f573a).
23
  Every file listed below was compared after upload with the checksum written when it was copied from the training machine:
24
  the LFS object id (sha256) and the size match.
25
 
 
29
  | Updates so far | 1 → 40,000 in this repository (40,000 × 1,048,576 tokens), 256 sequences × 4,096 tokens per update |
30
  | Schedule | Muon peak lr 0.02 + AdamW peak lr 0.0006, 2,000-update warmup, then **constant**; the run is configured for 120,000 updates with decay from update 96,000, which has not been reached |
31
  | Data | see the [data card](https://huggingface.co/SlayerLab/Slayer1B-1.09/blob/main/DATA_CARD.md); new data 20% Polish / 80% English by target allocation |
32
+ | Hardware | updates 1–40,000: 8 × NVIDIA B200 or 8 × H200, spot instances; after each preemption the run resumed from the most recent full checkpoint (written every 200 updates), so some updates were computed more than once. Training updates from 40,001: 2 × NVIDIA RTX 4090 with the same 256 × 4,096 tokens per update (gradient accumulation); a 100-update check (updates 35,001–35,100) matched the earlier hardware's training loss within 0.0002 on all 10 logged points (mean difference 0.00005) |
33
  | Trainer | `train_gpt_ref` r24.2 (sha256 `46f83b51…`); model definition `train_gpt_ref_r23.py` (sha256 `db36c150…`) |
34
  | Architecture | 1,073,912,851 parameters, 20 layers, width 2,048, 16 heads, SwiGLU, RMSNorm, RoPE base 500,000, QK normalization, value residuals |
35
  | Tokenizer | v6, vocabulary 32,768 (sha256 `0640d3bd…`), as in Slayer1B 1.08 |