--- language: - pl - en license: apache-2.0 library_name: pytorch pipeline_tag: text-generation tags: - polish - causal-lm - from-scratch datasets: - SlayerLab/slayer-pl-8x3b --- # Slayer1B 1.02 Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.02 is Slayer1B 1.01 trained further**: training resumed from the last checkpoint of 1.01 before its learning-rate decay (step 2250) and continued to 5720 steps, i.e. twice the tokens of 1.01 (5 997 854 720 tokens, ≈ 5.6 tokens per parameter). **From 1.02 on, the learning rate is not decayed at the end of a release.** Pre-training continues on the whole prepared corpus of about 24 billion tokens, and the final learning-rate decay is applied once, at the end of the whole Slayer1B line. Each release until then is a constant-learning-rate state; 1.00 and 1.01 ended with a decay. Scores of 1.02 and later releases are therefore not directly comparable to 1.01: on the same batches at step 2800, the training loss of 1.01 (in its decay) was 2.507 and that of 1.02 (constant lr) 2.581. - Previous versions: [SlayerLab/Slayer1B-1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01), [SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00) - Training checkpoints (with optimizer state) and run record: [SlayerLab/Slayer1B-1.02-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.02-checkpoints) - Training curve and evaluations during training: https://track.fabryka.ai/run/21ae1234-82c2-42b7-887b-cbcc69901ae1 ## Results Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot, BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final checkpoint of each release. | | 1.00 | 1.01 | **1.02** | |---|---:|---:|---:| | training tokens | 1.50 B | 3.00 B | **6.00 B** | | end of training | lr decay | lr decay | **constant lr (no decay)** | | ARC-Easy, accuracy % (2 376) | 42.89 | 47.98 | **48.99** | | BLiMP, accuracy % (67 000) | 71.51 | 73.63 | **74.71** | | WikiText-2, byte perplexity ↓ | 2.5857 | 2.4230 | **2.3735** | | MultiBLiMP Polish, accuracy % (3 272) | 97.89 | 98.29 | **98.38** | | ARC-Easy-PL, accuracy % (2 376) | 36.91 | 40.36 | **41.84** | Paired comparison 1.01 (step 2860) → 1.02 (step 5720), same items, paired bootstrap (10 000 resamples, 95 % interval): | task | difference, pp | 95 % interval | | |---|---:|---|---| | BLiMP | +1.08 | [+0.78; +1.37] | positive on this test | | ARC-Easy-PL | +1.47 | [−0.13; +3.07] | unresolved | | ARC-Easy | +1.01 | [−0.67; +2.74] | unresolved | | MultiBLiMP Polish | +0.09 | [−0.40; +0.58] | unresolved | This comparison combines two changes at once — twice the training tokens and no final learning-rate decay — so no difference can be attributed to either of them separately. ## Usage The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class is `GPT` in `train_gpt_ref_r23.py` (included). ```python import torch, load_slayer from tokenizers import Tokenizer model = load_slayer.load(".") # directory with the files of this repository tok = Tokenizer.from_file("tokenizer.json") ids = torch.tensor([tok.encode("Stolicą Polski jest").ids]) logits, _ = model(ids) # [batch, seq, 32768] ``` ## Model | | | |---|---| | Parameters | 1 073 912 851 (input and output embeddings tied, counted once) | | Architecture | decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) | | Blocks | pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases | | Context | 4096 tokens | | Tokenizer | byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 | ## Training data Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`. One stream of 4096-token windows, each window at most once; the same order as 1.01 up to its step 2860: | steps | pack | `train.bin` sha256 | tokens available | batches used | |---|---|---|---:|---:| | 1–1801 | pack-00 | `760ef0cc…` | 1 889 519 627 | 1801 | | 1802–3594 | pack-01 | `0494df58…` | 1 880 520 445 | 1793 | | 3595–5389 | pack-02 | `294a444d…` | 1 883 249 429 | 1795 | | 5390–5720 | pack-03 | `7d675734…` | 1 890 850 536 | 331 | Each pack has its own disjoint web slice; core, legal and English documents of each pack are selected from a document-hash interval disjoint from those of earlier packs. Mixing weights (cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10. Shares of this tokenizer's tokens in the training split: pack-02 web 52.09 %, core 21.63 %, legal 8.66 %, English 17.62 %; pack-03 51.68 / 22.27 / 8.59 / 17.46 (pack-00 and pack-01: see the 1.01 card). Personal data: same rule-based masking as in 1.00 (tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`, `[DOWOD]`); masking does not guarantee removal of every personal datum. A 13-gram overlap check of pack-02 and pack-03 against the evaluation sets has not been run (pack-01: see the 1.01 card). ## Training | | | |---|---| | Hardware | 8 × NVIDIA RTX 5090 (32 GB), DDP | | Batch | 256 × 4096 = 1 048 576 tokens per step | | Steps | resumed at 2250 (1.01 `ckpt_2250.pt`, sha256 `aeddfb02…`), ran 2251 → 5720 (22:17 → 07:23 UTC, ≈ 113 000 tokens/s) | | Tokens | 5720 × 1 048 576 = 5 997 854 720 in total | | Schedule | constant lr 6e-4 throughout. The run was started with a final decay planned from step 4578; at step 3300, still at constant lr, it was resumed with the decay removed (training loss at step 3310 before and after equal to four decimal places) | | Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest | | Final training loss | 2.4187 (step 5720); mean of the values logged at steps 5630–5720: 2.4379 (1.01 at the end of its decay, steps 2770–2860: 2.5083) | | Resume check | training loss at steps 2260, 2270, 2280 equal to 1.01 at the same steps (same batches, same lr) to four decimals | | Checkpoints | every 150 steps from 2400, step 4576 and step 5720; all are constant-lr states | | Trainer | `train_gpt_ref_r24.py`: the 1.00/1.01 trainer plus a chain of data streams and extra checkpoint steps; the model definition is unchanged (`train_gpt_ref_r23.py` in this repository) | ## Limitations - 1 073 912 851 parameters trained on 5 997 854 720 tokens (about 5.6 tokens per parameter). - Released without a final learning-rate decay (see above). - Polish-dominant data; English is a minority of the mixture. - Base model: no instruction tuning, no safety tuning. ## License Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions (see the 1.00 card and the dataset card).