Model card and SHA256SUMS of the release files

README.md

@@ -0,0 +1,130 @@
+---
+language:
+- pl
+- en
+license: apache-2.0
+library_name: pytorch
+pipeline_tag: text-generation
+tags:
+- polish
+- causal-lm
+- from-scratch
+datasets:
+- SlayerLab/slayer-pl-8x3b
+---
+
+# Slayer1B 1.02
+
+Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.02 is Slayer1B 1.01 trained
+further**: training resumed from the last checkpoint of 1.01 before its learning-rate decay (step 2250) and continued to
+5720 steps, i.e. twice the tokens of 1.01 (5 997 854 720 tokens, ≈ 5.6 tokens per parameter).
+
+**From 1.02 on, the learning rate is not decayed at the end of a release.** Pre-training continues on the whole prepared
+corpus of about 24 billion tokens, and the final learning-rate decay is applied once, at the end of the whole Slayer1B
+line. Each release until then is a constant-learning-rate state; 1.00 and 1.01 ended with a decay. Scores of 1.02 and
+later releases are therefore not directly comparable to 1.01: on the same batches at step 2800, the training loss of
+1.01 (in its decay) was 2.507 and that of 1.02 (constant lr) 2.581.
+
+- Previous versions: [SlayerLab/Slayer1B-1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01),
+  [SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00)
+- Training checkpoints (with optimizer state) and run record: [SlayerLab/Slayer1B-1.02-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.02-checkpoints)
+- Training curve and evaluations during training: https://track.fabryka.ai/run/21ae1234-82c2-42b7-887b-cbcc69901ae1
+
+## Results
+
+Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot,
+BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final
+checkpoint of each release.
+
+| | 1.00 | 1.01 | **1.02** |
+|---|---:|---:|---:|
+| training tokens | 1.50 B | 3.00 B | **6.00 B** |
+| end of training | lr decay | lr decay | **constant lr (no decay)** |
+| ARC-Easy, accuracy % (2 376) | 42.89 | 47.98 | **48.99** |
+| BLiMP, accuracy % (67 000) | 71.51 | 73.63 | **74.71** |
+| WikiText-2, byte perplexity ↓ | 2.5857 | 2.4230 | **2.3735** |
+| MultiBLiMP Polish, accuracy % (3 272) | 97.89 | 98.29 | **98.38** |
+| ARC-Easy-PL, accuracy % (2 376) | 36.91 | 40.36 | **41.84** |
+
+Paired comparison 1.01 (step 2860) → 1.02 (step 5720), same items, paired bootstrap (10 000 resamples, 95 % interval):
+
+| task | difference, pp | 95 % interval | |
+|---|---:|---|---|
+| BLiMP | +1.08 | [+0.78; +1.37] | positive on this test |
+| ARC-Easy-PL | +1.47 | [−0.13; +3.07] | unresolved |
+| ARC-Easy | +1.01 | [−0.67; +2.74] | unresolved |
+| MultiBLiMP Polish | +0.09 | [−0.40; +0.58] | unresolved |
+
+This comparison combines two changes at once — twice the training tokens and no final learning-rate decay — so no
+difference can be attributed to either of them separately.
+
+## Usage
+
+The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class
+is `GPT` in `train_gpt_ref_r23.py` (included).
+
+```python
+import torch, load_slayer
+from tokenizers import Tokenizer
+model = load_slayer.load(".")                      # directory with the files of this repository
+tok = Tokenizer.from_file("tokenizer.json")
+ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
+logits, _ = model(ids)                             # [batch, seq, 32768]
+```
+
+## Model
+
+| | |
+|---|---|
+| Parameters | 1 073 912 851 (input and output embeddings tied, counted once) |
+| Architecture | decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) |
+| Blocks | pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases |
+| Context | 4096 tokens |
+| Tokenizer | byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 |
+
+## Training data
+
+Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`. One stream of 4096-token
+windows, each window at most once; the same order as 1.01 up to its step 2860:
+
+| steps | pack | `train.bin` sha256 | tokens available | batches used |
+|---|---|---|---:|---:|
+| 1–1801 | pack-00 | `760ef0cc…` | 1 889 519 627 | 1801 |
+| 1802–3594 | pack-01 | `0494df58…` | 1 880 520 445 | 1793 |
+| 3595–5389 | pack-02 | `294a444d…` | 1 883 249 429 | 1795 |
+| 5390–5720 | pack-03 | `7d675734…` | 1 890 850 536 | 331 |
+
+Each pack has its own disjoint web slice; core, legal and English documents of each pack are selected from a
+document-hash interval disjoint from those of earlier packs. Mixing weights (cl100k tokens): web 0.63, core 0.19,
+legal 0.08, English 0.10. Shares of this tokenizer's tokens in the training split: pack-02 web 52.09 %, core 21.63 %,
+legal 8.66 %, English 17.62 %; pack-03 51.68 / 22.27 / 8.59 / 17.46 (pack-00 and pack-01: see the 1.01 card).
+Personal data: same rule-based masking as in 1.00 (tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`,
+`[DOWOD]`); masking does not guarantee removal of every personal datum. A 13-gram overlap check of pack-02 and pack-03
+against the evaluation sets has not been run (pack-01: see the 1.01 card).
+
+## Training
+
+| | |
+|---|---|
+| Hardware | 8 × NVIDIA RTX 5090 (32 GB), DDP |
+| Batch | 256 × 4096 = 1 048 576 tokens per step |
+| Steps | resumed at 2250 (1.01 `ckpt_2250.pt`, sha256 `aeddfb02…`), ran 2251 → 5720 (22:17 → 07:23 UTC, ≈ 113 000 tokens/s) |
+| Tokens | 5720 × 1 048 576 = 5 997 854 720 in total |
+| Schedule | constant lr 6e-4 throughout. The run was started with a final decay planned from step 4578; at step 3300, still at constant lr, it was resumed with the decay removed (training loss at step 3310 identical before and after) |
+| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
+| Final training loss | 2.4187 (step 5720); mean of the values logged at steps 5630–5720: 2.4379 (1.01 at the end of its decay, steps 2770–2860: 2.5083) |
+| Resume check | training loss at steps 2260, 2270, 2280 equal to 1.01 at the same steps (same batches, same lr) to four decimals |
+| Checkpoints | every 150 steps from 2400, step 4576 and step 5720; all are constant-lr states |
+| Trainer | `train_gpt_ref_r24.py`: the 1.00/1.01 trainer plus a chain of data streams and extra checkpoint steps; the model definition is unchanged (`train_gpt_ref_r23.py` in this repository) |
+
+## Limitations
+
+- 1 073 912 851 parameters trained on 5 997 854 720 tokens (about 5.6 tokens per parameter).
+- Released without a final learning-rate decay (see above).
+- Polish-dominant data; English is a minority of the mixture.
+- Base model: no instruction tuning, no safety tuning.
+
+## License
+
+Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions (see the 1.00
+card and the dataset card).

SHA256SUMS

@@ -0,0 +1,5 @@
+1c7cfc4bb1bd1dd1ce23c1b0982281b3f441ef3fb5d3994484787ebd12545d39  config.json
+53a04c726bebbf8d8a25ff1479ab4f70c3003022675d87332d130139cca1d020  load_slayer.py
+22583557d1b2f2af05fd5565a718ffc270298da22a785b40027c86c34e12b9e0  model.safetensors
+0640d3bd3674d7a6e59c540d945a2ad275a12e1bcaeb6007c37a90751f9815dd  tokenizer.json
+db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b  train_gpt_ref_r23.py