Model card (training facts; results to follow)

README.md

@@ -0,0 +1,96 @@
+---
+language:
+- pl
+- en
+license: apache-2.0
+library_name: pytorch
+pipeline_tag: text-generation
+tags:
+- polish
+- causal-lm
+- from-scratch
+datasets:
+- SlayerLab/slayer-pl-8x3b
+---
+
+# Slayer1B 1.01
+
+Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.01 is Slayer1B 1.00 trained
+further**: training resumed from the last checkpoint of 1.00 before its learning-rate decay (step 1050) and continued to
+2860 steps, i.e. twice the tokens per parameter of 1.00 (2 998 927 360 tokens, ≈ 2.8 tokens per parameter).
+
+- Previous version: [SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00)
+- Training checkpoints (with optimizer state): [SlayerLab/Slayer1B-1.01-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.01-checkpoints)
+- Training curve and evaluations during training: https://track.fabryka.ai/run/4934429a-d393-42e0-8aac-03eae6b71ff7
+
+## Results
+
+The final evaluation of step 2860 (same protocol as the 1.00 card) is being computed and will be added to this section.
+
+## Usage
+
+The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class
+is `GPT` in `train_gpt_ref_r23.py` (included).
+
+```python
+import torch, load_slayer
+from tokenizers import Tokenizer
+model = load_slayer.load(".")                      # directory with the files of this repository
+tok = Tokenizer.from_file("tokenizer.json")
+ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
+logits, _ = model(ids)                             # [batch, seq, 32768]
+```
+
+## Model
+
+| | |
+|---|---|
+| Parameters | 1 073 912 851 (input and output embeddings tied, counted once) |
+| Architecture | decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) |
+| Blocks | pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases |
+| Context | 4096 tokens |
+| Tokenizer | byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 |
+
+## Training data
+
+Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`.
+
+- Steps 1–1050 (from 1.00) and 1051–1800: the pack-00 stream of 1.00 (`train.bin` sha256 `760ef0cc…`), each 4096-token
+  window at most once, the same windows as 1.00 up to its step 1050.
+- Steps 1801–2860: a new stream from pack-01 (its own disjoint web slice; core, legal and English documents selected
+  from a hash interval disjoint from the one used for pack-00), `train.bin` sha256 `0494df58…`,
+  1,880,520,445 tokens available, of which 1 059 × 1 048 576 were consumed.
+
+Mixing weights of the pack (cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10. In this tokenizer's tokens
+the pack-01 stream is web 52.1 %, core 22.1 %, legal 8.2 %, English 17.6 % (pack-00: 51.8 / 22.2 / 8.5 / 17.6).
+Personal data: same rule-based masking as in 1.00 (tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`,
+`[DOWOD]`); masking does not guarantee removal of every personal datum.
+
+Overlap with evaluation sets (13-gram, pack-01 stream vs. the task files): no evaluation item has 90 % or more of its
+13-grams in the stream; MultiBLiMP-pl has 10 of 3 272 items with ≥ 50 % overlap, WikiText-2 has 26 of 62 documents with
+any shared 13-gram (none ≥ 50 %).
+
+## Training
+
+| | |
+|---|---|
+| Hardware | 8 × NVIDIA RTX 5090 (32 GB), DDP |
+| Batch | 256 × 4096 = 1 048 576 tokens per step |
+| Steps | resumed at 1050, ran 1051 → 2860 (13:56 → 18:54 UTC, ≈ 108 000 tokens/s) |
+| Tokens | 2860 × 1 048 576 = 2 998 927 360 in total |
+| Schedule | warmup 150, constant lr 6e-4 to step 2287, 1 − sqrt decay from step 2288 to lr / 10 |
+| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
+| Final training loss | 2.5067 (step 2860); mean of the last 100 steps 2.5083 (1.00: 2.6990) |
+| Resume check | mean training loss over steps 1060–1140 equal to the 1.00 run within 0.0001 nats |
+| Checkpoints | every 150 steps + final; last before the decay: step 2250 |
+
+## Limitations
+
+- 1 073 912 851 parameters trained on 2 998 927 360 tokens (about 2.8 tokens per parameter).
+- Polish-dominant data; English is a minority of the mixture.
+- Base model: no instruction tuning, no safety tuning.
+
+## License
+
+Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions (see the 1.00
+card and the dataset card).