README.md

---
language:
- pl
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- from-scratch
datasets:
- SlayerLab/slayer-pl-8x3b
---

# Slayer1B 1.02

Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.02 is Slayer1B 1.01 trained
further**: training resumed from the last checkpoint of 1.01 before its learning-rate decay (step 2250) and continued to
5720 steps, i.e. twice the tokens of 1.01 (5 997 854 720 tokens, ≈ 5.6 tokens per parameter).

**From 1.02 on, the learning rate is not decayed at the end of a release.** Pre-training continues on the whole prepared
corpus of about 24 billion tokens, and the final learning-rate decay is applied once, at the end of the whole Slayer1B
line. Each release until then is a constant-learning-rate state; 1.00 and 1.01 ended with a decay. Scores of 1.02 and
later releases are therefore not directly comparable to 1.01: on the same batches at step 2800, the training loss of
1.01 (in its decay) was 2.507 and that of 1.02 (constant lr) 2.581.

- Previous versions: [SlayerLab/Slayer1B-1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01),
  [SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00)
- Training checkpoints (with optimizer state) and run record: [SlayerLab/Slayer1B-1.02-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.02-checkpoints)
- Training curve and evaluations during training: https://track.fabryka.ai/run/21ae1234-82c2-42b7-887b-cbcc69901ae1

## Results

Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot,
BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final
checkpoint of each release.

| | 1.00 | 1.01 | **1.02** |
|---|---:|---:|---:|
| training tokens | 1.50 B | 3.00 B | **6.00 B** |
| end of training | lr decay | lr decay | **constant lr (no decay)** |
| ARC-Easy, accuracy % (2 376) | 42.89 | 47.98 | **48.99** |
| BLiMP, accuracy % (67 000) | 71.51 | 73.63 | **74.71** |
| WikiText-2, byte perplexity ↓ | 2.5857 | 2.4230 | **2.3735** |
| MultiBLiMP Polish, accuracy % (3 272) | 97.89 | 98.29 | **98.38** |
| ARC-Easy-PL, accuracy % (2 376) | 36.91 | 40.36 | **41.84** |

Paired comparison 1.01 (step 2860) → 1.02 (step 5720), same items, paired bootstrap (10 000 resamples, 95 % interval):

| task | difference, pp | 95 % interval | |
|---|---:|---|---|
| BLiMP | +1.08 | [+0.78; +1.37] | positive on this test |
| ARC-Easy-PL | +1.47 | [−0.13; +3.07] | unresolved |
| ARC-Easy | +1.01 | [−0.67; +2.74] | unresolved |
| MultiBLiMP Polish | +0.09 | [−0.40; +0.58] | unresolved |

This comparison combines two changes at once — twice the training tokens and no final learning-rate decay — so no
difference can be attributed to either of them separately.

## Usage

The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class
is `GPT` in `train_gpt_ref_r23.py` (included).

```python
import torch, load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".")                      # directory with the files of this repository
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
logits, _ = model(ids)                             # [batch, seq, 32768]
```

## Model

| | |
|---|---|
| Parameters | 1 073 912 851 (input and output embeddings tied, counted once) |
| Architecture | decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) |
| Blocks | pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases |
| Context | 4096 tokens |
| Tokenizer | byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 |

## Training data

Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`. One stream of 4096-token
windows, each window at most once; the same order as 1.01 up to its step 2860:

| steps | pack | `train.bin` sha256 | tokens available | batches used |
|---|---|---|---:|---:|
| 1–1801 | pack-00 | `760ef0cc…` | 1 889 519 627 | 1801 |
| 1802–3594 | pack-01 | `0494df58…` | 1 880 520 445 | 1793 |
| 3595–5389 | pack-02 | `294a444d…` | 1 883 249 429 | 1795 |
| 5390–5720 | pack-03 | `7d675734…` | 1 890 850 536 | 331 |

Each pack has its own disjoint web slice; core, legal and English documents of each pack are selected from a
document-hash interval disjoint from those of earlier packs. Mixing weights (cl100k tokens): web 0.63, core 0.19,
legal 0.08, English 0.10. Shares of this tokenizer's tokens in the training split: pack-02 web 52.09 %, core 21.63 %,
legal 8.66 %, English 17.62 %; pack-03 51.68 / 22.27 / 8.59 / 17.46 (pack-00 and pack-01: see the 1.01 card).
Personal data: same rule-based masking as in 1.00 (tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`,
`[DOWOD]`); masking does not guarantee removal of every personal datum. A 13-gram overlap check of pack-02 and pack-03
against the evaluation sets has not been run (pack-01: see the 1.01 card).

## Training

| | |
|---|---|
| Hardware | 8 × NVIDIA RTX 5090 (32 GB), DDP |
| Batch | 256 × 4096 = 1 048 576 tokens per step |
| Steps | resumed at 2250 (1.01 `ckpt_2250.pt`, sha256 `aeddfb02…`), ran 2251 → 5720 (22:17 → 07:23 UTC, ≈ 113 000 tokens/s) |
| Tokens | 5720 × 1 048 576 = 5 997 854 720 in total |
| Schedule | constant lr 6e-4 throughout. The run was started with a final decay planned from step 4578; at step 3300, still at constant lr, it was resumed with the decay removed (training loss at step 3310 before and after equal to four decimal places) |
| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
| Final training loss | 2.4187 (step 5720); mean of the values logged at steps 5630–5720: 2.4379 (1.01 at the end of its decay, steps 2770–2860: 2.5083) |
| Resume check | training loss at steps 2260, 2270, 2280 equal to 1.01 at the same steps (same batches, same lr) to four decimals |
| Checkpoints | every 150 steps from 2400, step 4576 and step 5720; all are constant-lr states |
| Trainer | `train_gpt_ref_r24.py`: the 1.00/1.01 trainer plus a chain of data streams and extra checkpoint steps; the model definition is unchanged (`train_gpt_ref_r23.py` in this repository) |

## Limitations

- 1 073 912 851 parameters trained on 5 997 854 720 tokens (about 5.6 tokens per parameter).
- Released without a final learning-rate decay (see above).
- Polish-dominant data; English is a minority of the mixture.
- Base model: no instruction tuning, no safety tuning.

## License

Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions (see the 1.00
card and the dataset card).