README.md

---
language:
- pl
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- from-scratch
datasets:
- SlayerLab/slayer-pl-8x3b
---

# Slayer1B 1.03

Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.03 is Slayer1B 1.02 trained
further**: training resumed from the final checkpoint of 1.02 (step 5720, constant learning rate) and continued to
step 7150 (7 497 318 400 tokens in total, ≈ 7.0 tokens per parameter).

Like 1.02, **1.03 is a constant-learning-rate state**: the final learning-rate decay is applied once, at the end of the
whole Slayer1B line. Pre-training continues from this checkpoint on the rest of the prepared corpus.

- Previous versions: [SlayerLab/Slayer1B-1.02](https://huggingface.co/SlayerLab/Slayer1B-1.02),
  [SlayerLab/Slayer1B-1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01),
  [SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00)
- Training checkpoints (with optimizer state) and run record: [SlayerLab/Slayer1B-1.03-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.03-checkpoints)
- Training curve (steps up to 7150; 6860–7150 from the L40S resume): https://track.fabryka.ai/run/7ca4f84f-3b07-4525-8cde-f3a936dc3425
- Benchmark series 1.00–1.03: https://track.fabryka.ai/run/1426c0a8-2f07-5a03-80fc-92f7e922aa4b

## Results

lm-evaluation-harness 0.4.13, zero-shot, full test sets, fp32, batch 8, final checkpoint of each release. Same runner
(`runner_benchy`, sha256 `4f274a79…`) for all columns; the 1.03 column was computed in a different software runtime
(torch 2.8.0 / numpy 2.2.6 instead of torch 2.14.1 / numpy 2.5.3), so it is not an identical-environment comparison.

| | 1.00 (1430) | 1.01 (2860) | 1.02 (5720) | **1.03 (7150)** |
|---|---:|---:|---:|---:|
| training tokens | 1.50 B | 3.00 B | 6.00 B | **7.50 B** |
| end of training | lr decay | lr decay | constant lr | **constant lr** |
| HellaSwag, acc_norm | 0.2859 | 0.3080 | 0.3382 | **0.3476** |
| PIQA, acc | 0.5865 | 0.5903 | 0.6224 | **0.6257** |
| LAMBADA (OpenAI), acc | 0.1995 | 0.2639 | 0.3109 | **0.3150** |
| LAMBADA (OpenAI), perplexity ↓ | 159.66 | 69.37 | 45.77 | **42.83** |
| ARC-Challenge, acc_norm ¹ | 0.2142 | 0.2432 | 0.2500 | **0.2611** |
| BoolQ, acc ¹ | 0.5343 | 0.6083 | 0.6055 | **0.6055** |
| MMLU, acc ¹ | 0.2301 | 0.2308 | 0.2323 | **0.2292** |

¹ **Benchmark caveat.** A leak scan of the whole training superset (SlayerLab/slayer-pl-8x3b@d50df04a, 22 603 122
documents) found n-gram overlap with BoolQ, MMLU and ARC test items in the English shared part (`shared/en`: 1428 /
1429 / 1345 documents). Overlap is not a measured contamination rate; the assessment is in progress. 1.00–1.03 were
trained on the unfiltered data; read these three rows with this caveat. MMLU is near chance level (0.25) for all
releases.

Scores of single checkpoints differ: the intermediate 1.03 checkpoints 6900 / 7050 scored e.g. LAMBADA perplexity
40.78 / 45.68 and BoolQ 0.6180 / 0.5908 (all points in the benchmark series above); no cause is attributed. The
leaderboard protocol used in the 1.00–1.02 cards (`glint_parity_eval.py`) has not been run on 1.03 yet.

## Usage

The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class
is `GPT` in `train_gpt_ref_r23.py` (included).

```python
import torch, load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".")                      # directory with the files of this repository
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
logits, _ = model(ids)                             # [batch, seq, 32768]
```

## Model

| | |
|---|---|
| Parameters | 1 073 912 851 (input and output embeddings tied, counted once) |
| Architecture | decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) |
| Blocks | pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases |
| Context | 4096 tokens |
| Tokenizer | byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 |
| Weights | `model.safetensors`, fp32, sha256 `86a7760c57d4f4243efdee8b128af337c30cc95fbfad4530d9fe0ec467f3e5fa`, exported from `ckpt_7150.pt` (sha256 `d7cad967…`) |

## Training data

Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`. One stream of 4096-token
windows, each window at most once; 1.03 continues the stream of 1.02:

| steps | pack | `train.bin` sha256 | tokens available | batches used |
|---|---|---|---:|---:|
| 1–1801 | pack-00 | `760ef0cc…` | 1 889 519 627 | 1801 |
| 1802–3594 | pack-01 | `0494df58…` | 1 880 520 445 | 1793 |
| 3595–5389 | pack-02 | `294a444d…` | 1 883 249 429 | 1795 |
| 5390–7150 | pack-03 | `7d675734…` | 1 890 850 536 | 1761 |

Mixing, personal-data masking and pack construction as described in the 1.02 card. Personal data: rule-based masking
(tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`, `[DOWOD]`) does not guarantee removal of every personal
datum. The 5133 documents matching the leak scan are excluded only from pack-04 on (first filtered update: zero-based
7192, native step 7193), i.e. after 1.03.

## Training

| | |
|---|---|
| Hardware | steps 5721–6750: 4 × NVIDIA RTX 4090 (24 GB); steps 6751–7150: 4 × NVIDIA L40S (48 GB), resumed from `ckpt_6750.pt` (sha256 `970518a1…`) with the same trainer, recipe, data order and batch; DDP |
| Batch | 256 × 4096 = 1 048 576 tokens per step (4 ranks × micro-batch 2 × gradient accumulation 32) |
| Steps | resumed at 5720 (1.02 `ckpt_5720.pt`, sha256 `6b2a9b11…`), ran 5721 → 7150 |
| Tokens | 7150 × 1 048 576 = 7 497 318 400 in total |
| Schedule | constant lr 6e-4 throughout |
| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
| Final training loss | 2.3955 (step 7150) |
| Resume check (L40S) | batch hash of zero-based iterations 6751 and 7149 (native steps 6752 and 7150) identical to the original run |
| Checkpoints | every 150 steps from 5850 and the final step 7150; all are constant-lr states |
| Trainer | `train_gpt_ref_r24_1.py` (sha256 `881cb055…`): the 1.02 trainer, releasing the checkpoint tensors from GPU memory after resuming; the model definition is unchanged (`train_gpt_ref_r23.py` in this repository) |

## Limitations

- 1 073 912 851 parameters trained on 7 497 318 400 tokens (about 7.0 tokens per parameter).
- Released without a final learning-rate decay (see above).
- Polish-dominant data; English is a minority of the mixture.
- Base model: no instruction tuning, no safety tuning.
- BoolQ / MMLU / ARC scores carry the leak-scan caveat above.

## License

Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions
(`shared/core_sa`, `shared/legal_sa`; see the 1.00 card and the dataset card).