| --- |
| language: |
| - pl |
| - en |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: text-generation |
| tags: |
| - polish |
| - causal-lm |
| - from-scratch |
| datasets: |
| - SlayerLab/slayer-pl-8x3b |
| --- |
| |
| # Slayer1B 1.05 |
|
|
| Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.05 is Slayer1B 1.04 trained |
| further**: training resumed from the final checkpoint of 1.04 (step 10010, constant learning rate 6e-4) and continued |
| to step 12870 (13 495 173 120 tokens in total, ≈ 12.6 tokens per parameter). |
|
|
| Like 1.02–1.04, **1.05 is a constant-learning-rate state**: no learning-rate decay was applied. Pre-training continued |
| from this checkpoint to the end of the corpus pass (release 1.06, step 13848). |
|
|
| - Previous versions: [SlayerLab/Slayer1B-1.04](https://huggingface.co/SlayerLab/Slayer1B-1.04), |
| [SlayerLab/Slayer1B-1.03](https://huggingface.co/SlayerLab/Slayer1B-1.03), |
| [SlayerLab/Slayer1B-1.02](https://huggingface.co/SlayerLab/Slayer1B-1.02), |
| [SlayerLab/Slayer1B-1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01), |
| [SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00) |
| - Training checkpoints (with optimizer state): [SlayerLab/Slayer1B-1.05-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.05-checkpoints) |
| (steps 10050–12870); the same files are also in [SlayerLab/Slayer1B-1.04-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.04-checkpoints) |
| (whole continuation 7200–13848) |
| - Training curve and training-time evaluation: https://track.fabryka.ai/run/f056bb5c-f1da-56d5-9776-cc33063f10a5 |
|
|
| ## Training |
|
|
| - Same model, tokenizer, corpus order and global batch as 1.04: 256 sequences × 4096 tokens = 1 048 576 tokens per |
| step, constant learning rate 6e-4. |
| - Steps 10010 → 12870 (2860 steps, 2 998 927 360 tokens) on 8× NVIDIA L40S. The run moved from 4× to 8× L40S at the |
| 1.04 checkpoint; the first update on the new hardware reproduced the global batch of the 4× run (same 256 × 4096 |
| samples), so 1.05 continues the same training trajectory rather than starting a new run. |
| - Data: [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b), continuing the single |
| pass over the corpus started in 1.00. |
|
|
| ## Training-time evaluation |
|
|
| Measured during training with the run's automatic evaluator (one instrument, NVIDIA A100, zero-shot). These are |
| **not** the lm-evaluation-harness benchmark used in the Results table of 1.00–1.03 and are not directly comparable |
| with it. |
|
|
| | metric | 1.04 (10010) | **1.05 (12870)** | |
| |---|---:|---:| |
| | BLiMP (EN), accuracy % | 74.17 | **75.53** | |
| | ARC-Easy (EN), accuracy % | 50.59 | **52.19** | |
| | WikiText-2, byte perplexity ↓ | 2.3078 | **2.2816** | |
| | WikiText-2, token perplexity ↓ | 31.29 | **29.86** | |
| | MultiBLiMP (PL), accuracy % | 98.87 | **98.47** | |
| | ARC-Easy (PL, eu20), accuracy % | 44.02 | **45.20** | |
|
|
| Single checkpoints differ: e.g. step 13000 scored ARC-Easy (EN) 51.56 and MultiBLiMP (PL) 98.81 (all points in the |
| training curve above); no cause is attributed. |
|
|
| The lm-evaluation-harness benchmark (HellaSwag, PIQA, LAMBADA, ARC-Challenge, BoolQ, MMLU) has not been run on 1.05. |
| The benchmark caveat of the 1.03 card (n-gram overlap of the English shared part with BoolQ, MMLU and ARC test items) |
| applies to this release as well: steps up to 7192 (1.00–1.03 and the first 42 steps of the continuation) used the |
| unfiltered data. From step 7193 (pack-04) on, the 5133 documents with n-gram overlap with BoolQ / MMLU / ARC test |
| items are excluded (list sha256 `84caed78733c25a7d7bbe5f0c9c1ef1d2bb249793ef559f1b73f4b390995b445`). |
|
|
| ## Files |
|
|
| `model.safetensors` (weights), `config.json`, `tokenizer.json`, `load_slayer.py` (loader), `train_gpt_ref_r23.py` and |
| `SHA256SUMS`. The `config.json` records the sha256 of the source checkpoint `ckpt_12870.pt` |
| (`435082ff36becfbb96803622139f38d2afa0ba1a9d2fc284bad96f5a8edc1181`, also published in the checkpoints repository). |
|
|
| `train_gpt_ref_r23.py` is the model definition (`GPT` class) imported by `load_slayer.py`; `trainer_sha256` in |
| `config.json` refers to this file. The continuation from 1.03 was trained with `train_gpt_ref` r24.1 |
| (sha256 `881cb055c38b73a24154d96e2359072e9c526f68875c41e0c1b4ed1c4b9eb62c`, steps 7150–10010) and r24.2 |
| (sha256 `46f83b51fcec18ffc3273d80fb9eab29e182c52d8667800dab27b80f7ff18afc`, steps 10010–12870); the architecture is unchanged. |
|
|
