File size: 3,571 Bytes
91f644f 09aa8ab 91f644f db8f765 91f644f db8f765 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | ---
language:
- pl
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- from-scratch
datasets:
- SlayerLab/slayer-pl-8x3b
---
# Slayer1B 1.04
Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.04 is Slayer1B 1.03 trained
further**: training resumed from the final checkpoint of 1.03 (step 7150, constant learning rate 6e-4) and continued to
step 10010 (10 496 245 760 tokens in total, ≈ 9.8 tokens per parameter).
Like 1.02 and 1.03, **1.04 is a constant-learning-rate state**: no learning-rate decay was applied. Pre-training of the
Slayer1B line continued from this checkpoint (releases 1.05 at step 12870 and 1.06 at step 13848).
- Previous versions: [SlayerLab/Slayer1B-1.03](https://huggingface.co/SlayerLab/Slayer1B-1.03),
[SlayerLab/Slayer1B-1.02](https://huggingface.co/SlayerLab/Slayer1B-1.02),
[SlayerLab/Slayer1B-1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01),
[SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00)
- Training checkpoints (with optimizer state): [SlayerLab/Slayer1B-1.04-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.04-checkpoints)
- Training curve and training-time evaluation (steps 7160–13848): https://track.fabryka.ai/run/f056bb5c-f1da-56d5-9776-cc33063f10a5
## Training
- Same model, tokenizer, corpus order and global batch as 1.03: 256 sequences × 4096 tokens = 1 048 576 tokens per
step, constant learning rate 6e-4.
- Steps 7150 → 10010 (2860 steps, 2 998 927 360 tokens) on 4× NVIDIA L40S.
- Data: [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b), continuing the single
pass over the corpus started in 1.00.
## Training-time evaluation
Measured during training on the released checkpoint (step 10010) with the run's automatic evaluator (one instrument,
NVIDIA A100, zero-shot). These are **not** the lm-evaluation-harness benchmark used in the Results table of 1.00–1.03
and are not directly comparable with it.
| metric | 1.04 (10010) |
|---|---:|
| BLiMP (EN), accuracy % | 74.17 |
| ARC-Easy (EN), accuracy % | 50.59 |
| WikiText-2, byte perplexity ↓ | 2.3078 |
| WikiText-2, token perplexity ↓ | 31.29 |
| MultiBLiMP (PL), accuracy % | 98.87 |
| ARC-Easy (PL, eu20), accuracy % | 44.02 |
The lm-evaluation-harness benchmark (HellaSwag, PIQA, LAMBADA, ARC-Challenge, BoolQ, MMLU) has not been run on 1.04.
The benchmark caveat of the 1.03 card (n-gram overlap of the English shared part with BoolQ, MMLU and ARC test items)
applies to this release as well: steps up to 7192 (1.00–1.03 and the first 42 steps of the continuation) used the
unfiltered data. From step 7193 (pack-04) on, the 5133 documents with n-gram overlap with BoolQ / MMLU / ARC test
items are excluded (list sha256 `84caed78733c25a7d7bbe5f0c9c1ef1d2bb249793ef559f1b73f4b390995b445`).
## Files
`model.safetensors` (weights), `config.json`, `tokenizer.json`, `load_slayer.py` (loader), `train_gpt_ref_r23.py` and
`SHA256SUMS`. The `config.json` records the sha256 of the source checkpoint `ckpt_10010.pt`
(`04438959d7b317d1a8d972b264e9610a06fd8addde7349c6e9f458ee56dc1e51`, also published in the checkpoints repository).
`train_gpt_ref_r23.py` is the model definition (`GPT` class) imported by `load_slayer.py`; `trainer_sha256` in
`config.json` refers to this file. The continuation from 1.03 (steps 7150–10010) was trained with `train_gpt_ref` r24.1
(sha256 `881cb055c38b73a24154d96e2359072e9c526f68875c41e0c1b4ed1c4b9eb62c`); the architecture is unchanged.
|