Card: bands written before the run; end of second window specified after 7 of 10 values

727b5f5 verified 11 days ago

7.21 kB

---
language:
- pl
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- from-scratch
datasets:
- SlayerLab/slayer-pl-8x3b
---
# Slayer1B 1.01

Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.01 is Slayer1B 1.00 trained
further**: training resumed from the last checkpoint of 1.00 before its learning-rate decay (step 1050) and continued to
2860 steps, i.e. twice the tokens per parameter of 1.00 (2 998 927 360 tokens, ≈ 2.8 tokens per parameter).

- Previous version: [SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00)
- Training checkpoints (with optimizer state): [SlayerLab/Slayer1B-1.01-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.01-checkpoints)
- Training curve and evaluations during training: https://track.fabryka.ai/run/4934429a-d393-42e0-8aac-03eae6b71ff7

## Results

Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot,
BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final
checkpoint (step 2860, trainer checkpoint sha256 `b0a28879…`). The same instrument reproduces the published 1.00
numbers exactly. The released `model.safetensors` (sha256 `607d1a6c…`) was exported twice from the step-2860
checkpoint, on two machines, with identical bytes.

| task | metric | n | 1.00 (step 1430) | **1.01 (step 2860)** |
|---|---|---:|---:|---:|
| BLiMP (67 phenomena) | accuracy | 67 000 | 71.51 % | **73.63 %** (49 335 correct) |
| ARC-Easy (test) | accuracy | 2 376 | 42.89 % | **47.98 %** (1 140 correct) |
| WikiText-2 (raw, test) | byte perplexity | 1 292 008 bytes | 2.5857 | **2.4230** |
| eff (leaderboard score from BLiMP, ARC-Easy and WikiText-2 byte perplexity) | leaderboard formula | — | 69.50 | **72.29** |
| MultiBLiMP, Polish (`jumelet/multiblimp` pol) | accuracy, summed log-probability, leading-space prefix | 3 272 | 97.89 % | **98.29 %** (3 216 correct) |
| ARC-Easy-PL (`Eurolingua/arcx` EU20, test) | accuracy, LL(q + choice) − LL(q) | 2 376 | 36.91 % | **40.36 %** (959 correct) |

### Evaluation history of 1.00 and 1.01

![Evaluation history](analysis/scores-history.png)

Measured values only (no fits, no trend lines): one checkpoint every 150 steps of both runs, same instrument.
Data: [`analysis/scores-history.csv`](analysis/scores-history.csv) (step, tokens, checkpoint sha256, scores, and which
of three GPUs computed the row; each GPU was checked against a known point before use). 1.00 at 1.26 B and 1.42 B tokens
is already in its learning-rate decay; 1.01 at the same token counts is not.

## Training loss vs. bands written before the run

Before the run, bands for the mean training loss were written down from the training-loss scenarios on the 1.00 card
(continuation of the 1.00 curve from step 1050):

| steps | band written before the run | measured mean of logged values |
|---|---|---:|
| 1700–1800 (constant lr, pack-00) | 2.692 – 2.882 | 2.6977 |
| 2770–2860 (end of the decay, pack-01) | 2.367 – 2.660 | 2.5083 |

Both bands were written before the run. The steps of the first window were also fixed before the run; the exact end of
the second window (the last 10 logged values, steps 2770–2860) was specified when 7 of those 10 values were already
logged. Both measurements fall inside their bands, the first one at the lower edge. This is one run, not a test of the
scenarios.

## Usage

The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class
is `GPT` in `train_gpt_ref_r23.py` (included).

```python
import torch, load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".")                      # directory with the files of this repository
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
logits, _ = model(ids)                             # [batch, seq, 32768]
```

## Model

| | |
|---|---|
| Parameters | 1 073 912 851 (input and output embeddings tied, counted once) |
| Architecture | decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) |
| Blocks | pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases |
| Context | 4096 tokens |
| Tokenizer | byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 |
## Training data
Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`.
- Steps 1–1050 (from 1.00) and 1051–1801: the pack-00 stream of 1.00 (`train.bin` sha256 `760ef0cc…`), each 4096-token
  window at most once, the same windows as 1.00 up to its step 1050.
- Steps 1802–2860: a new stream from pack-01 (its own disjoint web slice; core, legal and English documents selected
  from a hash interval disjoint from the one used for pack-00), `train.bin` sha256 `0494df58…`,
  1 880 520 445 tokens available, of which 1 059 × 1 048 576 were consumed.
Mixing weights of the pack (cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10. In this tokenizer's tokens
the training split (`train.bin`) of the pack-01 stream is web 52.16 %, core 22.14 %, legal 8.20 %, English 17.49 %
(pack-00 stream: 51.79 / 22.19 / 8.46 / 17.56).
Personal data: same rule-based masking as in 1.00 (tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`,
`[DOWOD]`); masking does not guarantee removal of every personal datum.
Overlap with evaluation sets (13-gram, pack-01 stream vs. the task files): no evaluation item has 90 % or more of its
13-grams in the stream; MultiBLiMP-pl has 10 of 3 272 items with ≥ 50 % overlap, WikiText-2 has 26 of 62 documents with
any shared 13-gram (none ≥ 50 %).
## Training
| | |
|---|---|
| Hardware | 8 × NVIDIA RTX 5090 (32 GB), DDP |
| Batch | 256 × 4096 = 1 048 576 tokens per step |
| Steps | resumed at 1050, ran 1051 → 2860 (13:56 → 18:54 UTC, ≈ 108 000 tokens/s) |
| Tokens | 2860 × 1 048 576 = 2 998 927 360 in total |
| Schedule | warmup 150, constant lr 6e-4 through step 2289, 1 − sqrt decay over steps 2290–2860 to lr / 10 (6.05e-5 at step 2860) |
| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
| Final training loss | 2.5067 (step 2860); mean of the values logged at steps 2770–2860: 2.5083 (1.00, steps 1340–1430: 2.6990) |
| Resume check | training loss after the resume vs. the 1.00 run at the same steps (same batches, same lr): mean difference about 1e-4 nats (two independent computations: −0.00008 over steps 1060–1140, −0.00013 over 1060–1130), largest single-step difference 0.0007 |
| Checkpoints | every 150 steps + final; last before the decay: step 2250 |
## Limitations
- 1 073 912 851 parameters trained on 2 998 927 360 tokens (about 2.8 tokens per parameter).
- Polish-dominant data; English is a minority of the mixture.
- Base model: no instruction tuning, no safety tuning.
## License
Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions (see the 1.00
card and the dataset card).

	---
	language:
	- pl
	- en
	license: apache-2.0
	library_name: pytorch
	pipeline_tag: text-generation
	tags:
	- polish
	- causal-lm
	- from-scratch
	datasets:
	- SlayerLab/slayer-pl-8x3b
	---

	# Slayer1B 1.01

	Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.01 is Slayer1B 1.00 trained
	further**: training resumed from the last checkpoint of 1.00 before its learning-rate decay (step 1050) and continued to
	2860 steps, i.e. twice the tokens per parameter of 1.00 (2 998 927 360 tokens, ≈ 2.8 tokens per parameter).

	- Previous version: [SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00)
	- Training checkpoints (with optimizer state): [SlayerLab/Slayer1B-1.01-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.01-checkpoints)
	- Training curve and evaluations during training: https://track.fabryka.ai/run/4934429a-d393-42e0-8aac-03eae6b71ff7

	## Results

	Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot,
	BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final
	checkpoint (step 2860, trainer checkpoint sha256 `b0a28879…`). The same instrument reproduces the published 1.00
	numbers exactly. The released `model.safetensors` (sha256 `607d1a6c…`) was exported twice from the step-2860
	checkpoint, on two machines, with identical bytes.

	\| task \| metric \| n \| 1.00 (step 1430) \| 1.01 (step 2860) \|
	\|---\|---\|---:\|---:\|---:\|
	\| BLiMP (67 phenomena) \| accuracy \| 67 000 \| 71.51 % \| 73.63 % (49 335 correct) \|
	\| ARC-Easy (test) \| accuracy \| 2 376 \| 42.89 % \| 47.98 % (1 140 correct) \|
	\| WikiText-2 (raw, test) \| byte perplexity \| 1 292 008 bytes \| 2.5857 \| 2.4230 \|
	\| eff (leaderboard score from BLiMP, ARC-Easy and WikiText-2 byte perplexity) \| leaderboard formula \| — \| 69.50 \| 72.29 \|
	\| MultiBLiMP, Polish (`jumelet/multiblimp` pol) \| accuracy, summed log-probability, leading-space prefix \| 3 272 \| 97.89 % \| 98.29 % (3 216 correct) \|
	\| ARC-Easy-PL (`Eurolingua/arcx` EU20, test) \| accuracy, LL(q + choice) − LL(q) \| 2 376 \| 36.91 % \| 40.36 % (959 correct) \|

	### Evaluation history of 1.00 and 1.01

	![Evaluation history](analysis/scores-history.png)

	Measured values only (no fits, no trend lines): one checkpoint every 150 steps of both runs, same instrument.
	Data: [`analysis/scores-history.csv`](analysis/scores-history.csv) (step, tokens, checkpoint sha256, scores, and which
	of three GPUs computed the row; each GPU was checked against a known point before use). 1.00 at 1.26 B and 1.42 B tokens
	is already in its learning-rate decay; 1.01 at the same token counts is not.

	## Training loss vs. bands written before the run

	Before the run, bands for the mean training loss were written down from the training-loss scenarios on the 1.00 card
	(continuation of the 1.00 curve from step 1050):

	\| steps \| band written before the run \| measured mean of logged values \|
	\|---\|---\|---:\|
	\| 1700–1800 (constant lr, pack-00) \| 2.692 – 2.882 \| 2.6977 \|
	\| 2770–2860 (end of the decay, pack-01) \| 2.367 – 2.660 \| 2.5083 \|

	Both bands were written before the run. The steps of the first window were also fixed before the run; the exact end of
	the second window (the last 10 logged values, steps 2770–2860) was specified when 7 of those 10 values were already
	logged. Both measurements fall inside their bands, the first one at the lower edge. This is one run, not a test of the
	scenarios.

	## Usage

	The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class
	is `GPT` in `train_gpt_ref_r23.py` (included).

	```python
	import torch, load_slayer
	from tokenizers import Tokenizer
	model = load_slayer.load(".") # directory with the files of this repository
	tok = Tokenizer.from_file("tokenizer.json")
	ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
	logits, _ = model(ids) # [batch, seq, 32768]
	```

	## Model

	\| \| \|
	\|---\|---\|
	\| Parameters \| 1 073 912 851 (input and output embeddings tied, counted once) \|
	\| Architecture \| decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) \|
	\| Blocks \| pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases \|
	\| Context \| 4096 tokens \|
	\| Tokenizer \| byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 \|

	## Training data

	Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`.

	- Steps 1–1050 (from 1.00) and 1051–1801: the pack-00 stream of 1.00 (`train.bin` sha256 `760ef0cc…`), each 4096-token
	window at most once, the same windows as 1.00 up to its step 1050.
	- Steps 1802–2860: a new stream from pack-01 (its own disjoint web slice; core, legal and English documents selected
	from a hash interval disjoint from the one used for pack-00), `train.bin` sha256 `0494df58…`,
	1 880 520 445 tokens available, of which 1 059 × 1 048 576 were consumed.

	Mixing weights of the pack (cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10. In this tokenizer's tokens
	the training split (`train.bin`) of the pack-01 stream is web 52.16 %, core 22.14 %, legal 8.20 %, English 17.49 %
	(pack-00 stream: 51.79 / 22.19 / 8.46 / 17.56).
	Personal data: same rule-based masking as in 1.00 (tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`,
	`[DOWOD]`); masking does not guarantee removal of every personal datum.

	Overlap with evaluation sets (13-gram, pack-01 stream vs. the task files): no evaluation item has 90 % or more of its
	13-grams in the stream; MultiBLiMP-pl has 10 of 3 272 items with ≥ 50 % overlap, WikiText-2 has 26 of 62 documents with
	any shared 13-gram (none ≥ 50 %).

	## Training

	\| \| \|
	\|---\|---\|
	\| Hardware \| 8 × NVIDIA RTX 5090 (32 GB), DDP \|
	\| Batch \| 256 × 4096 = 1 048 576 tokens per step \|
	\| Steps \| resumed at 1050, ran 1051 → 2860 (13:56 → 18:54 UTC, ≈ 108 000 tokens/s) \|
	\| Tokens \| 2860 × 1 048 576 = 2 998 927 360 in total \|
	\| Schedule \| warmup 150, constant lr 6e-4 through step 2289, 1 − sqrt decay over steps 2290–2860 to lr / 10 (6.05e-5 at step 2860) \|
	\| Optimizer \| Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest \|
	\| Final training loss \| 2.5067 (step 2860); mean of the values logged at steps 2770–2860: 2.5083 (1.00, steps 1340–1430: 2.6990) \|
	\| Resume check \| training loss after the resume vs. the 1.00 run at the same steps (same batches, same lr): mean difference about 1e-4 nats (two independent computations: −0.00008 over steps 1060–1140, −0.00013 over 1060–1130), largest single-step difference 0.0007 \|
	\| Checkpoints \| every 150 steps + final; last before the decay: step 2250 \|

	## Limitations

	- 1 073 912 851 parameters trained on 2 998 927 360 tokens (about 2.8 tokens per parameter).
	- Polish-dominant data; English is a minority of the mixture.
	- Base model: no instruction tuning, no safety tuning.

	## License

	Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions (see the 1.00
	card and the dataset card).