Card: step-3310 loss match stated to four decimal places

bf7117b verified 9 days ago

6.89 kB

---
language:
- pl
- en
license: apache-2.0
library_name: pytorch
pipeline_tag: text-generation
tags:
- polish
- causal-lm
- from-scratch
datasets:
- SlayerLab/slayer-pl-8x3b
---
# Slayer1B 1.02

Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.02 is Slayer1B 1.01 trained
further**: training resumed from the last checkpoint of 1.01 before its learning-rate decay (step 2250) and continued to
5720 steps, i.e. twice the tokens of 1.01 (5 997 854 720 tokens, ≈ 5.6 tokens per parameter).

**From 1.02 on, the learning rate is not decayed at the end of a release.** Pre-training continues on the whole prepared
corpus of about 24 billion tokens, and the final learning-rate decay is applied once, at the end of the whole Slayer1B
line. Each release until then is a constant-learning-rate state; 1.00 and 1.01 ended with a decay. Scores of 1.02 and
later releases are therefore not directly comparable to 1.01: on the same batches at step 2800, the training loss of
1.01 (in its decay) was 2.507 and that of 1.02 (constant lr) 2.581.

- Previous versions: [SlayerLab/Slayer1B-1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01),
  [SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00)
- Training checkpoints (with optimizer state) and run record: [SlayerLab/Slayer1B-1.02-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.02-checkpoints)
- Training curve and evaluations during training: https://track.fabryka.ai/run/21ae1234-82c2-42b7-887b-cbcc69901ae1

## Results

Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot,
BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final
checkpoint of each release.

| | 1.00 | 1.01 | **1.02** |
|---|---:|---:|---:|
| training tokens | 1.50 B | 3.00 B | **6.00 B** |
| end of training | lr decay | lr decay | **constant lr (no decay)** |
| ARC-Easy, accuracy % (2 376) | 42.89 | 47.98 | **48.99** |
| BLiMP, accuracy % (67 000) | 71.51 | 73.63 | **74.71** |
| WikiText-2, byte perplexity ↓ | 2.5857 | 2.4230 | **2.3735** |
| MultiBLiMP Polish, accuracy % (3 272) | 97.89 | 98.29 | **98.38** |
| ARC-Easy-PL, accuracy % (2 376) | 36.91 | 40.36 | **41.84** |

Paired comparison 1.01 (step 2860) → 1.02 (step 5720), same items, paired bootstrap (10 000 resamples, 95 % interval):

| task | difference, pp | 95 % interval | |
|---|---:|---|---|
| BLiMP | +1.08 | [+0.78; +1.37] | positive on this test |
| ARC-Easy-PL | +1.47 | [−0.13; +3.07] | unresolved |
| ARC-Easy | +1.01 | [−0.67; +2.74] | unresolved |
| MultiBLiMP Polish | +0.09 | [−0.40; +0.58] | unresolved |

This comparison combines two changes at once — twice the training tokens and no final learning-rate decay — so no
difference can be attributed to either of them separately.

## Usage

The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class
is `GPT` in `train_gpt_ref_r23.py` (included).

```python
import torch, load_slayer
from tokenizers import Tokenizer
model = load_slayer.load(".")                      # directory with the files of this repository
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
logits, _ = model(ids)                             # [batch, seq, 32768]
```

## Model

| | |
|---|---|
| Parameters | 1 073 912 851 (input and output embeddings tied, counted once) |
| Architecture | decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) |
| Blocks | pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases |
| Context | 4096 tokens |
| Tokenizer | byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 |
## Training data
Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`. One stream of 4096-token
windows, each window at most once; the same order as 1.01 up to its step 2860:
| steps | pack | `train.bin` sha256 | tokens available | batches used |
|---|---|---|---:|---:|
| 1–1801 | pack-00 | `760ef0cc…` | 1 889 519 627 | 1801 |
| 1802–3594 | pack-01 | `0494df58…` | 1 880 520 445 | 1793 |
| 3595–5389 | pack-02 | `294a444d…` | 1 883 249 429 | 1795 |
| 5390–5720 | pack-03 | `7d675734…` | 1 890 850 536 | 331 |
Each pack has its own disjoint web slice; core, legal and English documents of each pack are selected from a
document-hash interval disjoint from those of earlier packs. Mixing weights (cl100k tokens): web 0.63, core 0.19,
legal 0.08, English 0.10. Shares of this tokenizer's tokens in the training split: pack-02 web 52.09 %, core 21.63 %,
legal 8.66 %, English 17.62 %; pack-03 51.68 / 22.27 / 8.59 / 17.46 (pack-00 and pack-01: see the 1.01 card).
Personal data: same rule-based masking as in 1.00 (tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`,
`[DOWOD]`); masking does not guarantee removal of every personal datum. A 13-gram overlap check of pack-02 and pack-03
against the evaluation sets has not been run (pack-01: see the 1.01 card).
## Training
| | |
|---|---|
| Hardware | 8 × NVIDIA RTX 5090 (32 GB), DDP |
| Batch | 256 × 4096 = 1 048 576 tokens per step |
| Steps | resumed at 2250 (1.01 `ckpt_2250.pt`, sha256 `aeddfb02…`), ran 2251 → 5720 (22:17 → 07:23 UTC, ≈ 113 000 tokens/s) |
| Tokens | 5720 × 1 048 576 = 5 997 854 720 in total |
| Schedule | constant lr 6e-4 throughout. The run was started with a final decay planned from step 4578; at step 3300, still at constant lr, it was resumed with the decay removed (training loss at step 3310 before and after equal to four decimal places) |
| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
| Final training loss | 2.4187 (step 5720); mean of the values logged at steps 5630–5720: 2.4379 (1.01 at the end of its decay, steps 2770–2860: 2.5083) |
| Resume check | training loss at steps 2260, 2270, 2280 equal to 1.01 at the same steps (same batches, same lr) to four decimals |
| Checkpoints | every 150 steps from 2400, step 4576 and step 5720; all are constant-lr states |
| Trainer | `train_gpt_ref_r24.py`: the 1.00/1.01 trainer plus a chain of data streams and extra checkpoint steps; the model definition is unchanged (`train_gpt_ref_r23.py` in this repository) |

## Limitations

- 1 073 912 851 parameters trained on 5 997 854 720 tokens (about 5.6 tokens per parameter).
- Released without a final learning-rate decay (see above).
- Polish-dominant data; English is a minority of the mixture.
- Base model: no instruction tuning, no safety tuning.

## License

Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions (see the 1.00
card and the dataset card).

	---
	language:
	- pl
	- en
	license: apache-2.0
	library_name: pytorch
	pipeline_tag: text-generation
	tags:
	- polish
	- causal-lm
	- from-scratch
	datasets:
	- SlayerLab/slayer-pl-8x3b
	---

	# Slayer1B 1.02

	Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.02 is Slayer1B 1.01 trained
	further**: training resumed from the last checkpoint of 1.01 before its learning-rate decay (step 2250) and continued to
	5720 steps, i.e. twice the tokens of 1.01 (5 997 854 720 tokens, ≈ 5.6 tokens per parameter).

	From 1.02 on, the learning rate is not decayed at the end of a release. Pre-training continues on the whole prepared
	corpus of about 24 billion tokens, and the final learning-rate decay is applied once, at the end of the whole Slayer1B
	line. Each release until then is a constant-learning-rate state; 1.00 and 1.01 ended with a decay. Scores of 1.02 and
	later releases are therefore not directly comparable to 1.01: on the same batches at step 2800, the training loss of
	1.01 (in its decay) was 2.507 and that of 1.02 (constant lr) 2.581.

	- Previous versions: [SlayerLab/Slayer1B-1.01](https://huggingface.co/SlayerLab/Slayer1B-1.01),
	[SlayerLab/Slayer1B-1.00](https://huggingface.co/SlayerLab/Slayer1B-1.00)
	- Training checkpoints (with optimizer state) and run record: [SlayerLab/Slayer1B-1.02-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.02-checkpoints)
	- Training curve and evaluations during training: https://track.fabryka.ai/run/21ae1234-82c2-42b7-887b-cbcc69901ae1

	## Results

	Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot,
	BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final
	checkpoint of each release.

	\| \| 1.00 \| 1.01 \| 1.02 \|
	\|---\|---:\|---:\|---:\|
	\| training tokens \| 1.50 B \| 3.00 B \| 6.00 B \|
	\| end of training \| lr decay \| lr decay \| constant lr (no decay) \|
	\| ARC-Easy, accuracy % (2 376) \| 42.89 \| 47.98 \| 48.99 \|
	\| BLiMP, accuracy % (67 000) \| 71.51 \| 73.63 \| 74.71 \|
	\| WikiText-2, byte perplexity ↓ \| 2.5857 \| 2.4230 \| 2.3735 \|
	\| MultiBLiMP Polish, accuracy % (3 272) \| 97.89 \| 98.29 \| 98.38 \|
	\| ARC-Easy-PL, accuracy % (2 376) \| 36.91 \| 40.36 \| 41.84 \|

	Paired comparison 1.01 (step 2860) → 1.02 (step 5720), same items, paired bootstrap (10 000 resamples, 95 % interval):

	\| task \| difference, pp \| 95 % interval \| \|
	\|---\|---:\|---\|---\|
	\| BLiMP \| +1.08 \| [+0.78; +1.37] \| positive on this test \|
	\| ARC-Easy-PL \| +1.47 \| [−0.13; +3.07] \| unresolved \|
	\| ARC-Easy \| +1.01 \| [−0.67; +2.74] \| unresolved \|
	\| MultiBLiMP Polish \| +0.09 \| [−0.40; +0.58] \| unresolved \|

	This comparison combines two changes at once — twice the training tokens and no final learning-rate decay — so no
	difference can be attributed to either of them separately.

	## Usage

	The weights are a plain PyTorch state dict in safetensors; they are not a `transformers` AutoModel. The model class
	is `GPT` in `train_gpt_ref_r23.py` (included).

	```python
	import torch, load_slayer
	from tokenizers import Tokenizer
	model = load_slayer.load(".") # directory with the files of this repository
	tok = Tokenizer.from_file("tokenizer.json")
	ids = torch.tensor([tok.encode("Stolicą Polski jest").ids])
	logits, _ = model(ids) # [batch, seq, 32768]
	```

	## Model

	\| \| \|
	\|---\|---\|
	\| Parameters \| 1 073 912 851 (input and output embeddings tied, counted once) \|
	\| Architecture \| decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) \|
	\| Blocks \| pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases \|
	\| Context \| 4096 tokens \|
	\| Tokenizer \| byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 \|

	## Training data

	Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`. One stream of 4096-token
	windows, each window at most once; the same order as 1.01 up to its step 2860:

	\| steps \| pack \| `train.bin` sha256 \| tokens available \| batches used \|
	\|---\|---\|---\|---:\|---:\|
	\| 1–1801 \| pack-00 \| `760ef0cc…` \| 1 889 519 627 \| 1801 \|
	\| 1802–3594 \| pack-01 \| `0494df58…` \| 1 880 520 445 \| 1793 \|
	\| 3595–5389 \| pack-02 \| `294a444d…` \| 1 883 249 429 \| 1795 \|
	\| 5390–5720 \| pack-03 \| `7d675734…` \| 1 890 850 536 \| 331 \|

	Each pack has its own disjoint web slice; core, legal and English documents of each pack are selected from a
	document-hash interval disjoint from those of earlier packs. Mixing weights (cl100k tokens): web 0.63, core 0.19,
	legal 0.08, English 0.10. Shares of this tokenizer's tokens in the training split: pack-02 web 52.09 %, core 21.63 %,
	legal 8.66 %, English 17.62 %; pack-03 51.68 / 22.27 / 8.59 / 17.46 (pack-00 and pack-01: see the 1.01 card).
	Personal data: same rule-based masking as in 1.00 (tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`,
	`[DOWOD]`); masking does not guarantee removal of every personal datum. A 13-gram overlap check of pack-02 and pack-03
	against the evaluation sets has not been run (pack-01: see the 1.01 card).

	## Training

	\| \| \|
	\|---\|---\|
	\| Hardware \| 8 × NVIDIA RTX 5090 (32 GB), DDP \|
	\| Batch \| 256 × 4096 = 1 048 576 tokens per step \|
	\| Steps \| resumed at 2250 (1.01 `ckpt_2250.pt`, sha256 `aeddfb02…`), ran 2251 → 5720 (22:17 → 07:23 UTC, ≈ 113 000 tokens/s) \|
	\| Tokens \| 5720 × 1 048 576 = 5 997 854 720 in total \|
	\| Schedule \| constant lr 6e-4 throughout. The run was started with a final decay planned from step 4578; at step 3300, still at constant lr, it was resumed with the decay removed (training loss at step 3310 before and after equal to four decimal places) \|
	\| Optimizer \| Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest \|
	\| Final training loss \| 2.4187 (step 5720); mean of the values logged at steps 5630–5720: 2.4379 (1.01 at the end of its decay, steps 2770–2860: 2.5083) \|
	\| Resume check \| training loss at steps 2260, 2270, 2280 equal to 1.01 at the same steps (same batches, same lr) to four decimals \|
	\| Checkpoints \| every 150 steps from 2400, step 4576 and step 5720; all are constant-lr states \|
	\| Trainer \| `train_gpt_ref_r24.py`: the 1.00/1.01 trainer plus a chain of data streams and extra checkpoint steps; the model definition is unchanged (`train_gpt_ref_r23.py` in this repository) \|

	## Limitations

	- 1 073 912 851 parameters trained on 5 997 854 720 tokens (about 5.6 tokens per parameter).
	- Released without a final learning-rate decay (see above).
	- Polish-dominant data; English is a minority of the mixture.
	- Base model: no instruction tuning, no safety tuning.

	## License

	Weights: Apache-2.0. Training data components carry their own licenses, including CC-BY-SA partitions (see the 1.00
	card and the dataset card).