Model card: architecture, data, recipe; training in progress

README.md

@@ -0,0 +1,85 @@
+---
+language:
+- pl
+- en
+license: apache-2.0
+library_name: pytorch
+pipeline_tag: text-generation
+tags:
+- polish
+- causal-lm
+- from-scratch
+datasets:
+- SlayerLab/slayer-pl-8x3b
+---
+
+# Slayer1B 1.00
+
+Decoder-only language model, 1 073 912 851 parameters, trained from scratch on Polish-dominant text
+(pack-00 of [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b)).
+
+**Status:** training started 2026-10-23 02:35 UTC on 8 × RTX 5090 (1430 steps planned). Weights, training curve
+and evaluation results are added to this card when the run finishes.
+
+## Model
+
+| | |
+|---|---|
+| Parameters | 1 073 912 851 (input and output embeddings tied, counted once) |
+| Architecture | decoder-only transformer, 20 layers, d_model 2048, 16 attention heads (MHA, head dim 128) |
+| Blocks | pre-norm RMSNorm, QK-norm, RoPE (θ = 500 000), value residual, SwiGLU MLP (hidden 5462), no biases |
+| Context | 4096 tokens |
+| Tokenizer | byte-level BPE, 32 768 tokens (`tokenizer.json`, sha256 `0640d3bd…`), EOS id 0 |
+
+## Training data
+
+Source: `SlayerLab/slayer-pl-8x3b`, revision `d50df04a6ad46876fa559a87cca5d3e2a96e7098`, pack-00
+(its own disjoint web slice + the shared core, legal and English components), sampled per document with the pack's
+mixing weights (defined in cl100k tokens): web 0.63, core 0.19, legal 0.08, English 0.10.
+
+| component | share of tokens (this tokenizer) | license |
+|---|---|---|
+| web (HPLT v3 PL, disjoint slice) | 51.8 % | CC0-1.0 |
+| core (register-diverse; includes CC-BY-SA partition) | 22.2 % | mixed, incl. CC-BY-SA |
+| legal / public administration (includes CC-BY-SA partition) | 8.5 % | CC-BY-4.0 / PD / CC-BY-SA |
+| English | 17.6 % | ODC-BY |
+
+English is 10 % of the mixture in cl100k tokens and 17.6 % in this tokenizer's tokens: the tokenizer is trained on
+Polish and splits English into more pieces.
+
+Training stream: 1 889 519 627 tokens (`train.bin`, sha256 `760ef0cc…`). Held-out validation from the same pack,
+disjoint by document id: 1 562 647 PL tokens, 1 643 997 EN tokens.
+
+Personal data: before tokenization, e-mail addresses, phone numbers (Polish and international), PESEL, NIP, REGON and
+Polish ID-card numbers were replaced with tags `[EMAIL]`, `[TELEFON]`, `[PESEL]`, `[NIP]`, `[REGON]`, `[DOWOD]`
+(rule-based; counts in the data manifest: e-mail 19 080, phone 21 606, NIP 1 602, REGON 1 421, ID card 259,
+PESEL 131). The source data also contains upstream placeholders such as `[PII]`. Rule-based masking does not guarantee
+removal of every personal datum.
+
+## Training
+
+| | |
+|---|---|
+| Hardware | 8 × NVIDIA RTX 5090 (32 GB), DDP |
+| Batch | 256 × 4096 = 1 048 576 tokens per step |
+| Steps | 1430 |
+| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
+| Schedule | warmup 150 steps, constant, linear decay over the last 20 % of steps to lr / 10 |
+| Data order | each 4096-token window of the stream used at most once (epoch sampler), data seed 20261023 |
+| Precision | bf16 autocast, torch.compile, activation checkpointing |
+
+## Training run
+
+## Evaluation
+
+## Limitations
+
+- 1 073 912 851 parameters trained on the number of tokens given above; compare with the token counts of other
+  models when reading the scores.
+- Polish-dominant data; English is a minority of the mixture.
+- Base model: no instruction tuning, no safety tuning.
+- Masking of personal data is rule-based (see above).
+
+## License
+
+Weights: Apache-2.0. Training data components carry their own licenses (table above), including CC-BY-SA partitions.