Slayer1B 1.07: model card (training in progress, no weights), training-time evaluation up to step 18000

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+slayer1b_107_training.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,97 @@
+---
+language:
+- pl
+- en
+license: apache-2.0
+library_name: pytorch
+pipeline_tag: text-generation
+tags:
+- polish
+- causal-lm
+- from-scratch
+datasets:
+- SlayerLab/slayer-pl-8x3b
+- SlayerLab/slayer-1.07-pretrain
+---
+
+# Slayer1B 1.07 (training in progress)
+
+Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer1B 1.07 continues pretraining of
+Slayer1B 1.06** from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B
+replayed from the 1.06 data), at the same constant learning rate and global batch.
+
+**Status: training in progress.** The planned end is step 22327. This card reports intermediate training-time
+evaluations up to step 18000 and will be replaced by the final card when training ends. No 1.07 weights are
+published in this repository yet.
+
+- Previous version: [SlayerLab/Slayer1B-1.06](https://huggingface.co/SlayerLab/Slayer1B-1.06) (step 13848,
+  14 520 680 448 tokens, one full pass over [SlayerLab/slayer-pl-8x3b](https://huggingface.co/datasets/SlayerLab/slayer-pl-8x3b))
+- Training data of 1.07: [SlayerLab/slayer-1.07-pretrain](https://huggingface.co/datasets/SlayerLab/slayer-1.07-pretrain)
+- Training curve and training-time evaluation: https://track.fabryka.ai/run/4973c068-8dd5-59d8-b6c4-671ca3beb925
+
+![Slayer1B 1.07 training loss and training-time evaluation](slayer1b_107_training.png)
+
+[Full-size chart](https://huggingface.co/SlayerLab/Slayer1B-1.07/resolve/main/slayer1b_107_training.png)
+
+## Training
+
+- Start: Slayer1B 1.06 checkpoint `ckpt_13848.pt`
+  (sha256 `1fc7396ca1790805128244b2f827e02b9fe88e352534cd887d0c488f121d1e4d`).
+- Same model, tokenizer (v6, 32 768 tokens, sha256 `0640d3bd…`) and global batch as 1.06: 256 sequences × 4096 tokens
+  = 1 048 576 tokens per step, constant learning rate 6e-4, no learning-rate decay, no context extension.
+- Hardware: 4× NVIDIA RTX 4090 (gradient accumulation 32, so the global batch is unchanged from the 8-GPU runs),
+  about 45 000 tokens/s. Trainer `train_gpt_ref` r24.2 (sha256 `46f83b51…`), unchanged since 1.05.
+- Planned length: steps 13848 → 22327 (8 479 steps, one pass over the new data; each window is used once).
+- Training was interrupted on 4 November at step 17130 by an external stop of the process and resumed on 5 November
+  from the saved checkpoint at step 17000. After the resume the loss at step 17010 was identical to the first run
+  (2.3177), so the resumed run reproduces the original one.
+
+## Training data of 1.07
+
+8 891 005 707 tokens in 9 files, after removing documents that overlap evaluation sets (13-gram scan) and after
+masking personal data (e-mail addresses, phone numbers, national ID numbers) with the same procedure as in 1.06.
+
+| source | language, type | documents (built) | tokens (built) | tokens in training | license |
+|---|---|---:|---:|---:|---|
+| HPLT (Polish web, parts not used in 1.06) | PL, general web | 2 448 525 | 3 119 999 086 | 3 110 430 927 | compilation CC0; texts used under the text-and-data-mining exception (Art. 4, Directive 2019/790) |
+| Polish core texts, repeated from 1.06 | PL, curated texts | 3 165 531 | 999 785 312 | 982 781 646 | per source, as in slayer-pl-8x3b |
+| Polish legal texts, repeated from 1.06 | PL, legal | 261 590 | 1 004 151 322 | 911 514 295 | per source, as in slayer-pl-8x3b |
+| FineWeb-Edu (from minimal-en-corpus-5b) | EN, educational web | 840 133 | 978 166 858 | one file, 1 650 618 814 together | ODC-By |
+| FineMath | EN, mathematics | 152 623 | 237 458 786 | (same file) | ODC-By |
+| open-web-math | EN, mathematics from the web | 86 621 | 225 412 736 | (same file) | ODC-By |
+| DCLM-baseline (subset) | EN, general web | 177 946 | 244 996 577 | (same file) | CC BY 4.0 |
+| FineWeb-Edu `sample/10BT` | EN, educational web | 2 035 410 | 2 254 202 363 | 2 235 660 025 | ODC-By |
+
+Documents in the published training files (after leak removal): Polish web 2 445 601, Polish core 3 151 339,
+Polish legal 258 328, English mix 1 249 258, FineWeb-Edu `sample/10BT` 2 029 345 (9 133 871 in total).
+
+Share of training tokens: Polish web 35.0 %, Polish core 11.1 %, Polish legal 10.3 %, English 43.7 %.
+The Polish core and legal parts are a random (hash-ordered) subset of what 1.06 already saw; about 36 % of the 1.06
+core tokens and 68 % of the 1.06 legal tokens are seen a second time, the rest of the data once. English documents
+whose exact normalized text was already in the 1.06 English data were excluded.
+
+## Training-time evaluation
+
+Same automatic evaluator and the same zero-shot protocol as the 1.06 card (BLiMP, ARC-Easy, WikiText-2, MultiBLiMP-PL,
+ARC-Easy-PL eu20). 1.07 checkpoints are evaluated on CPU; the 1.06 checkpoint (step 13848) evaluated on the same CPU
+setup matches the 1.06 card values at card rounding. These are **not** the lm-evaluation-harness benchmark used in the
+Results table of 1.00–1.03.
+
+| metric | 1.06 (13848) | 13998 | 14000 | 15000 | 15730 | 16000 | 17000 | **18000** |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| BLiMP (EN), accuracy % | 76.12 | 76.29 | 76.33 | 77.94 | 77.32 | 77.13 | 77.61 | **77.88** |
+| ARC-Easy (EN), accuracy % | 52.99 | 54.34 | 54.21 | 54.59 | 55.64 | 56.73 | 56.48 | **56.52** |
+| WikiText-2, byte perplexity ↓ | 2.2849 | 2.2618 | 2.2601 | 2.2321 | 2.2237 | 2.2205 | 2.2138 | **2.2042** |
+| WikiText-2, token perplexity ↓ | 30.03 | 28.80 | 28.71 | 27.28 | 26.85 | 26.70 | 26.37 | **25.90** |
+| MultiBLiMP (PL), accuracy % | 98.41 | 98.66 | 98.59 | 98.62 | 98.62 | 98.81 | 98.87 | **98.84** |
+| ARC-Easy (PL, eu20), accuracy % | 45.03 | 45.37 | 44.91 | 45.92 | 45.75 | 46.25 | 45.41 | **46.21** |
+
+ARC-Easy has 2 376 questions per language, so differences below about one percentage point between neighbouring
+checkpoints are within noise. The training loss in the figure is measured on the 1.07 data mixture (44 % English),
+so it is not directly comparable with the training loss of 1.00–1.06.
+
+The benchmark caveat of the 1.03 card applies to the 1.00–1.06 part of the training. For 1.07, all nine data files
+were scanned for 13-gram overlap with the test items of the evaluation suite before training, and documents with any
+hit were removed. Exact matches of short answers (below 13 words) were counted but not removed, and one evaluation set
+(EWoK, distributed as a password-protected archive) was not part of the scan. The Slayer 1.06 core includes the source
+passage corpora of PolQA and PoQuAD; passages with a 13-gram hit were removed and the rest remain in the replay.