| --- |
| license: apache-2.0 |
| language: |
| - pl |
| - en |
| library_name: pytorch |
| tags: |
| - slayer |
| - slayer1b |
| - pretrained |
| - polish |
| - reasoning |
| - experimental |
| datasets: |
| - SlayerLab/slayer-1.08b-pretrain |
| --- |
| |
| # Slayer1B 1.08b (experimental) |
|
|
| **Status: trained and evaluated β no measurable reasoning gain over the control, worse than 1.07 on standard English benchmarks; weights are not published yet.** Slayer1B 1.08b is an |
| experimental continuation of [Slayer1B 1.07](https://huggingface.co/SlayerLab/Slayer1B-1.07). The suffix "b" marks an |
| experiment: the model would be released as Slayer1B 1.08 only if the evaluation showed it is better than 1.07. It did |
| not show that (see Evaluation), so the name stays 1.08b. |
|
|
| ## What was trained |
|
|
| Continued pretraining from the final Slayer1B 1.07 checkpoint (step 22327), as an ablation with two arms of equal |
| length and identical settings: |
|
|
| - **1.08b** (arm p00): step-by-step reasoning data (mathematics, code, logic) plus Polish replay; |
| - **1.08b control** (arm p00c): Polish replay only. |
|
|
| | | | |
| |---|---| |
| | Start | Slayer1B 1.07, step 22327 | |
| | Steps | 22328 β 23073 (746 steps per arm) | |
| | Training tokens per arm | 782 237 696 (746 steps Γ 1 048 576) | |
| | Data per arm | 782 469 318 tokens | |
| | Global batch | 256 sequences Γ 4096 tokens (1 048 576 tokens per step) | |
| | Learning rate | constant 6e-4 (no decay; the same as 1.07, so the arms differ only in data) | |
| | Hardware | 8 Γ NVIDIA RTX 5090, four GPUs per arm, both arms in parallel | |
| | Date | 7 November 2026 | |
|
|
| Training data and processing (decontamination, personal-data masking, held-out overlap scan) are described in the |
| data card [SlayerLab/slayer-1.08b-pretrain](https://huggingface.co/datasets/SlayerLab/slayer-1.08b-pretrain). |
|
|
| Training curves: |
| [1.08b](https://track.fabryka.ai/run/751dff2d-d5a2-5f51-b458-0afffb35abe0) Β· |
| [1.08b control](https://track.fabryka.ai/run/fa2a3d80-5bfc-5732-8294-0c78cd442c88). |
| The training loss of the two arms is measured on different data mixtures and is not comparable between them. |
|
|
| ## Model |
|
|
| - Parameters: ~1.07B |
| - Architecture: 20 layers, 2048 dim, 16 heads, RoPE, SwiGLU, QK-Norm, value residual |
| - Context: 4096 tokens |
| - Tokenizer: 32 768 (Slayer v6), the same as 1.00β1.07 |
| - Optimizer: Muon + AdamW |
|
|
| ## Evaluation |
|
|
| Comparison of 1.08b vs 1.08b control vs 1.07 (step 22327), pre-registered before any result was read: the final |
| checkpoint (step 23073) of both arms, paired per-item bootstrap (10 000 resamples, fixed seed), Holm correction within |
| each family. |
|
|
| **Held-out reasoning set.** 450 items generated with reasoning-gym (build 0.1.25, scored with 0.1.19). An overlap scan |
| against the training data found 37 exact matches; they are excluded, which gives 413. Then 50 `path_star` items that |
| cannot be scored across the two library versions are excluded, which gives **363 clean items**. A positive control |
| (gold answers scored 1.0 on 400/400 items, twice) confirmed that the scorer works. |
|
|
| | | 1.07 (22327) | 1.08b | 1.08b control | |
| |---|---:|---:|---:| |
| | pass@1, greedy, 363 clean items | 0.000 | 0.000 | 0.000 | |
| | Polish retention, bits per byte (lower is better) | 0.75459 | 0.76547 | 0.74847 | |
|
|
| - **Reasoning, first set: inconclusive.** pass@1 was 0/363 for each model; with this prompt the metric did not |
| discriminate between the models, and the cause of the zero scores was not measured. This is not evidence that |
| reasoning data did or did not help. In a few outputs read after the result, 1.08b produces text in a step-by-step |
| reasoning style (its correctness was not measured), where 1.07 and the control repeat themselves. This is an |
| unmeasured qualitative observation, not a result. |
| - **Polish retention: measured.** 1.08b is higher (worse) than 1.07 by 0.01088 bits per byte (95% CI [0.01027, |
| 0.01150], Holm p = 0.0002), about 1.4%. The control is lower (better) than 1.07 by 0.00612 bits per byte (95% CI |
| [β0.00688, β0.00547], Holm p = 0.0002), about 0.8%. |
| - The p-values of the degenerate pass@1 contrasts (all zeros) were clamped to 1.0 after a boundary bug fix; no other |
| field changed. |
|
|
| **Standard benchmark panel (reported separately from the verdict).** Same calibrated instrument as the 1.07 results; accuracies |
| in percent. |
|
|
| | | 1.07 (22327) | 1.08b | 1.08b control | |
| |---|---:|---:|---:| |
| | ARC-Easy | 58.67 | 52.10 | 52.06 | |
| | BLiMP | 77.90 | 74.64 | 74.44 | |
| | WikiText-2 byte perplexity (lower is better) | 2.184 | 2.299 | 2.263 | |
| | ARC-Easy PL | 46.93 | 46.17 | 45.92 | |
| | MultiBLiMP PL | 98.84 | 98.23 | 98.59 | |
|
|
| The method note pre-registered guards on this panel (written for the planned run with learning-rate decay; the actual |
| run used a constant rate): at most 1.0 pp drop on ARC-Easy and BLiMP, at most +0.02 byte perplexity, and at most 0.5 pp |
| drop on each Polish benchmark. By those guards the result is a regression in both arms. English: ARC-Easy β6.6 pp, |
| BLiMP β3.3 pp (1.08b) and β3.5 pp (control), byte perplexity +0.115 and +0.080. Polish: β0.6 pp (MultiBLiMP PL) and |
| β0.8 pp (ARC-Easy PL) for 1.08b; β0.3 pp and β1.0 pp for the control. Because the control dropped as much as 1.08b, |
| the drop is not specific to the reasoning data; its cause (for example the constant learning rate, or the absence of |
| general English text in both mixtures) has not been tested. |
|
|
| **Held-out v2 (second, recalibrated reasoning set).** 400 new reasoning-gym items, disjoint from the first set by |
| construction (checked). An overlap scan against the 1.08b training data and the instruction-tuning data excluded 40 |
| items, which gives **360 clean items**; prompt with 3 examples per task. Pre-registered condition before the arms were |
| run: 1.07 must reach at least 5% pass@1 on the 310 non-binary items (360 minus 50 yes/no syllogism items) with at |
| least 4 of 7 non-binary tasks above zero. It reached 7.74% (24/310) with exactly 4 tasks, so the set was used. |
|
|
| | pass@1 on held-out v2 | 1.07 (22327) | 1.08b | 1.08b control | |
| |---|---:|---:|---:| |
| | all 360 items (pre-registered metric) | 13.3% | 9.7% | 15.6% | |
| | 310 non-binary items (validity-condition subset) | 7.74% | 6.45% | 8.39% | |
| | arithmetic (46 items) | 2/46 | 3/46 | 3/46 | |
|
|
| - **No measurable reasoning gain.** On all 360 items (pre-registered metric): 1.08b 9.7% vs control 15.6% (β5.8 pp). |
| Variant on the 310 non-binary items: β1.94 pp, 95% CI [β5.48, +1.61]. The sensitivity of this set is low. |
| - On the 50 yes/no syllogism items (gold: 14 No, 36 Yes), 1.08b answered No 49 times, the control 16 and 1.07 24; in |
| this single run this accounts for most of the gap. None of the models beats the always-Yes baseline on these 50 |
| syllogism items (post-hoc, single run). |
| - With an equal few-shot prompt on held-out arithmetic, 1.08b and the control both scored 3/46. |
| - Instruction-tuned versions of 1.07, 1.08b and the control (same recipe, chat format, no examples) all stayed below |
| the pre-registered 5% validity threshold on this set, so they are not compared. |
|
|
| ## Limitations |
|
|
| - Base model, not instruction-tuned; responses may be inconsistent. |
| - Experimental: constant learning rate, one run per arm, one seed. No measurable reasoning gain over the control on held-out v2 (low sensitivity, see Evaluation). |
| - Worse than 1.07 on the standard English benchmarks (see the panel above); use 1.07 as the general-purpose base. |
|
|
| ## Authorship and license |
|
|
| Authorship: **Arkadiusz SΕota β Fabryka AI SlayerLab**. License: **apache-2.0** (as 1.00β1.07); data licenses are in |
| the data card. |
|
|
