Blame for README.md

---
license: apache-2.0
language:
- pl
- en
library_name: pytorch
tags:
- slayer
- slayer1b
- pretrained
- polish
- reasoning
- experimental
datasets:
- SlayerLab/slayer-1.08b-pretrain
---

# Slayer1B 1.08b (experimental)

**Status: trained and evaluated — no measurable reasoning gain over the control, worse than 1.07 on standard English benchmarks; weights are not published yet.** Slayer1B 1.08b is an
experimental continuation of [Slayer1B 1.07](https://huggingface.co/SlayerLab/Slayer1B-1.07). The suffix "b" marks an
experiment: the model would be released as Slayer1B 1.08 only if the evaluation showed it is better than 1.07. It did
not show that (see Evaluation), so the name stays 1.08b.

## What was trained

Continued pretraining from the final Slayer1B 1.07 checkpoint (step 22327), as an ablation with two arms of equal
length and identical settings:

- **1.08b** (arm p00): step-by-step reasoning data (mathematics, code, logic) plus Polish replay;
- **1.08b control** (arm p00c): Polish replay only.

| | |
|---|---|
| Start | Slayer1B 1.07, step 22327 |
| Steps | 22328 → 23073 (746 steps per arm) |
| Training tokens per arm | 782 237 696 (746 steps × 1 048 576) |
| Data per arm | 782 469 318 tokens |
| Global batch | 256 sequences × 4096 tokens (1 048 576 tokens per step) |
| Learning rate | constant 6e-4 (no decay; the same as 1.07, so the arms differ only in data) |
| Hardware | 8 × NVIDIA RTX 5090, four GPUs per arm, both arms in parallel |
| Date | 7 November 2026 |

Training data and processing (decontamination, personal-data masking, held-out overlap scan) are described in the
data card [SlayerLab/slayer-1.08b-pretrain](https://huggingface.co/datasets/SlayerLab/slayer-1.08b-pretrain).

Training curves:
[1.08b](https://track.fabryka.ai/run/751dff2d-d5a2-5f51-b458-0afffb35abe0) ·
[1.08b control](https://track.fabryka.ai/run/fa2a3d80-5bfc-5732-8294-0c78cd442c88).
The training loss of the two arms is measured on different data mixtures and is not comparable between them.

## Model

- Parameters: ~1.07B
- Architecture: 20 layers, 2048 dim, 16 heads, RoPE, SwiGLU, QK-Norm, value residual
- Context: 4096 tokens
- Tokenizer: 32 768 (Slayer v6), the same as 1.00–1.07
- Optimizer: Muon + AdamW

## Evaluation

Comparison of 1.08b vs 1.08b control vs 1.07 (step 22327), pre-registered before any result was read: the final
checkpoint (step 23073) of both arms, paired per-item bootstrap (10 000 resamples, fixed seed), Holm correction within
each family.

**Held-out reasoning set.** 450 items generated with reasoning-gym (build 0.1.25, scored with 0.1.19). An overlap scan
against the training data found 37 exact matches; they are excluded, which gives 413. Then 50 `path_star` items that
cannot be scored across the two library versions are excluded, which gives **363 clean items**. A positive control
(gold answers scored 1.0 on 400/400 items, twice) confirmed that the scorer works.

| | 1.07 (22327) | 1.08b | 1.08b control |
|---|---:|---:|---:|
| pass@1, greedy, 363 clean items | 0.000 | 0.000 | 0.000 |
| Polish retention, bits per byte (lower is better) | 0.75459 | 0.76547 | 0.74847 |

- **Reasoning, first set: inconclusive.** pass@1 was 0/363 for each model; with this prompt the metric did not
  discriminate between the models, and the cause of the zero scores was not measured. This is not evidence that
  reasoning data did or did not help. In a few outputs read after the result, 1.08b produces text in a step-by-step
  reasoning style (its correctness was not measured), where 1.07 and the control repeat themselves. This is an
  unmeasured qualitative observation, not a result.
- **Polish retention: measured.** 1.08b is higher (worse) than 1.07 by 0.01088 bits per byte (95% CI [0.01027,
  0.01150], Holm p = 0.0002), about 1.4%. The control is lower (better) than 1.07 by 0.00612 bits per byte (95% CI
  [−0.00688, −0.00547], Holm p = 0.0002), about 0.8%.
- The p-values of the degenerate pass@1 contrasts (all zeros) were clamped to 1.0 after a boundary bug fix; no other
  field changed.

**Standard benchmark panel (reported separately from the verdict).** Same calibrated instrument as the 1.07 results; accuracies
in percent.

| | 1.07 (22327) | 1.08b | 1.08b control |
|---|---:|---:|---:|
| ARC-Easy | 58.67 | 52.10 | 52.06 |
| BLiMP | 77.90 | 74.64 | 74.44 |
| WikiText-2 byte perplexity (lower is better) | 2.184 | 2.299 | 2.263 |
| ARC-Easy PL | 46.93 | 46.17 | 45.92 |
| MultiBLiMP PL | 98.84 | 98.23 | 98.59 |

The method note pre-registered guards on this panel (written for the planned run with learning-rate decay; the actual
run used a constant rate): at most 1.0 pp drop on ARC-Easy and BLiMP, at most +0.02 byte perplexity, and at most 0.5 pp
drop on each Polish benchmark. By those guards the result is a regression in both arms. English: ARC-Easy −6.6 pp,
BLiMP −3.3 pp (1.08b) and −3.5 pp (control), byte perplexity +0.115 and +0.080. Polish: −0.6 pp (MultiBLiMP PL) and
−0.8 pp (ARC-Easy PL) for 1.08b; −0.3 pp and −1.0 pp for the control. Because the control dropped as much as 1.08b,
the drop is not specific to the reasoning data; its cause (for example the constant learning rate, or the absence of
general English text in both mixtures) has not been tested.

**Held-out v2 (second, recalibrated reasoning set).** 400 new reasoning-gym items, disjoint from the first set by
construction (checked). An overlap scan against the 1.08b training data and the instruction-tuning data excluded 40
items, which gives **360 clean items**; prompt with 3 examples per task. Pre-registered condition before the arms were
run: 1.07 must reach at least 5% pass@1 on the 310 non-binary items (360 minus 50 yes/no syllogism items) with at
least 4 of 7 non-binary tasks above zero. It reached 7.74% (24/310) with exactly 4 tasks, so the set was used.

| pass@1 on held-out v2 | 1.07 (22327) | 1.08b | 1.08b control |
|---|---:|---:|---:|
| all 360 items (pre-registered metric) | 13.3% | 9.7% | 15.6% |
| 310 non-binary items (validity-condition subset) | 7.74% | 6.45% | 8.39% |
| arithmetic (46 items) | 2/46 | 3/46 | 3/46 |

- **No measurable reasoning gain.** On all 360 items (pre-registered metric): 1.08b 9.7% vs control 15.6% (−5.8 pp).
  Variant on the 310 non-binary items: −1.94 pp, 95% CI [−5.48, +1.61]. The sensitivity of this set is low.
- On the 50 yes/no syllogism items (gold: 14 No, 36 Yes), 1.08b answered No 49 times, the control 16 and 1.07 24; in
  this single run this accounts for most of the gap. None of the models beats the always-Yes baseline on these 50
  syllogism items (post-hoc, single run).
- With an equal few-shot prompt on held-out arithmetic, 1.08b and the control both scored 3/46.
- Instruction-tuned versions of 1.07, 1.08b and the control (same recipe, chat format, no examples) all stayed below
  the pre-registered 5% validity threshold on this set, so they are not compared.

## Limitations

- Base model, not instruction-tuned; responses may be inconsistent.
- Experimental: constant learning rate, one run per arm, one seed. No measurable reasoning gain over the control on held-out v2 (low sensitivity, see Evaluation).
- Worse than 1.07 on the standard English benchmarks (see the panel above); use 1.07 as the general-purpose base.

## Authorship and license

Authorship: **Arkadiusz Słota — Fabryka AI SlayerLab**. License: **apache-2.0** (as 1.00–1.07); data licenses are in
the data card.