SlayerLab/Slayer1B-1.08b

Slayer1B 1.08b (experimental)

Status: trained and evaluated — no measurable reasoning gain over the control, worse than 1.07 on standard English benchmarks; weights are not published yet. Slayer1B 1.08b is an experimental continuation of Slayer1B 1.07. The suffix "b" marks an experiment: the model would be released as Slayer1B 1.08 only if the evaluation showed it is better than 1.07. It did not show that (see Evaluation), so the name stays 1.08b.

What was trained

Continued pretraining from the final Slayer1B 1.07 checkpoint (step 22327), as an ablation with two arms of equal length and identical settings:

• 1.08b (arm p00): step-by-step reasoning data (mathematics, code, logic) plus Polish replay;
• 1.08b control (arm p00c): Polish replay only.

Start	Slayer1B 1.07, step 22327
Steps	22328 → 23073 (746 steps per arm)
Training tokens per arm	782 237 696 (746 steps × 1 048 576)
Data per arm	782 469 318 tokens
Global batch	256 sequences × 4096 tokens (1 048 576 tokens per step)
Learning rate	constant 6e-4 (no decay; the same as 1.07, so the arms differ only in data)
Hardware	8 × NVIDIA RTX 5090, four GPUs per arm, both arms in parallel
Date	7 November 2026

Training data and processing (decontamination, personal-data masking, held-out overlap scan) are described in the data card SlayerLab/slayer-1.08b-pretrain.

Training curves: 1.08b · 1.08b control. The training loss of the two arms is measured on different data mixtures and is not comparable between them.

Model

• Parameters: ~1.07B
• Architecture: 20 layers, 2048 dim, 16 heads, RoPE, SwiGLU, QK-Norm, value residual
• Context: 4096 tokens
• Tokenizer: 32 768 (Slayer v6), the same as 1.00–1.07
• Optimizer: Muon + AdamW

Evaluation

Comparison of 1.08b vs 1.08b control vs 1.07 (step 22327), pre-registered before any result was read: the final checkpoint (step 23073) of both arms, paired per-item bootstrap (10 000 resamples, fixed seed), Holm correction within each family.

Held-out reasoning set. 450 items generated with reasoning-gym (build 0.1.25, scored with 0.1.19). An overlap scan against the training data found 37 exact matches; they are excluded, which gives 413. Then 50 path_star items that cannot be scored across the two library versions are excluded, which gives 363 clean items. A positive control (gold answers scored 1.0 on 400/400 items, twice) confirmed that the scorer works.

	1.07 (22327)	1.08b	1.08b control
pass@1, greedy, 363 clean items	0.000	0.000	0.000
Polish retention, bits per byte (lower is better)	0.75459	0.76547	0.74847

• Reasoning, first set: inconclusive. pass@1 was 0/363 for each model; with this prompt the metric did not discriminate between the models, and the cause of the zero scores was not measured. This is not evidence that reasoning data did or did not help. In a few outputs read after the result, 1.08b produces text in a step-by-step reasoning style (its correctness was not measured), where 1.07 and the control repeat themselves. This is an unmeasured qualitative observation, not a result.
• Polish retention: measured. 1.08b is higher (worse) than 1.07 by 0.01088 bits per byte (95% CI [0.01027, 0.01150], Holm p = 0.0002), about 1.4%. The control is lower (better) than 1.07 by 0.00612 bits per byte (95% CI [−0.00688, −0.00547], Holm p = 0.0002), about 0.8%.
• The p-values of the degenerate pass@1 contrasts (all zeros) were clamped to 1.0 after a boundary bug fix; no other field changed.

Standard benchmark panel (reported separately from the verdict). Same calibrated instrument as the 1.07 results; accuracies in percent.

	1.07 (22327)	1.08b	1.08b control
ARC-Easy	58.67	52.10	52.06
BLiMP	77.90	74.64	74.44
WikiText-2 byte perplexity (lower is better)	2.184	2.299	2.263
ARC-Easy PL	46.93	46.17	45.92
MultiBLiMP PL	98.84	98.23	98.59

The method note pre-registered guards on this panel (written for the planned run with learning-rate decay; the actual run used a constant rate): at most 1.0 pp drop on ARC-Easy and BLiMP, at most +0.02 byte perplexity, and at most 0.5 pp drop on each Polish benchmark. By those guards the result is a regression in both arms. English: ARC-Easy −6.6 pp, BLiMP −3.3 pp (1.08b) and −3.5 pp (control), byte perplexity +0.115 and +0.080. Polish: −0.6 pp (MultiBLiMP PL) and −0.8 pp (ARC-Easy PL) for 1.08b; −0.3 pp and −1.0 pp for the control. Because the control dropped as much as 1.08b, the drop is not specific to the reasoning data; its cause (for example the constant learning rate, or the absence of general English text in both mixtures) has not been tested.

Held-out v2 (second, recalibrated reasoning set). 400 new reasoning-gym items, disjoint from the first set by construction (checked). An overlap scan against the 1.08b training data and the instruction-tuning data excluded 40 items, which gives 360 clean items; prompt with 3 examples per task. Pre-registered condition before the arms were run: 1.07 must reach at least 5% pass@1 on the 310 non-binary items (360 minus 50 yes/no syllogism items) with at least 4 of 7 non-binary tasks above zero. It reached 7.74% (24/310) with exactly 4 tasks, so the set was used.

pass@1 on held-out v2	1.07 (22327)	1.08b	1.08b control
all 360 items (pre-registered metric)	13.3%	9.7%	15.6%
310 non-binary items (validity-condition subset)	7.74%	6.45%	8.39%
arithmetic (46 items)	2/46	3/46	3/46

• No measurable reasoning gain. On all 360 items (pre-registered metric): 1.08b 9.7% vs control 15.6% (−5.8 pp). Variant on the 310 non-binary items: −1.94 pp, 95% CI [−5.48, +1.61]. The sensitivity of this set is low.
• On the 50 yes/no syllogism items (gold: 14 No, 36 Yes), 1.08b answered No 49 times, the control 16 and 1.07 24; in this single run this accounts for most of the gap. None of the models beats the always-Yes baseline on these 50 syllogism items (post-hoc, single run).
• With an equal few-shot prompt on held-out arithmetic, 1.08b and the control both scored 3/46.
• Instruction-tuned versions of 1.07, 1.08b and the control (same recipe, chat format, no examples) all stayed below the pre-registered 5% validity threshold on this set, so they are not compared.

Limitations

• Base model, not instruction-tuned; responses may be inconsistent.
• Experimental: constant learning rate, one run per arm, one seed. No measurable reasoning gain over the control on held-out v2 (low sensitivity, see Evaluation).
• Worse than 1.07 on the standard English benchmarks (see the panel above); use 1.07 as the general-purpose base.

Authorship and license

Authorship: Arkadiusz Słota — Fabryka AI SlayerLab. License: apache-2.0 (as 1.00–1.07); data licenses are in the data card.