Card 1.08b: evaluation result (inconclusive on reasoning, measured Polish retention)

README.md

@@ -17,9 +17,10 @@ datasets:
 
 # Slayer1B 1.08b (experimental)
 
-**Status: trained, evaluation in progress — weights are not published yet.** Slayer1B 1.08b is an experimental
-continuation of [Slayer1B 1.07](https://huggingface.co/SlayerLab/Slayer1B-1.07). The suffix "b" marks an experiment:
-if the evaluation shows that 1.08b is better than 1.07, the model will be released as Slayer1B 1.08.
+**Status: trained and evaluated — result inconclusive, weights are not published yet.** Slayer1B 1.08b is an
+experimental continuation of [Slayer1B 1.07](https://huggingface.co/SlayerLab/Slayer1B-1.07). The suffix "b" marks an
+experiment: the model would be released as Slayer1B 1.08 only if the evaluation showed it is better than 1.07. It did
+not show that (see Evaluation), so the name stays 1.08b.
 
 ## What was trained
 
@@ -58,14 +59,38 @@ The training loss of the two arms is measured on different data mixtures and is
 
 ## Evaluation
 
-In progress, on the same calibrated instrument as the 1.07 results. The comparison 1.08b vs 1.08b control vs 1.07
-(step 22327) uses the clean subset of 413 items of the held-out reasoning set and Polish retention (bits per byte on a
-fixed selection of Polish texts), with paired per-item confidence intervals. No results are published yet; do not assume them.
+Comparison of 1.08b vs 1.08b control vs 1.07 (step 22327), pre-registered before any result was read: the final
+checkpoint (step 23073) of both arms, paired per-item bootstrap (10 000 resamples, fixed seed), Holm correction within
+each family.
+
+**Held-out reasoning set.** 450 items generated with reasoning-gym (build 0.1.25, scored with 0.1.19). An overlap scan
+against the training data found 37 exact matches; they are excluded, which gives 413. Then 50 `path_star` items that
+cannot be scored across the two library versions are excluded, which gives **363 clean items**. A positive control
+(gold answers scored 1.0 on 400/400 items, twice) confirmed that the scorer works.
+
+| | 1.07 (22327) | 1.08b | 1.08b control |
+|---|---:|---:|---:|
+| pass@1, greedy, 363 clean items | 0.000 | 0.000 | 0.000 |
+| Polish retention, bits per byte (lower is better) | 0.75459 | 0.76547 | 0.74847 |
+
+- **Reasoning: inconclusive.** pass@1 is 0 for all three models because the base models almost never end with the
+  required answer marker. With this prompt, the metric cannot tell the models apart. This is not evidence that
+  reasoning data did or did not help. In a few outputs read after the result, 1.08b produces text in a step-by-step
+  reasoning style (its correctness was not measured), where 1.07 and the control repeat themselves. This is an
+  unmeasured qualitative observation, not a result.
+- **Polish retention: measured.** 1.08b is higher (worse) than 1.07 by 0.01088 bits per byte (95% CI [0.01027,
+  0.01150], Holm p = 0.0002), about 1.4%. The control is lower (better) than 1.07 by 0.00612 bits per byte (95% CI
+  [−0.00688, −0.00547], Holm p = 0.0002), about 0.8%.
+- The p-values of the degenerate pass@1 contrasts (all zeros) were clamped to 1.0 after a boundary bug fix; no other
+  field changed.
+
+A new held-out set with a pre-registered short answer format (and the condition that 1.07 scores above zero before
+the arms are run) and a comparison after instruction tuning of both arms are in preparation.
 
 ## Limitations
 
 - Base model, not instruction-tuned; responses may be inconsistent.
-- Experimental: constant learning rate, one run per arm, one seed.
+- Experimental: constant learning rate, one run per arm, one seed. Reasoning ability is not measured yet (see Evaluation).
 
 ## Authorship and license