Add final step 22327 to the evaluation table
Browse files- README.md +15 -13
- slayer1b_107_training.png +2 -2
README.md
CHANGED
|
@@ -20,9 +20,9 @@ Decoder-only language model, 1 073 912 851 parameters, Polish-dominant. **Slayer
|
|
| 20 |
Slayer1B 1.06** from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B
|
| 21 |
replayed from the 1.06 data), at the same constant learning rate and global batch.
|
| 22 |
|
| 23 |
-
**Status: training complete at step 22327 (6 November 2026).** The training-time evaluation table below
|
| 24 |
-
|
| 25 |
-
protocol, and each result is added here when it is ready. A 1.07 model release (converted weights) has not been
|
| 26 |
published in this repository yet. Native training checkpoints (raw PyTorch training state `ckpt_N.pt`, not a
|
| 27 |
ready-to-use model) are published file by file in
|
| 28 |
[SlayerLab/Slayer1B-1.07-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.07-checkpoints), with sha256 sums.
|
|
@@ -81,18 +81,20 @@ ARC-Easy-PL eu20). 1.07 checkpoints are evaluated on CPU; the 1.06 checkpoint (s
|
|
| 81 |
setup matches the 1.06 card values at card rounding. These are **not** the lm-evaluation-harness benchmark used in the
|
| 82 |
Results table of 1.00–1.03.
|
| 83 |
|
| 84 |
-
| metric | 1.06 (13848) | 13998 | 14000 | 15000 | 15730 | 16000 | 17000 | 18000 | 18590 | **
|
| 85 |
-
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 86 |
-
| BLiMP (EN), accuracy % | 76.12 | 76.29 | 76.33 | 77.94 | 77.32 | 77.13 | 77.61 | 77.88 | 77.46 | **77.
|
| 87 |
-
| ARC-Easy (EN), accuracy % | 52.99 | 54.34 | 54.21 | 54.59 | 55.64 | 56.73 | 56.48 | 56.52 | 56.52 |
|
| 88 |
-
| WikiText-2, byte perplexity ↓ | 2.2849 | 2.2618 | 2.2601 | 2.2321 | 2.2237 | 2.2205 | 2.2138 | 2.2042 | 2.2145 | **2.
|
| 89 |
-
| WikiText-2, token perplexity ↓ | 30.03 | 28.80 | 28.71 | 27.28 | 26.85 | 26.70 | 26.37 | 25.90 | 26.40 |
|
| 90 |
-
| MultiBLiMP (PL), accuracy % | 98.41 | 98.66 | 98.59 | 98.62 | 98.62 | 98.81 | 98.87 | 98.84 | 98.62 | **98.
|
| 91 |
-
| ARC-Easy (PL, eu20), accuracy % | 45.03 | 45.37 | 44.91 | 45.92 | 45.75 | 46.25 | 45.41 | 46.21 | 47.14 | **46.
|
| 92 |
|
| 93 |
ARC-Easy has 2 376 questions per language, so differences below about one percentage point between neighbouring
|
| 94 |
-
checkpoints are within noise. WikiText-2 byte perplexity at 18590 was 0.010 higher than at 18000
|
| 95 |
-
|
|
|
|
|
|
|
| 96 |
on the 1.07 data mixture (44 % English), so it is not directly comparable with the training loss of 1.00–1.06.
|
| 97 |
|
| 98 |
The benchmark caveat of the 1.03 card applies to the 1.00–1.06 part of the training. For 1.07, all nine data files
|
|
|
|
| 20 |
Slayer1B 1.06** from its final checkpoint (step 13848) on 8 891 005 707 training tokens (about 7.0 B new and 1.9 B
|
| 21 |
replayed from the 1.06 data), at the same constant learning rate and global batch.
|
| 22 |
|
| 23 |
+
**Status: training complete at step 22327 (6 November 2026).** The training-time evaluation table below includes the
|
| 24 |
+
final checkpoint (22327); the checkpoints between 19000 and 22327 (20000, 21000, 21450, 22000) are being evaluated
|
| 25 |
+
with the same protocol, and each result is added here when it is ready. A 1.07 model release (converted weights) has not been
|
| 26 |
published in this repository yet. Native training checkpoints (raw PyTorch training state `ckpt_N.pt`, not a
|
| 27 |
ready-to-use model) are published file by file in
|
| 28 |
[SlayerLab/Slayer1B-1.07-checkpoints](https://huggingface.co/SlayerLab/Slayer1B-1.07-checkpoints), with sha256 sums.
|
|
|
|
| 81 |
setup matches the 1.06 card values at card rounding. These are **not** the lm-evaluation-harness benchmark used in the
|
| 82 |
Results table of 1.00–1.03.
|
| 83 |
|
| 84 |
+
| metric | 1.06 (13848) | 13998 | 14000 | 15000 | 15730 | 16000 | 17000 | 18000 | 18590 | 19000 | **22327 (final)** |
|
| 85 |
+
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 86 |
+
| BLiMP (EN), accuracy % | 76.12 | 76.29 | 76.33 | 77.94 | 77.32 | 77.13 | 77.61 | 77.88 | 77.46 | 77.74 | **77.90** |
|
| 87 |
+
| ARC-Easy (EN), accuracy % | 52.99 | 54.34 | 54.21 | 54.59 | 55.64 | 56.73 | 56.48 | 56.52 | 56.52 | 56.65 | **58.67** |
|
| 88 |
+
| WikiText-2, byte perplexity ↓ | 2.2849 | 2.2618 | 2.2601 | 2.2321 | 2.2237 | 2.2205 | 2.2138 | 2.2042 | 2.2145 | 2.2029 | **2.1837** |
|
| 89 |
+
| WikiText-2, token perplexity ↓ | 30.03 | 28.80 | 28.71 | 27.28 | 26.85 | 26.70 | 26.37 | 25.90 | 26.40 | 25.84 | **24.92** |
|
| 90 |
+
| MultiBLiMP (PL), accuracy % | 98.41 | 98.66 | 98.59 | 98.62 | 98.62 | 98.81 | 98.87 | 98.84 | 98.62 | 98.53 | **98.84** |
|
| 91 |
+
| ARC-Easy (PL, eu20), accuracy % | 45.03 | 45.37 | 44.91 | 45.92 | 45.75 | 46.25 | 45.41 | 46.21 | 47.14 | 46.80 | **46.93** |
|
| 92 |
|
| 93 |
ARC-Easy has 2 376 questions per language, so differences below about one percentage point between neighbouring
|
| 94 |
+
checkpoints are within noise. WikiText-2 byte perplexity at 18590 was 0.010 higher than at 18000. The final checkpoint
|
| 95 |
+
(22327) has the lowest WikiText-2 perplexity and the highest English ARC-Easy accuracy of the run (+2.0 points over
|
| 96 |
+
19000, together with small gains on all other metrics); the checkpoints in between are not measured yet, so the chart
|
| 97 |
+
joins 19000 and 22327 directly. Each point is a single measurement of a constant-learning-rate run. The training loss in the figure is measured
|
| 98 |
on the 1.07 data mixture (44 % English), so it is not directly comparable with the training loss of 1.00–1.06.
|
| 99 |
|
| 100 |
The benchmark caveat of the 1.03 card applies to the 1.00–1.06 part of the training. For 1.07, all nine data files
|
slayer1b_107_training.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|