Card: final evaluation of step 2860; training loss vs. windows fixed before the run
Browse files
README.md
CHANGED
|
@@ -25,7 +25,36 @@ further**: training resumed from the last checkpoint of 1.00 before its learning
|
|
| 25 |
|
| 26 |
## Results
|
| 27 |
|
| 28 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
## Usage
|
| 31 |
|
|
|
|
| 25 |
|
| 26 |
## Results
|
| 27 |
|
| 28 |
+
Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot,
|
| 29 |
+
BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final
|
| 30 |
+
checkpoint (step 2860, trainer checkpoint sha256 `b0a28879…`). The same instrument reproduces the published 1.00
|
| 31 |
+
numbers exactly. The released `model.safetensors` (sha256 `607d1a6c…`) was exported twice from the step-2860
|
| 32 |
+
checkpoint, on two machines, with identical bytes.
|
| 33 |
+
|
| 34 |
+
| task | metric | n | 1.00 (step 1430) | **1.01 (step 2860)** |
|
| 35 |
+
|---|---|---:|---:|---:|
|
| 36 |
+
| BLiMP (67 phenomena) | accuracy | 67 000 | 71.51 % | **73.63 %** (49 335 correct) |
|
| 37 |
+
| ARC-Easy (test) | accuracy | 2 376 | 42.89 % | **47.98 %** (1 140 correct) |
|
| 38 |
+
| WikiText-2 (raw, test) | byte perplexity | 1 292 008 bytes | 2.5857 | **2.4230** |
|
| 39 |
+
| eff (leaderboard score from BLiMP, ARC-Easy and WikiText-2 byte perplexity) | leaderboard formula | — | 69.50 | **72.29** |
|
| 40 |
+
| MultiBLiMP, Polish (`jumelet/multiblimp` pol) | accuracy, summed log-probability, leading-space prefix | 3 272 | 97.89 % | **98.29 %** (3 216 correct) |
|
| 41 |
+
| ARC-Easy-PL (`Eurolingua/arcx` EU20, test) | accuracy, LL(q + choice) − LL(q) | 2 376 | 36.91 % | **40.36 %** (959 correct) |
|
| 42 |
+
|
| 43 |
+
Evaluations of every checkpoint from step 1200 to 2860 with the same instrument are on the training-curve page linked
|
| 44 |
+
above.
|
| 45 |
+
|
| 46 |
+
## Training loss vs. windows fixed before the run
|
| 47 |
+
|
| 48 |
+
Before the run, windows for the mean training loss were written down from the training-loss scenarios on the 1.00 card
|
| 49 |
+
(continuation of the 1.00 curve from step 1050):
|
| 50 |
+
|
| 51 |
+
| steps | window fixed before the run | measured mean of logged values |
|
| 52 |
+
|---|---|---:|
|
| 53 |
+
| 1700–1800 (constant lr, pack-00) | 2.692 – 2.882 | 2.6977 |
|
| 54 |
+
| 2770–2860 (end of the decay, pack-01) | 2.367 – 2.660 | 2.5083 |
|
| 55 |
+
|
| 56 |
+
Both measurements fall inside their windows; the first one at the lower edge. This is one run, not a test of the
|
| 57 |
+
scenarios.
|
| 58 |
|
| 59 |
## Usage
|
| 60 |
|