Maggio33 commited on
Commit
86b936e
·
verified ·
1 Parent(s): 98e99de

Card: final evaluation of step 2860; training loss vs. windows fixed before the run

Browse files
Files changed (1) hide show
  1. README.md +30 -1
README.md CHANGED
@@ -25,7 +25,36 @@ further**: training resumed from the last checkpoint of 1.00 before its learning
25
 
26
  ## Results
27
 
28
- The final evaluation of step 2860 (same protocol as the 1.00 card) is being computed and will be added to this section.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29
 
30
  ## Usage
31
 
 
25
 
26
  ## Results
27
 
28
+ Leaderboard protocol of https://track.fabryka.ai/models (`glint_parity_eval.py`, sha256 `d7a38ca0…`: zero-shot,
29
+ BLiMP raw summed log-probabilities, ARC-Easy raw accuracy, WikiText-2 non-overlapping 256-token chunks), fp32, final
30
+ checkpoint (step 2860, trainer checkpoint sha256 `b0a28879…`). The same instrument reproduces the published 1.00
31
+ numbers exactly. The released `model.safetensors` (sha256 `607d1a6c…`) was exported twice from the step-2860
32
+ checkpoint, on two machines, with identical bytes.
33
+
34
+ | task | metric | n | 1.00 (step 1430) | **1.01 (step 2860)** |
35
+ |---|---|---:|---:|---:|
36
+ | BLiMP (67 phenomena) | accuracy | 67 000 | 71.51 % | **73.63 %** (49 335 correct) |
37
+ | ARC-Easy (test) | accuracy | 2 376 | 42.89 % | **47.98 %** (1 140 correct) |
38
+ | WikiText-2 (raw, test) | byte perplexity | 1 292 008 bytes | 2.5857 | **2.4230** |
39
+ | eff (leaderboard score from BLiMP, ARC-Easy and WikiText-2 byte perplexity) | leaderboard formula | — | 69.50 | **72.29** |
40
+ | MultiBLiMP, Polish (`jumelet/multiblimp` pol) | accuracy, summed log-probability, leading-space prefix | 3 272 | 97.89 % | **98.29 %** (3 216 correct) |
41
+ | ARC-Easy-PL (`Eurolingua/arcx` EU20, test) | accuracy, LL(q + choice) − LL(q) | 2 376 | 36.91 % | **40.36 %** (959 correct) |
42
+
43
+ Evaluations of every checkpoint from step 1200 to 2860 with the same instrument are on the training-curve page linked
44
+ above.
45
+
46
+ ## Training loss vs. windows fixed before the run
47
+
48
+ Before the run, windows for the mean training loss were written down from the training-loss scenarios on the 1.00 card
49
+ (continuation of the 1.00 curve from step 1050):
50
+
51
+ | steps | window fixed before the run | measured mean of logged values |
52
+ |---|---|---:|
53
+ | 1700–1800 (constant lr, pack-00) | 2.692 – 2.882 | 2.6977 |
54
+ | 2770–2860 (end of the decay, pack-01) | 2.367 – 2.660 | 2.5083 |
55
+
56
+ Both measurements fall inside their windows; the first one at the lower edge. This is one run, not a test of the
57
+ scenarios.
58
 
59
  ## Usage
60