helloboy91 commited on
Commit
ee1d10e
·
verified ·
1 Parent(s): c55cf27

Add staged evaluation comparison

Browse files
Files changed (1) hide show
  1. README.md +12 -10
README.md CHANGED
@@ -28,16 +28,18 @@ The source revisions, filters, tokenizer hash, model configuration, and data man
28
 
29
  ## Evaluation
30
 
31
- Full zero-shot evaluation used `lm-evaluation-harness` 0.4.13, batch size 1, no bootstrap resampling, and the native checkpoint. `standard-eval.json` contains the complete protocol and raw result.
32
-
33
- | Task | Raw accuracy |
34
- | --- | ---: |
35
- | ARC-Easy | 49.92% |
36
- | ARC-Challenge | 21.42% |
37
- | HellaSwag | 29.87% |
38
- | PIQA | 62.95% |
39
- | SciQ | 71.90% |
40
- | WinoGrande | 50.67% |
 
 
41
 
42
  These scores show clear progress over the project’s earlier checkpoints, but they do not make Alloy competitive with mature, heavily trained public models of a similar parameter count. Benchmark contamination and harness-version differences remain possible; do not compare scores across protocols as exact leaderboard rankings.
43
 
 
28
 
29
  ## Evaluation
30
 
31
+ Full zero-shot evaluation used `lm-evaluation-harness` 0.4.13, batch size 1, no bootstrap resampling, and the native checkpoint. `standard-eval.json` contains the complete protocol and raw Alloy result. The stages below used the same zero-shot task suite and raw-accuracy metric.
32
+
33
+ | Task | Bronze | Silver | Gold | Alloy |
34
+ | --- | ---: | ---: | ---: | ---: |
35
+ | ARC-Easy | 42.93% | 47.98% | 49.75% | **49.92%** |
36
+ | ARC-Challenge | 18.77% | 21.25% | 20.99% | **21.42%** |
37
+ | HellaSwag | 27.39% | 27.95% | 28.63% | **29.87%** |
38
+ | PIQA | 59.09% | 59.96% | 60.66% | **62.95%** |
39
+ | SciQ | 64.70% | 68.80% | 69.50% | **71.90%** |
40
+ | WinoGrande | 50.36% | 49.41% | 49.64% | **50.67%** |
41
+
42
+ Alloy improves on Gold on every listed task. Bronze, Silver, and Gold correspond to the successive 1B-token stages; Alloy adds the 2B-token Lead continuation for roughly 5B total recorded training tokens.
43
 
44
  These scores show clear progress over the project’s earlier checkpoints, but they do not make Alloy competitive with mature, heavily trained public models of a similar parameter count. Benchmark contamination and harness-version differences remain possible; do not compare scores across protocols as exact leaderboard rankings.
45