Add staged evaluation comparison
Browse files
README.md
CHANGED
|
@@ -28,16 +28,18 @@ The source revisions, filters, tokenizer hash, model configuration, and data man
|
|
| 28 |
|
| 29 |
## Evaluation
|
| 30 |
|
| 31 |
-
Full zero-shot evaluation used `lm-evaluation-harness` 0.4.13, batch size 1, no bootstrap resampling, and the native checkpoint. `standard-eval.json` contains the complete protocol and raw result.
|
| 32 |
-
|
| 33 |
-
| Task |
|
| 34 |
-
| --- | ---: |
|
| 35 |
-
| ARC-Easy | 49.92% |
|
| 36 |
-
| ARC-Challenge | 21.42% |
|
| 37 |
-
| HellaSwag | 29.87% |
|
| 38 |
-
| PIQA | 62.95% |
|
| 39 |
-
| SciQ | 71.90% |
|
| 40 |
-
| WinoGrande | 50.67% |
|
|
|
|
|
|
|
| 41 |
|
| 42 |
These scores show clear progress over the project’s earlier checkpoints, but they do not make Alloy competitive with mature, heavily trained public models of a similar parameter count. Benchmark contamination and harness-version differences remain possible; do not compare scores across protocols as exact leaderboard rankings.
|
| 43 |
|
|
|
|
| 28 |
|
| 29 |
## Evaluation
|
| 30 |
|
| 31 |
+
Full zero-shot evaluation used `lm-evaluation-harness` 0.4.13, batch size 1, no bootstrap resampling, and the native checkpoint. `standard-eval.json` contains the complete protocol and raw Alloy result. The stages below used the same zero-shot task suite and raw-accuracy metric.
|
| 32 |
+
|
| 33 |
+
| Task | Bronze | Silver | Gold | Alloy |
|
| 34 |
+
| --- | ---: | ---: | ---: | ---: |
|
| 35 |
+
| ARC-Easy | 42.93% | 47.98% | 49.75% | **49.92%** |
|
| 36 |
+
| ARC-Challenge | 18.77% | 21.25% | 20.99% | **21.42%** |
|
| 37 |
+
| HellaSwag | 27.39% | 27.95% | 28.63% | **29.87%** |
|
| 38 |
+
| PIQA | 59.09% | 59.96% | 60.66% | **62.95%** |
|
| 39 |
+
| SciQ | 64.70% | 68.80% | 69.50% | **71.90%** |
|
| 40 |
+
| WinoGrande | 50.36% | 49.41% | 49.64% | **50.67%** |
|
| 41 |
+
|
| 42 |
+
Alloy improves on Gold on every listed task. Bronze, Silver, and Gold correspond to the successive 1B-token stages; Alloy adds the 2B-token Lead continuation for roughly 5B total recorded training tokens.
|
| 43 |
|
| 44 |
These scores show clear progress over the project’s earlier checkpoints, but they do not make Alloy competitive with mature, heavily trained public models of a similar parameter count. Benchmark contamination and harness-version differences remain possible; do not compare scores across protocols as exact leaderboard rankings.
|
| 45 |
|