Update README.md
Browse files
README.md
CHANGED
|
@@ -65,7 +65,20 @@ Rather than training one model, we trained multiple models, each on a different
|
|
| 65 |
|
| 66 |
# Results
|
| 67 |
|
| 68 |
-
|
| 69 |
-
|
|
| 70 |
-
|
|
| 71 |
-
| [
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
|
| 66 |
# Results
|
| 67 |
|
| 68 |
+
Accuracy is normalized by length and shown as a percentage.
|
| 69 |
+
| Dataset | ARC-Easy ↑ | ARC-Challenge ↑ | HellaSwag ↑ | PIQA ↑ | SWAG ↑ | BLiMP ↑ | Avg ↑ | **II ↑** |
|
| 70 |
+
| :------------------------------------------------------ | ---------: | --------------: | ----------: | -----: | -----: | ------: | -----: | ---------: |
|
| 71 |
+
| [FineWeb](https://huggingface.co/HuggingFaceFW/fineweb) | 27.82% | 22.01% | 26.89% | 52.83% | 30.34% | 65.21% | 37.52% | **39.33%** |
|
| 72 |
+
| DCLM-1.0-Baseline | 29.08% | 22.41% | 27.06% | 52.23% | 31.00% | 66.16% | 37.99% | **39.82%** |
|
| 73 |
+
| DOAB | 31.14% | 20.99% | 28.00% | 52.18% | 29.43% | 62.15% | 37.32% | **39.10%** |
|
| 74 |
+
|
| 75 |
+
The II is calculated based on these weights:
|
| 76 |
+
|
| 77 |
+
| Benchmark | Weight | Note (Why?) |
|
| 78 |
+
| :------------ | -----: | :------------------------------------------------------------------------------------------ |
|
| 79 |
+
| ARC-Easy | 1.00 | — |
|
| 80 |
+
| ARC-Challenge | 0.65 | ARC-Challenge is consistently below chance at small scales, so we assign it a lower weight. |
|
| 81 |
+
| HellaSwag | 1.00 | — |
|
| 82 |
+
| SWAG | 0.95 | SWAG is an easier version of HellaSwag, so we assign it a slightly lower weight. |
|
| 83 |
+
| PIQA | 1.00 | — |
|
| 84 |
+
| BLiMP | 0.65 | BLiMP primarily measures grammatical correctness rather than broader reasoning ability. |
|