Update README.md

README.md

@@ -66,12 +66,13 @@ Rather than training one model, we trained multiple models, each on a different
 # Results
 
 Accuracy is normalized by length and shown as a percentage.
-| Dataset                                                 | ARC-Easy ↑ | ARC-Challenge ↑ | HellaSwag ↑ | PIQA ↑ | SWAG ↑ | BLiMP ↑ |  Avg ↑ |   **II ↑** |
-| :------------------------------------------------------ | ---------: | --------------: | ----------: | -----: | -----: | ------: | -----: | ---------: |
-| [FineWeb](https://huggingface.co/HuggingFaceFW/fineweb) |     27.82% |          22.01% |      26.89% | 52.83% | 30.34% |  65.21% | 37.52% | **37.49%** |
-| DCLM-1.0-Baseline                                       |     29.08% |          22.41% |      27.06% | 52.23% | 31.00% |  66.16% | 37.99% | **37.92%** |
-| DOAB                                                    |     31.14% |          20.99% |      28.00% | 52.18% | 29.43% |  62.15% | 37.32% | **37.61%** |
-| Project Gutenberg                                       |     25.84% |          23.04% |      24.63% | 50.21% | 27.54% |  64.07% | 35.89% | **35.61%**
+| Dataset           | ARC-Easy | ARC-Challenge | HellaSwag |   PIQA |  BLiMP |  **Avg ↑** |   **II ↑** |
+| :---------------- | -------: | ------------: | --------: | -----: | -----: | ---------: | ---------: |
+| FineWeb           |   27.82% |        22.01% |    26.89% | 52.83% | 65.21% | **38.95%** | **37.51%** |
+| DCLM-1.0-Baseline |   29.08% |        22.41% |    27.06% | 52.23% | 66.16% | **39.39%** | **37.95%** |
+| DOAB              |   31.14% |        20.99% |    28.00% | 52.18% | 62.15% | **38.89%** | **37.59%** |
+| Project Gutenberg |   25.84% |        23.04% |    24.63% | 50.21% | 64.07% | **37.56%** | **36.35%** |
+| EOT-2004-Raw      |   29.08% |        21.50% |    27.54% | 51.31% | 59.92% | **37.87%** | **36.91%** |
 
 The II is calculated based on these weights:
 
@@ -80,6 +81,5 @@ The II is calculated based on these weights:
 | ARC-Easy      |   1.00 | —                                                                                           |
 | ARC-Challenge |   0.5  | ARC-Challenge is consistently below chance at small scales, so we assign it a much lower weight. |
 | HellaSwag     |   1.00 | —                                                                                           |
-| SWAG          |   0.75 | SWAG is an easier version of HellaSwag, so we assign it a lower weight.            |
 | PIQA          |   1.00 | —                                                                                           |
-| BLiMP         |   0.65 | BLiMP primarily measures grammatical correctness rather than broader reasoning ability.     |
+| BLiMP         |   0.60 | BLiMP primarily measures grammatical correctness rather than broader reasoning ability.     |