Update README.md
Browse files
README.md
CHANGED
|
@@ -88,33 +88,33 @@ Therefore, our results represent a best-effort attempt at a fair comparison.
|
|
| 88 |
| Pre-training tokens | 12T | 20T | **45T** | N/A | 23.8T |
|
| 89 |
| Training FLOPs (6ND) | 4.7e24 | 6.6e24 | **1.1e25** | N/A | 3.3e24 |
|
| 90 |
| **Knowledge** | | | | | |
|
| 91 |
-
| MMLU |
|
| 92 |
-
| MMLU-Pro |
|
| 93 |
-
| GPQA Diamond |
|
| 94 |
| **Math** | | | | | |
|
| 95 |
-
| AIME 2024 |
|
| 96 |
-
| AIME 2025 |
|
| 97 |
-
| AIME 2026 |
|
| 98 |
-
| HMMT Feb 2026 |
|
| 99 |
| **Coding & Agentic** | | | | | |
|
| 100 |
| Terminal-Bench 2.1 (reported) | **80.52** | 56.4 | 63.8 | 76.4 | *80.1* |
|
| 101 |
| Terminal-Bench 2.1 (Terminus 2) | **80.52** | *51.31* | *55.43* | *79.78* | *67.79* |
|
| 102 |
| SWE-bench Verified | β | 71.9 | 77.6 | *78.67* | **81.6**[^n] |
|
| 103 |
| SWE-bench Pro v1.0 | β | *46.92* | 54.3 | β | **66.1** |
|
| 104 |
| SWE-bench Pro v2.0 Hard | β | *30.72* | *1.96* | *52.29* | 76.5 |
|
| 105 |
-
| DeepSWE 1.1 | β | *0.22* | *1.96* | **60**
|
| 106 |
-
| SciCode |
|
| 107 |
| ΟΒ³-bench banking | β| 22.6 | 23.7 | *14.43* | **36.1** |
|
| 108 |
| AutomationBench (Public) | β | *8.39* | *14.56* | *42.44* | 39.3 |
|
| 109 |
-
| MCP Atlas | β | *62.2* | 76.0
|
| 110 |
| **Safety** | | | | | |
|
| 111 |
-
| FORTRESS Adversarial |
|
| 112 |
-
| FORTRESS Benign |
|
| 113 |
-
| StrongREJECT |
|
| 114 |
| **Instruction Following** | | | | | |
|
| 115 |
-
| IFBench |
|
| 116 |
| **Long Context** | | | | | |
|
| 117 |
-
| AA-LCR |
|
| 118 |
|
| 119 |
</div>
|
| 120 |
|
|
|
|
| 88 |
| Pre-training tokens | 12T | 20T | **45T** | N/A | 23.8T |
|
| 89 |
| Training FLOPs (6ND) | 4.7e24 | 6.6e24 | **1.1e25** | N/A | 3.3e24 |
|
| 90 |
| **Knowledge** | | | | | |
|
| 91 |
+
| MMLU | 89.60 | *92.0* | *91.3* | **92.6** | *92.4* |
|
| 92 |
+
| MMLU-Pro | 84.15 | 86.8 | *87.3* | **87.95** | *76.5* |
|
| 93 |
+
| GPQA Diamond | 85.19 | 87.0 | 87.2 | 89.0 | **90.6** |
|
| 94 |
| **Math** | | | | | |
|
| 95 |
+
| AIME 2024 | 97.08 | *94.17* | *97.5* | **97.92** | *96.67* |
|
| 96 |
+
| AIME 2025 | 98.33 | *94.17* | *91.67* | **100.0** | *99.58* |
|
| 97 |
+
| AIME 2026 | 95.00 | *95.42* | 97.1 | **98.33** | *97.7* |
|
| 98 |
+
| HMMT Feb 2026 | 90.53 | *80.3* | *78.79* | **93.94** | **93.94** |
|
| 99 |
| **Coding & Agentic** | | | | | |
|
| 100 |
| Terminal-Bench 2.1 (reported) | **80.52** | 56.4 | 63.8 | 76.4 | *80.1* |
|
| 101 |
| Terminal-Bench 2.1 (Terminus 2) | **80.52** | *51.31* | *55.43* | *79.78* | *67.79* |
|
| 102 |
| SWE-bench Verified | β | 71.9 | 77.6 | *78.67* | **81.6**[^n] |
|
| 103 |
| SWE-bench Pro v1.0 | β | *46.92* | 54.3 | β | **66.1** |
|
| 104 |
| SWE-bench Pro v2.0 Hard | β | *30.72* | *1.96* | *52.29* | 76.5 |
|
| 105 |
+
| DeepSWE 1.1 | β | *0.22* | *1.96* | **60.0** | 43.3 |
|
| 106 |
+
| SciCode | 39.40 | 44.6 | *39.18* | **58.5** | 49.2 |
|
| 107 |
| ΟΒ³-bench banking | β| 22.6 | 23.7 | *14.43* | **36.1** |
|
| 108 |
| AutomationBench (Public) | β | *8.39* | *14.56* | *42.44* | 39.3 |
|
| 109 |
+
| MCP Atlas | β | *62.2* | 76.0 | **89.2** | 80.8 |
|
| 110 |
| **Safety** | | | | | |
|
| 111 |
+
| FORTRESS Adversarial | 83.83 | *65.80* | 78.0 | *89.07* | *81.33* |
|
| 112 |
+
| FORTRESS Benign | 95.47 | *89.13* | 95.9 | *88.13* | *86.40* |
|
| 113 |
+
| StrongREJECT | 97.47 | *100.0* | 98.6 | *96.59* | *100.0* |
|
| 114 |
| **Instruction Following** | | | | | |
|
| 115 |
+
| IFBench | 79.22 | **81.7** | 79.8 | *78.56* | 79.6 |
|
| 116 |
| **Long Context** | | | | | |
|
| 117 |
+
| AA-LCR | 75.00 | 65.4 | *78.33* | *80.67* | **78.2** |
|
| 118 |
|
| 119 |
</div>
|
| 120 |
|