Update README.md
Browse files
README.md
CHANGED
|
@@ -103,13 +103,13 @@ Therefore, our results represent a best-effort attempt at a fair comparison.
|
|
| 103 |
| Pre-training tokens | 12T | 20T | **45T** | N/A | 23.8T |
|
| 104 |
| **Knowledge** | | | | | |
|
| 105 |
| MMLU | 89.6 | *92.0* | *91.3* | ***92.6*** | *92.4* |
|
| 106 |
-
| MMLU-Pro | 84.2 | 86.8 | *87.3* | ***
|
| 107 |
| GPQA Diamond | 85.2 | 87.0 | 87.2 | 89.0 | **90.6** |
|
| 108 |
| **Math** | | | | | |
|
| 109 |
| AIME 2024 | 97.1 | *94.2* | *97.5* | ***97.9*** | *96.7* |
|
| 110 |
| AIME 2025 | 98.3 | *94.2* | *91.7* | ***100.0*** | *99.6* |
|
| 111 |
| AIME 2026 | 95.0 | *95.4* | 97.1 | ***98.3*** | 97.7 |
|
| 112 |
-
| HMMT Feb 2026 | 90.5 | *80.3* |
|
| 113 |
| **Coding & Agentic** | | | | | |
|
| 114 |
| Terminal-Bench 2.1 (self-reported) | **80.5** | 56.4 | 63.8 | 76.4 | 80.1 |
|
| 115 |
| Terminal-Bench 2.1 (Terminus 2) | **80.5** | *55.1* | *56.2* | *78.7* | *70.8* |
|
|
@@ -117,13 +117,13 @@ Therefore, our results represent a best-effort attempt at a fair comparison.
|
|
| 117 |
| SWE-bench Pro v1.0 | 61.42 | *46.9* | 54.3 | *58.2* | **66.1** |
|
| 118 |
| SWE-bench Pro v2.0 Hard | **79.1** | *30.7* | *2.0* | *52.3* | 76.5 |
|
| 119 |
| DeepSWE 1.1 | 44.2 | *0.2* | *1.8* | **60.0** | 43.3 |
|
| 120 |
-
| SciCode (Inspect AI) | 39.4 | 44.6 |
|
| 121 |
| τ³-bench banking | 31.27 | 22.6 | 23.7 | *14.4* | **36.1** |
|
| 122 |
| AutomationBench (Public) | 36.4 | *8.4* | *14.6* | *42.4* | 39.3 |
|
| 123 |
| MCP Atlas | 71.67 | *62.2* | 76.0 | **89.2** | 80.8 |
|
| 124 |
| **Safety** | | | | | |
|
| 125 |
| FORTRESS Adversarial | 83.8 | *79.3* | 78.0 | **94.2** | *92.1* |
|
| 126 |
-
| FORTRESS Benign | 95.5 | *89.1* | **95.9** | *
|
| 127 |
| StrongREJECT | 97.5 | *98.2* | **98.6** | *94.7* | *95.8* |
|
| 128 |
| **Instruction Following** | | | | | |
|
| 129 |
| IFBench | 79.2 | **81.7** | 79.8 | *78.6* | 79.6 |
|
|
|
|
| 103 |
| Pre-training tokens | 12T | 20T | **45T** | N/A | 23.8T |
|
| 104 |
| **Knowledge** | | | | | |
|
| 105 |
| MMLU | 89.6 | *92.0* | *91.3* | ***92.6*** | *92.4* |
|
| 106 |
+
| MMLU-Pro | 84.2 | 86.8 | *87.3* | ***87.9*** | *76.5* |
|
| 107 |
| GPQA Diamond | 85.2 | 87.0 | 87.2 | 89.0 | **90.6** |
|
| 108 |
| **Math** | | | | | |
|
| 109 |
| AIME 2024 | 97.1 | *94.2* | *97.5* | ***97.9*** | *96.7* |
|
| 110 |
| AIME 2025 | 98.3 | *94.2* | *91.7* | ***100.0*** | *99.6* |
|
| 111 |
| AIME 2026 | 95.0 | *95.4* | 97.1 | ***98.3*** | 97.7 |
|
| 112 |
+
| HMMT Feb 2026 | 90.5 | *80.3* | 86.3 | ***93.9*** | ***93.9*** |
|
| 113 |
| **Coding & Agentic** | | | | | |
|
| 114 |
| Terminal-Bench 2.1 (self-reported) | **80.5** | 56.4 | 63.8 | 76.4 | 80.1 |
|
| 115 |
| Terminal-Bench 2.1 (Terminus 2) | **80.5** | *55.1* | *56.2* | *78.7* | *70.8* |
|
|
|
|
| 117 |
| SWE-bench Pro v1.0 | 61.42 | *46.9* | 54.3 | *58.2* | **66.1** |
|
| 118 |
| SWE-bench Pro v2.0 Hard | **79.1** | *30.7* | *2.0* | *52.3* | 76.5 |
|
| 119 |
| DeepSWE 1.1 | 44.2 | *0.2* | *1.8* | **60.0** | 43.3 |
|
| 120 |
+
| SciCode (Inspect AI) | 39.4 | 44.6 | 46.1 | **58.5** | 49.2 |
|
| 121 |
| τ³-bench banking | 31.27 | 22.6 | 23.7 | *14.4* | **36.1** |
|
| 122 |
| AutomationBench (Public) | 36.4 | *8.4* | *14.6* | *42.4* | 39.3 |
|
| 123 |
| MCP Atlas | 71.67 | *62.2* | 76.0 | **89.2** | 80.8 |
|
| 124 |
| **Safety** | | | | | |
|
| 125 |
| FORTRESS Adversarial | 83.8 | *79.3* | 78.0 | **94.2** | *92.1* |
|
| 126 |
+
| FORTRESS Benign | 95.5 | *89.1* | **95.9** | *82.6* | *86.4* |
|
| 127 |
| StrongREJECT | 97.5 | *98.2* | **98.6** | *94.7* | *95.8* |
|
| 128 |
| **Instruction Following** | | | | | |
|
| 129 |
| IFBench | 79.2 | **81.7** | 79.8 | *78.6* | 79.6 |
|