dzhang50 commited on
Commit
22a320c
Β·
verified Β·
1 Parent(s): 777bd59

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +15 -15
README.md CHANGED
@@ -88,33 +88,33 @@ Therefore, our results represent a best-effort attempt at a fair comparison.
88
  | Pre-training tokens | 12T | 20T | **45T** | N/A | 23.8T |
89
  | Training FLOPs (6ND) | 4.7e24 | 6.6e24 | **1.1e25** | N/A | 3.3e24 |
90
  | **Knowledge** | | | | | |
91
- | MMLU | *89.60* | *92.0* | *91.3* | **92.6** | *92.4* |
92
- | MMLU-Pro | *84.15* | 86.8 | *87.3* | **87.95** | *76.5* |
93
- | GPQA Diamond | *85.19* | 87.0 | 87.2 | 89.0 | **90.6** |
94
  | **Math** | | | | | |
95
- | AIME 2024 | *97.08* | *94.17* | *97.5* | **97.92** | *96.67* |
96
- | AIME 2025 | *98.33* | *94.17* | *91.67* | **100.0** | *99.58* |
97
- | AIME 2026 | *95.00* | *95.42* | 97.1 | **98.33** | *97.7* |
98
- | HMMT Feb 2026 | *90.53* | *80.3* | *78.79* | **93.94** | **93.94** |
99
  | **Coding & Agentic** | | | | | |
100
  | Terminal-Bench 2.1 (reported) | **80.52** | 56.4 | 63.8 | 76.4 | *80.1* |
101
  | Terminal-Bench 2.1 (Terminus 2) | **80.52** | *51.31* | *55.43* | *79.78* | *67.79* |
102
  | SWE-bench Verified | β€” | 71.9 | 77.6 | *78.67* | **81.6**[^n] |
103
  | SWE-bench Pro v1.0 | β€” | *46.92* | 54.3 | β€” | **66.1** |
104
  | SWE-bench Pro v2.0 Hard | β€” | *30.72* | *1.96* | *52.29* | 76.5 |
105
- | DeepSWE 1.1 | β€” | *0.22* | *1.96* | **60**[^j] | 43.3 |
106
- | SciCode | *39.40* | 44.6 | *39.18* | **58.5** | 49.2 |
107
  | τ³-bench banking | β€”| 22.6 | 23.7 | *14.43* | **36.1** |
108
  | AutomationBench (Public) | β€” | *8.39* | *14.56* | *42.44* | 39.3 |
109
- | MCP Atlas | β€” | *62.2* | 76.0[^i] | **89.2** | 80.8 |
110
  | **Safety** | | | | | |
111
- | FORTRESS Adversarial | *83.83* | *65.80* | 78.0 | *89.07* | *81.33* |
112
- | FORTRESS Benign | *95.47* | *89.13* | 95.9 | *88.13* | *86.40* |
113
- | StrongREJECT | *97.47* | *100.0* | 98.6 | *96.59* | *100.0* |
114
  | **Instruction Following** | | | | | |
115
- | IFBench | *79.22* | **81.7** | 79.8 | *78.56* | 79.6 |
116
  | **Long Context** | | | | | |
117
- | AA-LCR | *75.00* | 65.4 | *78.33* | *80.67* | **78.2** |
118
 
119
  </div>
120
 
 
88
  | Pre-training tokens | 12T | 20T | **45T** | N/A | 23.8T |
89
  | Training FLOPs (6ND) | 4.7e24 | 6.6e24 | **1.1e25** | N/A | 3.3e24 |
90
  | **Knowledge** | | | | | |
91
+ | MMLU | 89.60 | *92.0* | *91.3* | **92.6** | *92.4* |
92
+ | MMLU-Pro | 84.15 | 86.8 | *87.3* | **87.95** | *76.5* |
93
+ | GPQA Diamond | 85.19 | 87.0 | 87.2 | 89.0 | **90.6** |
94
  | **Math** | | | | | |
95
+ | AIME 2024 | 97.08 | *94.17* | *97.5* | **97.92** | *96.67* |
96
+ | AIME 2025 | 98.33 | *94.17* | *91.67* | **100.0** | *99.58* |
97
+ | AIME 2026 | 95.00 | *95.42* | 97.1 | **98.33** | *97.7* |
98
+ | HMMT Feb 2026 | 90.53 | *80.3* | *78.79* | **93.94** | **93.94** |
99
  | **Coding & Agentic** | | | | | |
100
  | Terminal-Bench 2.1 (reported) | **80.52** | 56.4 | 63.8 | 76.4 | *80.1* |
101
  | Terminal-Bench 2.1 (Terminus 2) | **80.52** | *51.31* | *55.43* | *79.78* | *67.79* |
102
  | SWE-bench Verified | β€” | 71.9 | 77.6 | *78.67* | **81.6**[^n] |
103
  | SWE-bench Pro v1.0 | β€” | *46.92* | 54.3 | β€” | **66.1** |
104
  | SWE-bench Pro v2.0 Hard | β€” | *30.72* | *1.96* | *52.29* | 76.5 |
105
+ | DeepSWE 1.1 | β€” | *0.22* | *1.96* | **60.0** | 43.3 |
106
+ | SciCode | 39.40 | 44.6 | *39.18* | **58.5** | 49.2 |
107
  | τ³-bench banking | β€”| 22.6 | 23.7 | *14.43* | **36.1** |
108
  | AutomationBench (Public) | β€” | *8.39* | *14.56* | *42.44* | 39.3 |
109
+ | MCP Atlas | β€” | *62.2* | 76.0 | **89.2** | 80.8 |
110
  | **Safety** | | | | | |
111
+ | FORTRESS Adversarial | 83.83 | *65.80* | 78.0 | *89.07* | *81.33* |
112
+ | FORTRESS Benign | 95.47 | *89.13* | 95.9 | *88.13* | *86.40* |
113
+ | StrongREJECT | 97.47 | *100.0* | 98.6 | *96.59* | *100.0* |
114
  | **Instruction Following** | | | | | |
115
+ | IFBench | 79.22 | **81.7** | 79.8 | *78.56* | 79.6 |
116
  | **Long Context** | | | | | |
117
+ | AA-LCR | 75.00 | 65.4 | *78.33* | *80.67* | **78.2** |
118
 
119
  </div>
120