Update README.md
Browse files
README.md
CHANGED
|
@@ -77,7 +77,8 @@ All models are tested at full BF16 precision with maximum effort levels.
|
|
| 77 |
|
| 78 |
Each lab typically only provides a limited subset of available benchmarks.
|
| 79 |
Our table below shows official model card scores where available, and we use our own in-house eval flow for missing evals (italized).
|
| 80 |
-
|
|
|
|
| 81 |
A higher number is better for all benchmarks.
|
| 82 |
|
| 83 |
Replicating benchmark scores has been challenging, particularly for modern agentic evals, due to the number of factors which can cause
|
|
@@ -111,7 +112,7 @@ Therefore, our results represent a best-effort attempt at a fair comparison.
|
|
| 111 |
| SWE-bench Pro v1.0 | 61.42 | *46.9* | 54.3 | *58.2* | **66.1** |
|
| 112 |
| SWE-bench Pro v2.0 Hard | **79.1** | *30.7* | *2.0* | *52.3* | 76.5 |
|
| 113 |
| DeepSWE 1.1 | 44.2 | *0.2* | *1.8* | **60.0** | 43.3 |
|
| 114 |
-
| SciCode | 39.4 | 44.6 | *47.0* | **58.5** | 49.2 |
|
| 115 |
| τ³-bench banking | 31.27 | 22.6 | 23.7 | *14.4* | **36.1** |
|
| 116 |
| AutomationBench (Public) | 36.4 | *8.4* | *14.6* | *42.4* | 39.3 |
|
| 117 |
| MCP Atlas | 71.67 | *62.2* | 76.0 | **89.2** | 80.8 |
|
|
|
|
| 77 |
|
| 78 |
Each lab typically only provides a limited subset of available benchmarks.
|
| 79 |
Our table below shows official model card scores where available, and we use our own in-house eval flow for missing evals (italized).
|
| 80 |
+
For SWE-bench Pro v2.0 Hard and DeepSWE 1.1, we use Terminus 2 for Ripple-65B Preview.
|
| 81 |
+
In other cases where the harness is not specified, we use OpenCode.
|
| 82 |
A higher number is better for all benchmarks.
|
| 83 |
|
| 84 |
Replicating benchmark scores has been challenging, particularly for modern agentic evals, due to the number of factors which can cause
|
|
|
|
| 112 |
| SWE-bench Pro v1.0 | 61.42 | *46.9* | 54.3 | *58.2* | **66.1** |
|
| 113 |
| SWE-bench Pro v2.0 Hard | **79.1** | *30.7* | *2.0* | *52.3* | 76.5 |
|
| 114 |
| DeepSWE 1.1 | 44.2 | *0.2* | *1.8* | **60.0** | 43.3 |
|
| 115 |
+
| SciCode (Inspect AI) | 39.4 | 44.6 | *47.0* | **58.5** | 49.2 |
|
| 116 |
| τ³-bench banking | 31.27 | 22.6 | 23.7 | *14.4* | **36.1** |
|
| 117 |
| AutomationBench (Public) | 36.4 | *8.4* | *14.6* | *42.4* | 39.3 |
|
| 118 |
| MCP Atlas | 71.67 | *62.2* | 76.0 | **89.2** | 80.8 |
|