dzhang50 commited on
Commit
3354ae6
·
verified ·
1 Parent(s): f9d6641

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -2
README.md CHANGED
@@ -77,7 +77,8 @@ All models are tested at full BF16 precision with maximum effort levels.
77
 
78
  Each lab typically only provides a limited subset of available benchmarks.
79
  Our table below shows official model card scores where available, and we use our own in-house eval flow for missing evals (italized).
80
- In cases where the harness is not specified, we use OpenCode.
 
81
  A higher number is better for all benchmarks.
82
 
83
  Replicating benchmark scores has been challenging, particularly for modern agentic evals, due to the number of factors which can cause
@@ -111,7 +112,7 @@ Therefore, our results represent a best-effort attempt at a fair comparison.
111
  | SWE-bench Pro v1.0 | 61.42 | *46.9* | 54.3 | *58.2* | **66.1** |
112
  | SWE-bench Pro v2.0 Hard | **79.1** | *30.7* | *2.0* | *52.3* | 76.5 |
113
  | DeepSWE 1.1 | 44.2 | *0.2* | *1.8* | **60.0** | 43.3 |
114
- | SciCode | 39.4 | 44.6 | *47.0* | **58.5** | 49.2 |
115
  | τ³-bench banking | 31.27 | 22.6 | 23.7 | *14.4* | **36.1** |
116
  | AutomationBench (Public) | 36.4 | *8.4* | *14.6* | *42.4* | 39.3 |
117
  | MCP Atlas | 71.67 | *62.2* | 76.0 | **89.2** | 80.8 |
 
77
 
78
  Each lab typically only provides a limited subset of available benchmarks.
79
  Our table below shows official model card scores where available, and we use our own in-house eval flow for missing evals (italized).
80
+ For SWE-bench Pro v2.0 Hard and DeepSWE 1.1, we use Terminus 2 for Ripple-65B Preview.
81
+ In other cases where the harness is not specified, we use OpenCode.
82
  A higher number is better for all benchmarks.
83
 
84
  Replicating benchmark scores has been challenging, particularly for modern agentic evals, due to the number of factors which can cause
 
112
  | SWE-bench Pro v1.0 | 61.42 | *46.9* | 54.3 | *58.2* | **66.1** |
113
  | SWE-bench Pro v2.0 Hard | **79.1** | *30.7* | *2.0* | *52.3* | 76.5 |
114
  | DeepSWE 1.1 | 44.2 | *0.2* | *1.8* | **60.0** | 43.3 |
115
+ | SciCode (Inspect AI) | 39.4 | 44.6 | *47.0* | **58.5** | 49.2 |
116
  | τ³-bench banking | 31.27 | 22.6 | 23.7 | *14.4* | **36.1** |
117
  | AutomationBench (Public) | 36.4 | *8.4* | *14.6* | *42.4* | 39.3 |
118
  | MCP Atlas | 71.67 | *62.2* | 76.0 | **89.2** | 80.8 |