UnbindAI commited on
Commit
f23734e
·
verified ·
1 Parent(s): 2a68b70

Add eight branded benchmark comparisons and animated overview

Browse files
.gitattributes CHANGED
@@ -38,3 +38,13 @@ DeepSeek-V4-Flash-Q2_K_S-00002-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
38
  assets/unbind-flash-v1.png filter=lfs diff=lfs merge=lfs -text
39
  assets/hosted-capability.png filter=lfs diff=lfs merge=lfs -text
40
  assets/hosted-refusal.png filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
38
  assets/unbind-flash-v1.png filter=lfs diff=lfs merge=lfs -text
39
  assets/hosted-capability.png filter=lfs diff=lfs merge=lfs -text
40
  assets/hosted-refusal.png filter=lfs diff=lfs merge=lfs -text
41
+ assets/benchmarks/01-deepswe.png filter=lfs diff=lfs merge=lfs -text
42
+ assets/benchmarks/02-terminal.png filter=lfs diff=lfs merge=lfs -text
43
+ assets/benchmarks/03-hle.png filter=lfs diff=lfs merge=lfs -text
44
+ assets/benchmarks/04-program.png filter=lfs diff=lfs merge=lfs -text
45
+ assets/benchmarks/05-nl2repo.png filter=lfs diff=lfs merge=lfs -text
46
+ assets/benchmarks/06-automation.png filter=lfs diff=lfs merge=lfs -text
47
+ assets/benchmarks/07-agents.png filter=lfs diff=lfs merge=lfs -text
48
+ assets/benchmarks/08-apex.png filter=lfs diff=lfs merge=lfs -text
49
+ assets/benchmarks/benchmark-overview.gif filter=lfs diff=lfs merge=lfs -text
50
+ assets/benchmarks/comparison-sheet.png filter=lfs diff=lfs merge=lfs -text
BENCHMARKS.md ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Unbind benchmark comparison gallery
2
+
3
+ Unbind-branded comparisons of DeepSeek-V4.1-Flash-1017 and the other models in its publisher’s table. Scores were reported by DeepSeek, not measured by Unbind. They do not evaluate Unbind’s abliterated hosted checkpoint or the older public GGUF.
4
+
5
+ Each chart compares models within one benchmark on a 0–100 axis. Evaluation setups differ across benchmarks; do not average these scores into an overall rating.
6
+
7
+ [Source table and evaluation notes](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017).
8
+
9
+ ## Animated overview
10
+
11
+ ![Eight benchmark comparisons](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/benchmark-overview.gif)
12
+
13
+ ## DeepSWE v1.1
14
+
15
+ ![DeepSWE v1.1 comparison](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/01-deepswe.png)
16
+
17
+ Mini-SWE-V2 harness. Publisher-reported benchmark scores.
18
+
19
+ ## Terminal-Bench 4.0
20
+
21
+ ![Terminal-Bench 4.0 comparison](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/02-terminal.png)
22
+
23
+ Claude Code harness. Publisher-reported benchmark scores.
24
+
25
+ ## HLE without tools
26
+
27
+ ![HLE without tools comparison](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/03-hle.png)
28
+
29
+ Full HLE figures only. GLM-5.3 omitted: the source gives a text-only subset score.
30
+
31
+ ## ProgramBench
32
+
33
+ ![ProgramBench comparison](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/04-program.png)
34
+
35
+ Publisher-reported benchmark scores. Additional per-row settings are not specified.
36
+
37
+ ## NL2Repo-Bench
38
+
39
+ ![NL2Repo-Bench comparison](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/05-nl2repo.png)
40
+
41
+ Publisher-reported benchmark scores. Additional per-row settings are not specified.
42
+
43
+ ## Automation-Bench v1.0.6
44
+
45
+ ![Automation-Bench v1.0.6 comparison](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/06-automation.png)
46
+
47
+ Publisher-reported benchmark scores. Additional per-row settings are not specified.
48
+
49
+ ## Agents’ Last Exam
50
+
51
+ ![Agents’ Last Exam comparison](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/07-agents.png)
52
+
53
+ Publisher-reported benchmark scores. Additional per-row settings are not specified.
54
+
55
+ ## Apex Bench v1.1
56
+
57
+ ![Apex Bench v1.1 comparison](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/08-apex.png)
58
+
59
+ Publisher-reported benchmark scores. Additional per-row settings are not specified.
60
+
61
+ ## Data and reuse
62
+
63
+ [CSV](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/source-data.csv) · [JSON](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/source-data.json). The accompanying SVG files provide editable vector exports.
64
+
65
+ All plotted values and model names reproduce the source table. GLM-5.3 is omitted from HLE because only its text-only subset score is supplied. These are reported benchmark comparisons, not newly run evaluations.
README.md CHANGED
@@ -44,6 +44,14 @@ The publisher reports **320/320 classified COMPLY**, including **0/320 explicit
44
 
45
  [Read the source evaluation and method](https://huggingface.co/dealignai/DeepSeek-V4.1-Flash-1017-UNCENSORED-FP8#internal-benchmarks-greedy-t0-this-repo).
46
 
 
 
 
 
 
 
 
 
47
  ## Download details
48
 
49
  Download both GGUF shards from **Files and versions**. Runtime memory requirements exceed the file size and depend on your software, hardware and context length. The source notes that this Q2 quantization does not use imatrix calibration.
 
44
 
45
  [Read the source evaluation and method](https://huggingface.co/dealignai/DeepSeek-V4.1-Flash-1017-UNCENSORED-FP8#internal-benchmarks-greedy-t0-this-repo).
46
 
47
+ ## Base-model comparison gallery
48
+
49
+ Eight Unbind-branded comparisons reproduce the original V4.1-Flash-1017 publisher table, including competitor results. These evaluate the named original checkpoints, not Unbind or the abliterated hosted edition.
50
+
51
+ ![Eight base-model benchmark comparisons](https://huggingface.co/UnbindAI/Unbind-Flash-V1/resolve/main/assets/benchmarks/benchmark-overview.gif)
52
+
53
+ [Browse all eight charts, methodology notes and source data](https://huggingface.co/UnbindAI/Unbind-Flash-V1/blob/main/BENCHMARKS.md).
54
+
55
  ## Download details
56
 
57
  Download both GGUF shards from **Files and versions**. Runtime memory requirements exceed the file size and depend on your software, hardware and context length. The source notes that this Q2 quantization does not use imatrix calibration.
assets/benchmarks/01-deepswe.png ADDED

Git LFS Details

  • SHA256: b57766e9cf317ebd8cc3d449100535ccd6dd5ed35c78fa1c0294fcc22f30d8b9
  • Pointer size: 131 Bytes
  • Size of remote file: 108 kB
assets/benchmarks/01-deepswe.svg ADDED
assets/benchmarks/02-terminal.png ADDED

Git LFS Details

  • SHA256: 1a5616505966474f8f54cb646ea8f73ae04d08c7a1b82937270fa6941276249d
  • Pointer size: 131 Bytes
  • Size of remote file: 106 kB
assets/benchmarks/02-terminal.svg ADDED
assets/benchmarks/03-hle.png ADDED

Git LFS Details

  • SHA256: a6be0d1f868148ead4d45c00430a97b9f954155a7ee5eee18f1a37e25cdd6066
  • Pointer size: 131 Bytes
  • Size of remote file: 107 kB
assets/benchmarks/03-hle.svg ADDED
assets/benchmarks/04-program.png ADDED

Git LFS Details

  • SHA256: 6bf64ad9c3d7cbe713f99a8c73bee8be4ba64c0ed2722f922b5f53fc5405a81b
  • Pointer size: 131 Bytes
  • Size of remote file: 109 kB
assets/benchmarks/04-program.svg ADDED
assets/benchmarks/05-nl2repo.png ADDED

Git LFS Details

  • SHA256: 6e01e962b12ae09233831af69b97133390b4e936cebe06386186b20b34a600ea
  • Pointer size: 131 Bytes
  • Size of remote file: 112 kB
assets/benchmarks/05-nl2repo.svg ADDED
assets/benchmarks/06-automation.png ADDED

Git LFS Details

  • SHA256: b3c982eb116146abb2b8a509ed31dde4fe05de094b5564ec307d25fdf20bb9bb
  • Pointer size: 131 Bytes
  • Size of remote file: 110 kB
assets/benchmarks/06-automation.svg ADDED
assets/benchmarks/07-agents.png ADDED

Git LFS Details

  • SHA256: 36ef3252b81b96adb77ddd09e739aa7c4f4b92fe409ea947aa17a4bae1c1019e
  • Pointer size: 131 Bytes
  • Size of remote file: 109 kB
assets/benchmarks/07-agents.svg ADDED
assets/benchmarks/08-apex.png ADDED

Git LFS Details

  • SHA256: 49b384d48a9d36e86df8cee629bd6879dad5be3b5421335f5de2087c1c90386e
  • Pointer size: 131 Bytes
  • Size of remote file: 108 kB
assets/benchmarks/08-apex.svg ADDED
assets/benchmarks/benchmark-overview.gif ADDED

Git LFS Details

  • SHA256: 7cb72e26bc35d907dc9c29d33c65cdcff81300b6494d66e537f613ee27fcb17e
  • Pointer size: 131 Bytes
  • Size of remote file: 402 kB
assets/benchmarks/comparison-sheet.png ADDED

Git LFS Details

  • SHA256: cb5ec247d4f8d0d6371c62c79b85bc36dbe166e6085cb5d3ef9faf82e69a3f99
  • Pointer size: 131 Bytes
  • Size of remote file: 286 kB
assets/benchmarks/source-data.csv ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Benchmark,Model,Reported score,Note,Source
2
+ DeepSWE v1.1,DeepSeek V4.1 Flash 1017,77.0,Mini-SWE-V2 harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
3
+ DeepSWE v1.1,DeepSeek V4.1 Flash,74.2,Mini-SWE-V2 harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
4
+ DeepSWE v1.1,K3,67.5,Mini-SWE-V2 harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
5
+ DeepSWE v1.1,GLM-5.3,66.9,Mini-SWE-V2 harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
6
+ DeepSWE v1.1,GPT-5.6 Sol,73.0,Mini-SWE-V2 harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
7
+ DeepSWE v1.1,Opus 5,74.0,Mini-SWE-V2 harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
8
+ Terminal-Bench 4.0,DeepSeek V4.1 Flash 1017,42.3,Claude Code harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
9
+ Terminal-Bench 4.0,DeepSeek V4.1 Flash,31.2,Claude Code harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
10
+ Terminal-Bench 4.0,K3,12.6,Claude Code harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
11
+ Terminal-Bench 4.0,GLM-5.3,37.9,Claude Code harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
12
+ Terminal-Bench 4.0,GPT-5.6 Sol,39.9,Claude Code harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
13
+ Terminal-Bench 4.0,Opus 5,51.8,Claude Code harness. Publisher-reported benchmark scores.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
14
+ HLE without tools,DeepSeek V4.1 Flash 1017,42.6,Full HLE figures only. GLM-5.3 omitted: the source gives a text-only subset score.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
15
+ HLE without tools,DeepSeek V4.1 Flash,36.8,Full HLE figures only. GLM-5.3 omitted: the source gives a text-only subset score.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
16
+ HLE without tools,K3,43.5,Full HLE figures only. GLM-5.3 omitted: the source gives a text-only subset score.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
17
+ HLE without tools,GPT-5.6 Sol,44.5,Full HLE figures only. GLM-5.3 omitted: the source gives a text-only subset score.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
18
+ HLE without tools,Opus 5,56.3,Full HLE figures only. GLM-5.3 omitted: the source gives a text-only subset score.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
19
+ ProgramBench,DeepSeek V4.1 Flash 1017,29.1,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
20
+ ProgramBench,DeepSeek V4.1 Flash,20.3,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
21
+ ProgramBench,K3,17.5,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
22
+ ProgramBench,GLM-5.3,19.0,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
23
+ ProgramBench,GPT-5.6 Sol,23.0,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
24
+ ProgramBench,Opus 5,37.0,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
25
+ NL2Repo-Bench,DeepSeek V4.1 Flash 1017,68.2,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
26
+ NL2Repo-Bench,DeepSeek V4.1 Flash,65.4,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
27
+ NL2Repo-Bench,K3,58.0,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
28
+ NL2Repo-Bench,GLM-5.3,58.0,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
29
+ NL2Repo-Bench,GPT-5.6 Sol,56.8,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
30
+ NL2Repo-Bench,Opus 5,75.3,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
31
+ Automation-Bench v1.0.6,DeepSeek V4.1 Flash 1017,60.5,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
32
+ Automation-Bench v1.0.6,DeepSeek V4.1 Flash,54.8,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
33
+ Automation-Bench v1.0.6,K3,46.7,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
34
+ Automation-Bench v1.0.6,GLM-5.3,48.8,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
35
+ Automation-Bench v1.0.6,GPT-5.6 Sol,45.8,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
36
+ Automation-Bench v1.0.6,Opus 5,50.3,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
37
+ Agents’ Last Exam,DeepSeek V4.1 Flash 1017,35.4,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
38
+ Agents’ Last Exam,DeepSeek V4.1 Flash,31.8,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
39
+ Agents’ Last Exam,K3,27.6,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
40
+ Agents’ Last Exam,GLM-5.3,28.5,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
41
+ Agents’ Last Exam,GPT-5.6 Sol,26.7,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
42
+ Agents’ Last Exam,Opus 5,28.6,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
43
+ Apex Bench v1.1,DeepSeek V4.1 Flash 1017,72.1,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
44
+ Apex Bench v1.1,DeepSeek V4.1 Flash,64.6,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
45
+ Apex Bench v1.1,K3,47.8,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
46
+ Apex Bench v1.1,GLM-5.3,56.6,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
47
+ Apex Bench v1.1,GPT-5.6 Sol,54.3,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
48
+ Apex Bench v1.1,Opus 5,65.8,Publisher-reported benchmark scores. Additional per-row settings are not specified.,https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md
assets/benchmarks/source-data.json ADDED
@@ -0,0 +1,126 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "source": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017/raw/main/README.md",
3
+ "model_names": [
4
+ "DeepSeek V4.1 Flash 1017",
5
+ "DeepSeek V4.1 Flash",
6
+ "K3",
7
+ "GLM-5.3",
8
+ "GPT-5.6 Sol",
9
+ "Opus 5"
10
+ ],
11
+ "highlight_index": 0,
12
+ "series": [
13
+ {
14
+ "id": "01-deepswe",
15
+ "title": "Software engineering",
16
+ "benchmark": "DeepSWE v1.1",
17
+ "values": [
18
+ 77.0,
19
+ 74.2,
20
+ 67.5,
21
+ 66.9,
22
+ 73.0,
23
+ 74.0
24
+ ],
25
+ "note": "Mini-SWE-V2 harness. Publisher-reported benchmark scores."
26
+ },
27
+ {
28
+ "id": "02-terminal",
29
+ "title": "Terminal tasks",
30
+ "benchmark": "Terminal-Bench 4.0",
31
+ "values": [
32
+ 42.3,
33
+ 31.2,
34
+ 12.6,
35
+ 37.9,
36
+ 39.9,
37
+ 51.8
38
+ ],
39
+ "note": "Claude Code harness. Publisher-reported benchmark scores."
40
+ },
41
+ {
42
+ "id": "03-hle",
43
+ "title": "Reasoning without tools",
44
+ "benchmark": "HLE without tools",
45
+ "values": [
46
+ 42.6,
47
+ 36.8,
48
+ 43.5,
49
+ null,
50
+ 44.5,
51
+ 56.3
52
+ ],
53
+ "note": "Full HLE figures only. GLM-5.3 omitted: the source gives a text-only subset score."
54
+ },
55
+ {
56
+ "id": "04-program",
57
+ "title": "Programming tasks",
58
+ "benchmark": "ProgramBench",
59
+ "values": [
60
+ 29.1,
61
+ 20.3,
62
+ 17.5,
63
+ 19.0,
64
+ 23.0,
65
+ 37.0
66
+ ],
67
+ "note": "Publisher-reported benchmark scores. Additional per-row settings are not specified."
68
+ },
69
+ {
70
+ "id": "05-nl2repo",
71
+ "title": "From instructions to repositories",
72
+ "benchmark": "NL2Repo-Bench",
73
+ "values": [
74
+ 68.2,
75
+ 65.4,
76
+ 58.0,
77
+ 58.0,
78
+ 56.8,
79
+ 75.3
80
+ ],
81
+ "note": "Publisher-reported benchmark scores. Additional per-row settings are not specified."
82
+ },
83
+ {
84
+ "id": "06-automation",
85
+ "title": "Automation tasks",
86
+ "benchmark": "Automation-Bench v1.0.6",
87
+ "values": [
88
+ 60.5,
89
+ 54.8,
90
+ 46.7,
91
+ 48.8,
92
+ 45.8,
93
+ 50.3
94
+ ],
95
+ "note": "Publisher-reported benchmark scores. Additional per-row settings are not specified."
96
+ },
97
+ {
98
+ "id": "07-agents",
99
+ "title": "Agent evaluation",
100
+ "benchmark": "Agents’ Last Exam",
101
+ "values": [
102
+ 35.4,
103
+ 31.8,
104
+ 27.6,
105
+ 28.5,
106
+ 26.7,
107
+ 28.6
108
+ ],
109
+ "note": "Publisher-reported benchmark scores. Additional per-row settings are not specified."
110
+ },
111
+ {
112
+ "id": "08-apex",
113
+ "title": "Apex benchmark",
114
+ "benchmark": "Apex Bench v1.1",
115
+ "values": [
116
+ 72.1,
117
+ 64.6,
118
+ 47.8,
119
+ 56.6,
120
+ 54.3,
121
+ 65.8
122
+ ],
123
+ "note": "Publisher-reported benchmark scores. Additional per-row settings are not specified."
124
+ }
125
+ ]
126
+ }