File size: 27,528 Bytes
afb4866
34e99ea
f3b8a90
 
 
 
 
 
 
 
 
 
34e99ea
 
f3b8a90
 
 
34e99ea
 
19a5ca3
34e99ea
b4891e7
 
 
 
 
19a5ca3
b4891e7
19a5ca3
34e99ea
 
6052894
19a5ca3
 
b4891e7
 
 
 
 
 
 
 
 
 
 
 
6052894
b4891e7
 
 
 
06e607e
 
 
dfad153
06e607e
 
 
 
 
 
 
 
 
 
 
 
 
b4891e7
19a5ca3
 
928cf15
19a5ca3
 
 
 
 
b4891e7
06e607e
19a5ca3
6052894
19a5ca3
 
34e99ea
19a5ca3
b4891e7
19a5ca3
 
34e99ea
19a5ca3
 
 
 
b4891e7
19a5ca3
 
 
6052894
19a5ca3
 
34e99ea
19a5ca3
b4891e7
19a5ca3
 
34e99ea
19a5ca3
 
 
 
 
 
 
 
6052894
19a5ca3
 
34e99ea
19a5ca3
06e607e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b4891e7
19a5ca3
 
34e99ea
19a5ca3
 
 
 
 
 
 
 
6052894
19a5ca3
 
34e99ea
19a5ca3
6052894
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b4891e7
 
 
 
 
 
 
 
 
 
 
 
6052894
b4891e7
 
 
 
6052894
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
dfad153
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b4891e7
19a5ca3
 
34e99ea
19a5ca3
 
 
 
 
 
 
 
6052894
19a5ca3
 
34e99ea
19a5ca3
b4891e7
19a5ca3
 
34e99ea
19a5ca3
 
 
 
 
 
 
 
6052894
19a5ca3
 
34e99ea
19a5ca3
b4891e7
19a5ca3
 
34e99ea
19a5ca3
 
 
 
 
 
 
 
6052894
19a5ca3
 
34e99ea
19a5ca3
6052894
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19a5ca3
 
 
6052894
dfad153
6052894
 
 
 
dfad153
6052894
 
 
 
 
 
 
 
19a5ca3
6052894
8ba9c90
f3b8a90
6052894
f3b8a90
 
afb4866
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Local LLM Benchmarks</title>
<link rel="stylesheet" href="style.css">
</head>
<body>
<header class="top">
  <a class="brand" href="index.html">Local LLM Benchmarks</a>
  <nav>
    <a href="index.html" class="active">Overview</a>
    <a href="methodology.html">Methodology</a>
  </nav>
</header>
<main>
<h1>Overview</h1>
<p class="lead">Local language models compared: knowledge, math, code and German, plus speed. Each model variant is shown with its most recent completed run. How the numbers are measured is described under <a href="methodology.html">Methodology</a>.</p>
<section id="v3-quick">
<h2>Suite v2b · profile quick</h2>
<p class="segmented" role="group" aria-label="Filter by reasoning mode" data-filter-table="table-v3-quick">
  <button type="button" class="active" aria-pressed="true" data-reasoning-filter="all">All runs</button>
  <button type="button" aria-pressed="false" data-reasoning-filter="off">Reasoning off</button>
  <button type="button" aria-pressed="false" data-reasoning-filter="on">Reasoning on</button>
</p>
<div class="table-wrap">
<table class="sortable" id="table-v3-quick">
<thead><tr>
  <th data-sort="text">Model</th>
  <th data-sort="num" class="num">Overall</th>
<th data-sort="num" class="num">Knowledge &amp; reasoning</th><th data-sort="num" class="num">Math</th><th data-sort="num" class="num">Instruction following</th><th data-sort="num" class="num">Code</th><th data-sort="num" class="num">German</th><th data-sort="num" class="num">gen t/s</th><th data-sort="num" class="num" title="Generation per request measured during the benchmark run; gen t/s before it comes from the synthetic speed test">gen t/s run</th><th data-sort="num" class="num">pp t/s 4k</th><th data-sort="num" class="num">TTFT 4k ms</th>  <th data-sort="text">Date</th>
</tr></thead>
<tbody>
<tr data-reasoning="on">
  <td data-value="gpt-oss-120b">
    <a href="runs/18.html" class="strong">gpt-oss-120b</a>
    <div class="sub">MXFP4 · Reasoning on · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="73.08253056884635">73.1</td>
  <td class="num" data-value="68.52727272727272" title="mmlu_pro: 71.0 ± 2.6 (n=300)&#10;gpqa_diamond_gen: 68.2 ± 3.3 (n=198)&#10;musr_gen: 66.4 ± 3.0 (n=250)&#10;">68.5</td>
  <td class="num" data-value="70.39999999999999" title="omni_math_gen: 70.4 ± 2.9 (n=250)&#10;">70.4</td>
  <td class="num strong" data-value="66.1203007518797" title="ifbench: 60.0 ± 2.8 (n=300)&#10;ifeval: 84.7 ± 3.0 (n=150)&#10;">66.1</td>
  <td class="num strong" data-value="83.0" title="livecodebench_gen: 83.0 ± 2.7 (n=200)&#10;">83.0</td>
  <td class="num" data-value="77.36507936507935" title="mmlu_prox_de: 80.3 ± 2.3 (n=300)&#10;include_de_gen: 71.4 ± 3.7 (n=154)&#10;">77.4</td>
  <td class="num" data-value="">–</td>
  <td class="num" data-value="19.638176195686693" title="overall 58.4 t/s with 3 parallel requests, from 9 of 9 tasks">20</td>
  <td class="num" data-value="">–</td>
  <td class="num" data-value="">–</td>
  <td data-value="2026-09-18">2026-09-18</td>
</tr>
<tr data-reasoning="off">
  <td data-value="qwen3.8-flash-next-abliterated (pool)">
    <a href="runs/32.html" class="strong">qwen3.8-flash-next-abliterated (pool)</a>
    <div class="sub">W4A16-NVFP4 · Reasoning off · direct · 2× NVIDIA DGX Spark, je 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="72.53692007797271">72.5</td>
  <td class="num" data-value="77.15084175084174" title="mmlu_pro: 87.0 ± 1.9 (n=300)&#10;gpqa_diamond_gen: 75.3 ± 3.1 (n=198)&#10;musr_gen: 69.2 ± 2.9 (n=250)&#10;">77.2</td>
  <td class="num" data-value="72.0" title="omni_math_gen: 72.0 ± 2.8 (n=250)&#10;">72.0</td>
  <td class="num" data-value="55.16290726817043" title="ifbench: 43.7 ± 2.9 (n=300)&#10;ifeval: 90.0 ± 2.5 (n=150)&#10;">55.2</td>
  <td class="num" data-value="77.0" title="livecodebench_gen: 77.0 ± 3.0 (n=200)&#10;">77.0</td>
  <td class="num strong" data-value="81.37085137085137" title="mmlu_prox_de: 86.7 ± 2.0 (n=300)&#10;include_de_gen: 70.8 ± 3.7 (n=154)&#10;">81.4</td>
  <td class="num" data-value="41.18692185852276">41</td>
  <td class="num" data-value="37.23226449422559" title="overall 113.3 t/s with 3 parallel requests, from 9 of 9 tasks">37</td>
  <td class="num" data-value="2569.0663350056934">2569</td>
  <td class="num" data-value="1566.3277919957181">1566</td>
  <td data-value="2026-09-27">2026-09-27</td>
</tr>
<tr data-reasoning="off">
  <td data-value="qwen3.8-flash-next-ple4">
    <a href="runs/15.html" class="strong">qwen3.8-flash-next-ple4</a>
    <div class="sub">W4A16 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="71.45393635604162">71.5</td>
  <td class="num strong" data-value="77.86868686868686" title="mmlu_pro: 86.3 ± 2.0 (n=300)&#10;gpqa_diamond_gen: 77.3 ± 3.0 (n=198)&#10;musr_gen: 70.0 ± 2.9 (n=250)&#10;">77.9</td>
  <td class="num strong" data-value="73.2" title="omni_math_gen: 73.2 ± 2.8 (n=250)&#10;">73.2</td>
  <td class="num" data-value="49.56390977443609" title="ifbench: 36.0 ± 2.8 (n=300)&#10;ifeval: 90.7 ± 2.4 (n=150)&#10;">49.6</td>
  <td class="num" data-value="75.5" title="livecodebench_gen: 75.5 ± 3.0 (n=200)&#10;">75.5</td>
  <td class="num" data-value="81.13708513708514" title="mmlu_prox_de: 85.7 ± 2.0 (n=300)&#10;include_de_gen: 72.1 ± 3.6 (n=154)&#10;">81.1</td>
  <td class="num" data-value="26.376464212349354">26</td>
  <td class="num" data-value="21.59310838133867" title="overall 64.4 t/s with 3 parallel requests, from 5 of 9 tasks">22</td>
  <td class="num" data-value="2041.4640201348918">2041</td>
  <td class="num" data-value="1973.583644023165">1974</td>
  <td data-value="2026-09-17">2026-09-17</td>
</tr>
<tr data-reasoning="off">
  <td data-value="gemma-4-12b-it">
    <a href="runs/21.html" class="strong">gemma-4-12b-it</a>
    <div class="sub">Q8 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="65.82462114883167">65.8</td>
  <td class="num" data-value="67.48821548821549" title="mmlu_pro: 79.3 ± 2.3 (n=300)&#10;gpqa_diamond_gen: 63.1 ± 3.4 (n=198)&#10;musr_gen: 60.0 ± 3.1 (n=250)&#10;">67.5</td>
  <td class="num" data-value="60.8" title="omni_math_gen: 60.8 ± 3.1 (n=250)&#10;">60.8</td>
  <td class="num" data-value="51.318295739348365" title="ifbench: 38.3 ± 2.8 (n=300)&#10;ifeval: 90.7 ± 2.4 (n=150)&#10;">51.3</td>
  <td class="num" data-value="75.0" title="livecodebench_gen: 75.0 ± 3.1 (n=200)&#10;">75.0</td>
  <td class="num" data-value="74.51659451659451" title="mmlu_prox_de: 78.3 ± 2.4 (n=300)&#10;include_de_gen: 66.9 ± 3.8 (n=154)&#10;">74.5</td>
  <td class="num" data-value="16.51787775562755">17</td>
  <td class="num" data-value="13.757625586585043" title="overall 41.2 t/s with 3 parallel requests, from 5 of 9 tasks">14</td>
  <td class="num" data-value="1359.0489942173106">1359</td>
  <td class="num" data-value="2952.064286917448">2952</td>
  <td data-value="2026-09-18">2026-09-18</td>
</tr>
<tr data-reasoning="off">
  <td data-value="gemma-4-12b-it-heretic">
    <a href="runs/22.html" class="strong">gemma-4-12b-it-heretic</a>
    <div class="sub">Q8 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="65.82311637680058">65.8</td>
  <td class="num" data-value="68.44713804713804" title="mmlu_pro: 80.0 ± 2.3 (n=300)&#10;gpqa_diamond_gen: 64.1 ± 3.4 (n=198)&#10;musr_gen: 61.2 ± 3.1 (n=250)&#10;">68.4</td>
  <td class="num" data-value="59.199999999999996" title="omni_math_gen: 59.2 ± 3.1 (n=250)&#10;">59.2</td>
  <td class="num" data-value="49.97493734335839" title="ifbench: 35.7 ± 2.8 (n=300)&#10;ifeval: 93.3 ± 2.0 (n=150)&#10;">50.0</td>
  <td class="num" data-value="75.0" title="livecodebench_gen: 75.0 ± 3.1 (n=200)&#10;">75.0</td>
  <td class="num" data-value="76.4935064935065" title="mmlu_prox_de: 80.0 ± 2.3 (n=300)&#10;include_de_gen: 69.5 ± 3.7 (n=154)&#10;">76.5</td>
  <td class="num" data-value="16.51271607105954">17</td>
  <td class="num" data-value="13.819760405181762" title="overall 41.0 t/s with 3 parallel requests, from 5 of 9 tasks">14</td>
  <td class="num" data-value="1358.107397713914">1358</td>
  <td class="num" data-value="2954.110997961834">2954</td>
  <td data-value="2026-09-19">2026-09-19</td>
</tr>
<tr data-reasoning="off">
  <td data-value="qwen3-coder-next">
    <a href="runs/30.html" class="strong">qwen3-coder-next</a>
    <div class="sub">Q6 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="65.05957469431154">65.1</td>
  <td class="num" data-value="71.26060606060607" title="mmlu_pro: 82.0 ± 2.2 (n=300)&#10;gpqa_diamond_gen: 68.2 ± 3.3 (n=198)&#10;musr_gen: 63.6 ± 3.0 (n=250)&#10;">71.3</td>
  <td class="num" data-value="61.6" title="omni_math_gen: 61.6 ± 3.1 (n=250)&#10;">61.6</td>
  <td class="num" data-value="45.89974937343358" title="ifbench: 32.7 ± 2.7 (n=300)&#10;ifeval: 86.0 ± 2.8 (n=150)&#10;">45.9</td>
  <td class="num" data-value="70.5" title="livecodebench_gen: 70.5 ± 3.2 (n=200)&#10;">70.5</td>
  <td class="num" data-value="76.03751803751804" title="mmlu_prox_de: 78.7 ± 2.4 (n=300)&#10;include_de_gen: 70.8 ± 3.7 (n=154)&#10;">76.0</td>
  <td class="num" data-value="43.69461761825484">44</td>
  <td class="num" data-value="23.80832467007433" title="overall 59.7 t/s with 3 parallel requests, from 9 of 9 tasks">24</td>
  <td class="num" data-value="987.8475690816915">988</td>
  <td class="num" data-value="4077.5521710747853">4078</td>
  <td data-value="2026-09-23">2026-09-23</td>
</tr>
<tr data-reasoning="off">
  <td data-value="qwen3.8-27b-mtp">
    <a href="runs/20.html" class="strong">qwen3.8-27b-mtp</a>
    <div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="64.20301055669476">64.2</td>
  <td class="num" data-value="69.85656565656565" title="mmlu_pro: 80.3 ± 2.3 (n=300)&#10;gpqa_diamond_gen: 63.6 ± 3.4 (n=198)&#10;musr_gen: 65.6 ± 3.0 (n=250)&#10;">69.9</td>
  <td class="num" data-value="60.4" title="omni_math_gen: 60.4 ± 3.1 (n=250)&#10;">60.4</td>
  <td class="num" data-value="49.498746867167924" title="ifbench: 38.3 ± 2.8 (n=300)&#10;ifeval: 83.3 ± 3.1 (n=150)&#10;">49.5</td>
  <td class="num" data-value="65.0" title="livecodebench_gen: 65.0 ± 3.4 (n=200)&#10;">65.0</td>
  <td class="num" data-value="76.25974025974027" title="mmlu_prox_de: 79.0 ± 2.4 (n=300)&#10;include_de_gen: 70.8 ± 3.7 (n=154)&#10;">76.3</td>
  <td class="num" data-value="15.103805505442727">15</td>
  <td class="num" data-value="22.383948355383822" title="overall 67.6 t/s with 3 parallel requests, from 5 of 9 tasks">22</td>
  <td class="num" data-value="2550.5619640606633">2551</td>
  <td class="num" data-value="1577.6915270835161">1578</td>
  <td data-value="2026-09-18">2026-09-18</td>
</tr>
<tr data-reasoning="off">
  <td data-value="occamy-1.0">
    <a href="runs/27.html" class="strong">occamy-1.0</a>
    <div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="64.03163768005874">64.0</td>
  <td class="num" data-value="73.1057239057239" title="mmlu_pro: 84.0 ± 2.1 (n=300)&#10;gpqa_diamond_gen: 71.7 ± 3.2 (n=198)&#10;musr_gen: 63.6 ± 3.0 (n=250)&#10;">73.1</td>
  <td class="num" data-value="62.4" title="omni_math_gen: 62.4 ± 3.1 (n=250)&#10;">62.4</td>
  <td class="num" data-value="44.73182957393484" title="ifbench: 31.3 ± 2.7 (n=300)&#10;ifeval: 85.3 ± 2.9 (n=150)&#10;">44.7</td>
  <td class="num" data-value="63.0" title="livecodebench_gen: 63.0 ± 3.4 (n=200)&#10;">63.0</td>
  <td class="num" data-value="76.92063492063492" title="mmlu_prox_de: 79.7 ± 2.3 (n=300)&#10;include_de_gen: 71.4 ± 3.7 (n=154)&#10;">76.9</td>
  <td class="num" data-value="36.719412085128695">37</td>
  <td class="num" data-value="31.562617796089718" title="overall 94.1 t/s with 3 parallel requests, from 9 of 9 tasks">32</td>
  <td class="num" data-value="5539.414736774487">5539</td>
  <td class="num" data-value="726.4305330463685">726</td>
  <td data-value="2026-09-21">2026-09-21</td>
</tr>
<tr data-reasoning="off">
  <td data-value="qwen3.8-27b-abliterated">
    <a href="runs/23.html" class="strong">qwen3.8-27b-abliterated</a>
    <div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="61.428784587731954">61.4</td>
  <td class="num" data-value="71.50235690235691" title="mmlu_pro: 81.0 ± 2.3 (n=300)&#10;gpqa_diamond_gen: 70.7 ± 3.2 (n=198)&#10;musr_gen: 62.8 ± 3.1 (n=250)&#10;">71.5</td>
  <td class="num" data-value="57.99999999999999" title="omni_math_gen: 58.0 ± 3.1 (n=250)&#10;">58.0</td>
  <td class="num" data-value="46.07518796992481" title="ifbench: 34.0 ± 2.7 (n=300)&#10;ifeval: 82.7 ± 3.1 (n=150)&#10;">46.1</td>
  <td class="num" data-value="59.5" title="livecodebench_gen: 59.5 ± 3.5 (n=200)&#10;">59.5</td>
  <td class="num" data-value="72.06637806637806" title="mmlu_prox_de: 74.3 ± 2.5 (n=300)&#10;include_de_gen: 67.5 ± 3.8 (n=154)&#10;">72.1</td>
  <td class="num" data-value="15.903617023850842">16</td>
  <td class="num" data-value="21.942627395574263" title="overall 67.0 t/s with 3 parallel requests, from 9 of 9 tasks">22</td>
  <td class="num" data-value="2472.0116600865786">2472</td>
  <td class="num" data-value="1627.824036986567">1628</td>
  <td data-value="2026-09-20">2026-09-20</td>
</tr>
<tr data-reasoning="off">
  <td data-value="ternary-bonsai-2-27b">
    <a href="runs/26.html" class="strong">ternary-bonsai-2-27b</a>
    <div class="sub">FP8 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="61.28656742867268">61.3</td>
  <td class="num" data-value="69.5077441077441" title="mmlu_pro: 77.7 ± 2.4 (n=300)&#10;gpqa_diamond_gen: 65.7 ± 3.4 (n=198)&#10;musr_gen: 65.2 ± 3.0 (n=250)&#10;">69.5</td>
  <td class="num" data-value="59.599999999999994" title="omni_math_gen: 59.6 ± 3.1 (n=250)&#10;">59.6</td>
  <td class="num" data-value="46.32581453634085" title="ifbench: 34.3 ± 2.7 (n=300)&#10;ifeval: 82.7 ± 3.1 (n=150)&#10;">46.3</td>
  <td class="num" data-value="56.49999999999999" title="livecodebench_gen: 56.5 ± 3.5 (n=200)&#10;">56.5</td>
  <td class="num" data-value="74.4992784992785" title="mmlu_prox_de: 77.3 ± 2.4 (n=300)&#10;include_de_gen: 68.8 ± 3.7 (n=154)&#10;">74.5</td>
  <td class="num" data-value="28.488199681723682">28</td>
  <td class="num" data-value="19.303084023060574" title="overall 58.5 t/s with 3 parallel requests, from 9 of 9 tasks">19</td>
  <td class="num" data-value="898.1292759650371">898</td>
  <td class="num" data-value="4480.424041044898">4480</td>
  <td data-value="2026-09-21">2026-09-21</td>
</tr>
<tr data-reasoning="off">
  <td data-value="mimo-v2.6-flash (pool)">
    <a href="runs/31.html" class="strong">mimo-v2.6-flash (pool)</a>
    <div class="sub">FP8 + MXFP4 · Reasoning off · direct · 2× NVIDIA DGX Spark, je 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="59.427507658033974">59.4</td>
  <td class="num" data-value="69.69562289562289" title="mmlu_pro: 82.0 ± 2.2 (n=300)&#10;gpqa_diamond_gen: 68.7 ± 3.3 (n=198)&#10;musr_gen: 58.4 ± 3.1 (n=250)&#10;">69.7</td>
  <td class="num" data-value="41.6" title="omni_math_gen: 41.6 ± 3.1 (n=250)&#10;">41.6</td>
  <td class="num" data-value="49.819548872180455" title="ifbench: 37.0 ± 2.8 (n=300)&#10;ifeval: 88.7 ± 2.6 (n=150)&#10;">49.8</td>
  <td class="num" data-value="61.5" title="livecodebench_gen: 61.5 ± 3.4 (n=200)&#10;">61.5</td>
  <td class="num" data-value="74.52236652236651" title="mmlu_prox_de: 78.7 ± 2.4 (n=300)&#10;include_de_gen: 66.2 ± 3.8 (n=154)&#10;">74.5</td>
  <td class="num" data-value="27.055489674547132">27</td>
  <td class="num" data-value="21.7770272387994" title="overall 70.2 t/s with 3 parallel requests, from 9 of 9 tasks">22</td>
  <td class="num" data-value="2177.560718839664">2178</td>
  <td class="num" data-value="1847.9392860026564">1848</td>
  <td data-value="2026-09-26">2026-09-26</td>
</tr>
<tr data-reasoning="off">
  <td data-value="gemma-4-12b-it-heretic">
    <a href="runs/17.html" class="strong">gemma-4-12b-it-heretic</a>
    <div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="57.7166917293233">57.7</td>
  <td class="num" data-value="63.70909090909091" title="mmlu_pro: 76.7 ± 2.4 (n=300)&#10;gpqa_diamond_gen: 56.1 ± 3.5 (n=198)&#10;musr_gen: 58.4 ± 3.1 (n=250)&#10;">63.7</td>
  <td class="num" data-value="48.4" title="omni_math_gen: 48.4 ± 3.2 (n=250)&#10;">48.4</td>
  <td class="num" data-value="46.81203007518797" title="ifbench: 33.0 ± 2.7 (n=300)&#10;ifeval: 88.7 ± 2.6 (n=150)&#10;">46.8</td>
  <td class="num" data-value="62.0" title="livecodebench_gen: 62.0 ± 3.4 (n=200)&#10;">62.0</td>
  <td class="num" data-value="67.66233766233766" title="mmlu_prox_de: 70.0 ± 2.7 (n=300)&#10;include_de_gen: 63.0 ± 3.9 (n=154)&#10;">67.7</td>
  <td class="num" data-value="21.360681311630678">21</td>
  <td class="num" data-value="21.121219019103503" title="overall 62.6 t/s with 3 parallel requests, from 5 of 9 tasks">21</td>
  <td class="num" data-value="5258.027383235864">5258</td>
  <td class="num" data-value="763.404164230451">763</td>
  <td data-value="2026-09-18">2026-09-18</td>
</tr>
<tr data-reasoning="off">
  <td data-value="nemotron-3.5-lightning-dspark">
    <a href="runs/16.html" class="strong">nemotron-3.5-lightning-dspark</a>
    <div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="53.13354497354497">53.1</td>
  <td class="num" data-value="55.756902356902366" title="mmlu_pro: 71.0 ± 2.6 (n=300)&#10;gpqa_diamond_gen: 57.1 ± 3.5 (n=198)&#10;musr_gen: 39.2 ± 3.1 (n=250)&#10;">55.8</td>
  <td class="num" data-value="40.400000000000006" title="omni_math_gen: 40.4 ± 3.1 (n=250)&#10;">40.4</td>
  <td class="num" data-value="46.0" title="ifbench: 35.0 ± 2.8 (n=300)&#10;ifeval: 79.3 ± 3.3 (n=150)&#10;">46.0</td>
  <td class="num" data-value="53.0" title="livecodebench_gen: 53.0 ± 3.5 (n=200)&#10;">53.0</td>
  <td class="num" data-value="70.5108225108225" title="mmlu_prox_de: 72.0 ± 2.6 (n=300)&#10;include_de_gen: 67.5 ± 3.8 (n=154)&#10;">70.5</td>
  <td class="num" data-value="83.7975869946743">84</td>
  <td class="num" data-value="70.42131680616991" title="overall 213.6 t/s with 3 parallel requests, from 5 of 9 tasks">70</td>
  <td class="num" data-value="7236.849600555504">7237</td>
  <td class="num" data-value="554.1085170116276">554</td>
  <td data-value="2026-09-17">2026-09-17</td>
</tr>
<tr data-reasoning="off">
  <td data-value="qwen3.5-9b-heretic-v2-mtp">
    <a href="runs/19.html" class="strong">qwen3.5-9b-heretic-v2-mtp</a>
    <div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="51.662634870003295">51.7</td>
  <td class="num" data-value="65.07138047138046" title="mmlu_pro: 76.3 ± 2.5 (n=300)&#10;gpqa_diamond_gen: 58.1 ± 3.5 (n=198)&#10;musr_gen: 60.8 ± 3.1 (n=250)&#10;">65.1</td>
  <td class="num" data-value="35.6" title="omni_math_gen: 35.6 ± 3.0 (n=250)&#10;">35.6</td>
  <td class="num" data-value="44.140350877192986" title="ifbench: 29.7 ± 2.6 (n=300)&#10;ifeval: 88.0 ± 2.7 (n=150)&#10;">44.1</td>
  <td class="num" data-value="46.5" title="livecodebench_gen: 46.5 ± 3.5 (n=200)&#10;">46.5</td>
  <td class="num" data-value="67.00144300144301" title="mmlu_prox_de: 69.3 ± 2.7 (n=300)&#10;include_de_gen: 62.3 ± 3.9 (n=154)&#10;">67.0</td>
  <td class="num" data-value="31.530246994028012">32</td>
  <td class="num" data-value="35.229241139471675" title="overall 106.0 t/s with 3 parallel requests, from 5 of 9 tasks">35</td>
  <td class="num" data-value="6290.925546819461">6291</td>
  <td class="num" data-value="639.6515059750527">640</td>
  <td data-value="2026-09-18">2026-09-18</td>
</tr>
<tr data-reasoning="off">
  <td data-value="zgcm-1-7b">
    <a href="runs/29.html" class="strong">zgcm-1-7b</a>
    <div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
  </td>
  <td class="num strong" data-value="33.36777195514038">33.4</td>
  <td class="num" data-value="34.68552188552189" title="mmlu_pro: 25.3 ± 2.5 (n=300)&#10;gpqa_diamond_gen: 32.3 ± 3.3 (n=198)&#10;musr_gen: 46.4 ± 3.2 (n=250)&#10;">34.7</td>
  <td class="num" data-value="40.0" title="omni_math_gen: 40.0 ± 3.1 (n=250)&#10;">40.0</td>
  <td class="num" data-value="44.33082706766917" title="ifbench: 33.0 ± 2.7 (n=300)&#10;ifeval: 78.7 ± 3.4 (n=150)&#10;">44.3</td>
  <td class="num" data-value="23.0" title="livecodebench_gen: 23.0 ± 3.0 (n=200)&#10;">23.0</td>
  <td class="num" data-value="24.82251082251082" title="mmlu_prox_de: 21.0 ± 2.4 (n=300)&#10;include_de_gen: 32.5 ± 3.8 (n=154)&#10;">24.8</td>
  <td class="num" data-value="33.67868964254804">34</td>
  <td class="num" data-value="34.0307978654491" title="overall 102.4 t/s with 3 parallel requests, from 9 of 9 tasks">34</td>
  <td class="num" data-value="3957.900821130251">3958</td>
  <td class="num" data-value="1022.5117260124534">1023</td>
  <td data-value="2026-09-23">2026-09-23</td>
</tr>
</tbody>
</table>
</div>
<div class="chart-set">
<script type="application/json" class="chart-data" id="charts-v3-quick">{"categories": [{"key": "overall", "label": "Overall"}, {"key": "knowledge", "label": "Knowledge \u0026 reasoning"}, {"key": "math", "label": "Math"}, {"key": "instructions", "label": "Instruction following"}, {"key": "code", "label": "Code"}, {"key": "german", "label": "German"}], "missing_tps": false, "reasoning_mark": "\ud83d\udcad", "runs": [{"label": "gpt-oss-120b \u00b7 \ud83d\udcad", "run_id": 18, "scores": {"code": 83.0, "german": 77.4, "instructions": 66.1, "knowledge": 68.5, "math": 70.4, "overall": 73.1}, "tps": 19.6, "tps_source": "run"}, {"label": "qwen3.8-flash-next-abliterated (pool)", "run_id": 32, "scores": {"code": 77.0, "german": 81.4, "instructions": 55.2, "knowledge": 77.2, "math": 72.0, "overall": 72.5}, "tps": 37.2, "tps_source": "run"}, {"label": "qwen3.8-flash-next-ple4", "run_id": 15, "scores": {"code": 75.5, "german": 81.1, "instructions": 49.6, "knowledge": 77.9, "math": 73.2, "overall": 71.5}, "tps": 21.6, "tps_source": "run"}, {"label": "gemma-4-12b-it", "run_id": 21, "scores": {"code": 75.0, "german": 74.5, "instructions": 51.3, "knowledge": 67.5, "math": 60.8, "overall": 65.8}, "tps": 13.8, "tps_source": "run"}, {"label": "gemma-4-12b-it-heretic", "run_id": 22, "scores": {"code": 75.0, "german": 76.5, "instructions": 50.0, "knowledge": 68.4, "math": 59.2, "overall": 65.8}, "tps": 13.8, "tps_source": "run"}, {"label": "qwen3-coder-next", "run_id": 30, "scores": {"code": 70.5, "german": 76.0, "instructions": 45.9, "knowledge": 71.3, "math": 61.6, "overall": 65.1}, "tps": 23.8, "tps_source": "run"}, {"label": "qwen3.8-27b-mtp", "run_id": 20, "scores": {"code": 65.0, "german": 76.3, "instructions": 49.5, "knowledge": 69.9, "math": 60.4, "overall": 64.2}, "tps": 22.4, "tps_source": "run"}, {"label": "occamy-1.0", "run_id": 27, "scores": {"code": 63.0, "german": 76.9, "instructions": 44.7, "knowledge": 73.1, "math": 62.4, "overall": 64.0}, "tps": 31.6, "tps_source": "run"}, {"label": "qwen3.8-27b-abliterated", "run_id": 23, "scores": {"code": 59.5, "german": 72.1, "instructions": 46.1, "knowledge": 71.5, "math": 58.0, "overall": 61.4}, "tps": 21.9, "tps_source": "run"}, {"label": "ternary-bonsai-2-27b", "run_id": 26, "scores": {"code": 56.5, "german": 74.5, "instructions": 46.3, "knowledge": 69.5, "math": 59.6, "overall": 61.3}, "tps": 19.3, "tps_source": "run"}, {"label": "mimo-v2.6-flash (pool)", "run_id": 31, "scores": {"code": 61.5, "german": 74.5, "instructions": 49.8, "knowledge": 69.7, "math": 41.6, "overall": 59.4}, "tps": 21.8, "tps_source": "run"}, {"label": "gemma-4-12b-it-heretic", "run_id": 17, "scores": {"code": 62.0, "german": 67.7, "instructions": 46.8, "knowledge": 63.7, "math": 48.4, "overall": 57.7}, "tps": 21.1, "tps_source": "run"}, {"label": "nemotron-3.5-lightning-dspark", "run_id": 16, "scores": {"code": 53.0, "german": 70.5, "instructions": 46.0, "knowledge": 55.8, "math": 40.4, "overall": 53.1}, "tps": 70.4, "tps_source": "run"}, {"label": "qwen3.5-9b-heretic-v2-mtp", "run_id": 19, "scores": {"code": 46.5, "german": 67.0, "instructions": 44.1, "knowledge": 65.1, "math": 35.6, "overall": 51.7}, "tps": 35.2, "tps_source": "run"}, {"label": "zgcm-1-7b", "run_id": 29, "scores": {"code": 23.0, "german": 24.8, "instructions": 44.3, "knowledge": 34.7, "math": 40.0, "overall": 33.4}, "tps": 34.0, "tps_source": "run"}], "texts": {"overall": "Overall", "overall_title": "Overall score", "points": "points", "score_axis": "Score (0\u2013100)", "source_run": "measured during the run", "source_test": "from the speed test", "tps_axis": "Generation (tokens/s)", "vs": " vs. generation (tokens/s)"}}</script>
<p class="segmented" role="group" aria-label="Chart category"><button type="button" data-chart-category="overall" class="active" aria-pressed="true">Overall</button><button type="button" data-chart-category="knowledge" class="" aria-pressed="false">Knowledge &amp; reasoning</button><button type="button" data-chart-category="math" class="" aria-pressed="false">Math</button><button type="button" data-chart-category="instructions" class="" aria-pressed="false">Instruction following</button><button type="button" data-chart-category="code" class="" aria-pressed="false">Code</button><button type="button" data-chart-category="german" class="" aria-pressed="false">German</button></p>
<div class="charts">
  <figure class="chart-card">
    <figcaption><span data-chart-title="bar">Overall score</span> <span class="sub">(💭 = with reasoning)</span></figcaption>
    <div class="chart-box" data-chart-box="bar" style="height: 520px"><canvas data-chart="bar" role="img" aria-label="Score per model variant; the values are also in the table"></canvas></div>
  </figure>
  <figure class="chart-card">
    <figcaption><span data-chart-title="scatter">Overall score vs. generation (tokens/s)</span> <span class="sub">(💭 = with reasoning)</span></figcaption>
    <div class="chart-box" style="height: 320px"><canvas data-chart="scatter" role="img" aria-label="Score against generation speed in tokens per second; the values are also in the table"></canvas></div>
    <p class="hint">x-axis: generation per request measured during the run; hollow points only have the speed-test value.</p>
  </figure>
</div>
</div>
</section>
<p class="hint">Scores run from 0 to 100; the best value per category is shown in bold. Hovering a category cell lists its tasks with standard error (±) and number of items (n). Speed: generation in tokens/s from the speed test and measured during the run (gen t/s run, per request), prompt processing at 4k context in tokens/s, time to first token at 4k context in ms. Click a column header to sort.</p>
<footer class="page-foot">As of 2026-10-10 · scores 0–100 · built with <code>bench publish</code></footer>
</main>
<script src="chart.umd.min.js"></script>
<script src="app.js"></script>
</body>
</html>