LocalLLMBenchmarks / index.html
Kaleto's picture
Ergebnisse aktualisiert (15 Läufe)
8ba9c90 verified
Raw History Blame Contribute Delete
27.5 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Local LLM Benchmarks</title>
<link rel="stylesheet" href="style.css">
</head>
<body>
<header class="top">
<a class="brand" href="index.html">Local LLM Benchmarks</a>
<nav>
<a href="index.html" class="active">Overview</a>
<a href="methodology.html">Methodology</a>
</nav>
</header>
<main>
<h1>Overview</h1>
<p class="lead">Local language models compared: knowledge, math, code and German, plus speed. Each model variant is shown with its most recent completed run. How the numbers are measured is described under <a href="methodology.html">Methodology</a>.</p>
<section id="v3-quick">
<h2>Suite v2b · profile quick</h2>
<p class="segmented" role="group" aria-label="Filter by reasoning mode" data-filter-table="table-v3-quick">
<button type="button" class="active" aria-pressed="true" data-reasoning-filter="all">All runs</button>
<button type="button" aria-pressed="false" data-reasoning-filter="off">Reasoning off</button>
<button type="button" aria-pressed="false" data-reasoning-filter="on">Reasoning on</button>
</p>
<div class="table-wrap">
<table class="sortable" id="table-v3-quick">
<thead><tr>
<th data-sort="text">Model</th>
<th data-sort="num" class="num">Overall</th>
<th data-sort="num" class="num">Knowledge &amp; reasoning</th><th data-sort="num" class="num">Math</th><th data-sort="num" class="num">Instruction following</th><th data-sort="num" class="num">Code</th><th data-sort="num" class="num">German</th><th data-sort="num" class="num">gen t/s</th><th data-sort="num" class="num" title="Generation per request measured during the benchmark run; gen t/s before it comes from the synthetic speed test">gen t/s run</th><th data-sort="num" class="num">pp t/s 4k</th><th data-sort="num" class="num">TTFT 4k ms</th> <th data-sort="text">Date</th>
</tr></thead>
<tbody>
<tr data-reasoning="on">
<td data-value="gpt-oss-120b">
<a href="runs/18.html" class="strong">gpt-oss-120b</a>
<div class="sub">MXFP4 · Reasoning on · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="73.08253056884635">73.1</td>
<td class="num" data-value="68.52727272727272" title="mmlu_pro: 71.0 ± 2.6 (n=300)&#10;gpqa_diamond_gen: 68.2 ± 3.3 (n=198)&#10;musr_gen: 66.4 ± 3.0 (n=250)&#10;">68.5</td>
<td class="num" data-value="70.39999999999999" title="omni_math_gen: 70.4 ± 2.9 (n=250)&#10;">70.4</td>
<td class="num strong" data-value="66.1203007518797" title="ifbench: 60.0 ± 2.8 (n=300)&#10;ifeval: 84.7 ± 3.0 (n=150)&#10;">66.1</td>
<td class="num strong" data-value="83.0" title="livecodebench_gen: 83.0 ± 2.7 (n=200)&#10;">83.0</td>
<td class="num" data-value="77.36507936507935" title="mmlu_prox_de: 80.3 ± 2.3 (n=300)&#10;include_de_gen: 71.4 ± 3.7 (n=154)&#10;">77.4</td>
<td class="num" data-value="">–</td>
<td class="num" data-value="19.638176195686693" title="overall 58.4 t/s with 3 parallel requests, from 9 of 9 tasks">20</td>
<td class="num" data-value="">–</td>
<td class="num" data-value="">–</td>
<td data-value="2026-09-18">2026-09-18</td>
</tr>
<tr data-reasoning="off">
<td data-value="qwen3.8-flash-next-abliterated (pool)">
<a href="runs/32.html" class="strong">qwen3.8-flash-next-abliterated (pool)</a>
<div class="sub">W4A16-NVFP4 · Reasoning off · direct · 2× NVIDIA DGX Spark, je 128 GB RAM</div>
</td>
<td class="num strong" data-value="72.53692007797271">72.5</td>
<td class="num" data-value="77.15084175084174" title="mmlu_pro: 87.0 ± 1.9 (n=300)&#10;gpqa_diamond_gen: 75.3 ± 3.1 (n=198)&#10;musr_gen: 69.2 ± 2.9 (n=250)&#10;">77.2</td>
<td class="num" data-value="72.0" title="omni_math_gen: 72.0 ± 2.8 (n=250)&#10;">72.0</td>
<td class="num" data-value="55.16290726817043" title="ifbench: 43.7 ± 2.9 (n=300)&#10;ifeval: 90.0 ± 2.5 (n=150)&#10;">55.2</td>
<td class="num" data-value="77.0" title="livecodebench_gen: 77.0 ± 3.0 (n=200)&#10;">77.0</td>
<td class="num strong" data-value="81.37085137085137" title="mmlu_prox_de: 86.7 ± 2.0 (n=300)&#10;include_de_gen: 70.8 ± 3.7 (n=154)&#10;">81.4</td>
<td class="num" data-value="41.18692185852276">41</td>
<td class="num" data-value="37.23226449422559" title="overall 113.3 t/s with 3 parallel requests, from 9 of 9 tasks">37</td>
<td class="num" data-value="2569.0663350056934">2569</td>
<td class="num" data-value="1566.3277919957181">1566</td>
<td data-value="2026-09-27">2026-09-27</td>
</tr>
<tr data-reasoning="off">
<td data-value="qwen3.8-flash-next-ple4">
<a href="runs/15.html" class="strong">qwen3.8-flash-next-ple4</a>
<div class="sub">W4A16 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="71.45393635604162">71.5</td>
<td class="num strong" data-value="77.86868686868686" title="mmlu_pro: 86.3 ± 2.0 (n=300)&#10;gpqa_diamond_gen: 77.3 ± 3.0 (n=198)&#10;musr_gen: 70.0 ± 2.9 (n=250)&#10;">77.9</td>
<td class="num strong" data-value="73.2" title="omni_math_gen: 73.2 ± 2.8 (n=250)&#10;">73.2</td>
<td class="num" data-value="49.56390977443609" title="ifbench: 36.0 ± 2.8 (n=300)&#10;ifeval: 90.7 ± 2.4 (n=150)&#10;">49.6</td>
<td class="num" data-value="75.5" title="livecodebench_gen: 75.5 ± 3.0 (n=200)&#10;">75.5</td>
<td class="num" data-value="81.13708513708514" title="mmlu_prox_de: 85.7 ± 2.0 (n=300)&#10;include_de_gen: 72.1 ± 3.6 (n=154)&#10;">81.1</td>
<td class="num" data-value="26.376464212349354">26</td>
<td class="num" data-value="21.59310838133867" title="overall 64.4 t/s with 3 parallel requests, from 5 of 9 tasks">22</td>
<td class="num" data-value="2041.4640201348918">2041</td>
<td class="num" data-value="1973.583644023165">1974</td>
<td data-value="2026-09-17">2026-09-17</td>
</tr>
<tr data-reasoning="off">
<td data-value="gemma-4-12b-it">
<a href="runs/21.html" class="strong">gemma-4-12b-it</a>
<div class="sub">Q8 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="65.82462114883167">65.8</td>
<td class="num" data-value="67.48821548821549" title="mmlu_pro: 79.3 ± 2.3 (n=300)&#10;gpqa_diamond_gen: 63.1 ± 3.4 (n=198)&#10;musr_gen: 60.0 ± 3.1 (n=250)&#10;">67.5</td>
<td class="num" data-value="60.8" title="omni_math_gen: 60.8 ± 3.1 (n=250)&#10;">60.8</td>
<td class="num" data-value="51.318295739348365" title="ifbench: 38.3 ± 2.8 (n=300)&#10;ifeval: 90.7 ± 2.4 (n=150)&#10;">51.3</td>
<td class="num" data-value="75.0" title="livecodebench_gen: 75.0 ± 3.1 (n=200)&#10;">75.0</td>
<td class="num" data-value="74.51659451659451" title="mmlu_prox_de: 78.3 ± 2.4 (n=300)&#10;include_de_gen: 66.9 ± 3.8 (n=154)&#10;">74.5</td>
<td class="num" data-value="16.51787775562755">17</td>
<td class="num" data-value="13.757625586585043" title="overall 41.2 t/s with 3 parallel requests, from 5 of 9 tasks">14</td>
<td class="num" data-value="1359.0489942173106">1359</td>
<td class="num" data-value="2952.064286917448">2952</td>
<td data-value="2026-09-18">2026-09-18</td>
</tr>
<tr data-reasoning="off">
<td data-value="gemma-4-12b-it-heretic">
<a href="runs/22.html" class="strong">gemma-4-12b-it-heretic</a>
<div class="sub">Q8 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="65.82311637680058">65.8</td>
<td class="num" data-value="68.44713804713804" title="mmlu_pro: 80.0 ± 2.3 (n=300)&#10;gpqa_diamond_gen: 64.1 ± 3.4 (n=198)&#10;musr_gen: 61.2 ± 3.1 (n=250)&#10;">68.4</td>
<td class="num" data-value="59.199999999999996" title="omni_math_gen: 59.2 ± 3.1 (n=250)&#10;">59.2</td>
<td class="num" data-value="49.97493734335839" title="ifbench: 35.7 ± 2.8 (n=300)&#10;ifeval: 93.3 ± 2.0 (n=150)&#10;">50.0</td>
<td class="num" data-value="75.0" title="livecodebench_gen: 75.0 ± 3.1 (n=200)&#10;">75.0</td>
<td class="num" data-value="76.4935064935065" title="mmlu_prox_de: 80.0 ± 2.3 (n=300)&#10;include_de_gen: 69.5 ± 3.7 (n=154)&#10;">76.5</td>
<td class="num" data-value="16.51271607105954">17</td>
<td class="num" data-value="13.819760405181762" title="overall 41.0 t/s with 3 parallel requests, from 5 of 9 tasks">14</td>
<td class="num" data-value="1358.107397713914">1358</td>
<td class="num" data-value="2954.110997961834">2954</td>
<td data-value="2026-09-19">2026-09-19</td>
</tr>
<tr data-reasoning="off">
<td data-value="qwen3-coder-next">
<a href="runs/30.html" class="strong">qwen3-coder-next</a>
<div class="sub">Q6 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="65.05957469431154">65.1</td>
<td class="num" data-value="71.26060606060607" title="mmlu_pro: 82.0 ± 2.2 (n=300)&#10;gpqa_diamond_gen: 68.2 ± 3.3 (n=198)&#10;musr_gen: 63.6 ± 3.0 (n=250)&#10;">71.3</td>
<td class="num" data-value="61.6" title="omni_math_gen: 61.6 ± 3.1 (n=250)&#10;">61.6</td>
<td class="num" data-value="45.89974937343358" title="ifbench: 32.7 ± 2.7 (n=300)&#10;ifeval: 86.0 ± 2.8 (n=150)&#10;">45.9</td>
<td class="num" data-value="70.5" title="livecodebench_gen: 70.5 ± 3.2 (n=200)&#10;">70.5</td>
<td class="num" data-value="76.03751803751804" title="mmlu_prox_de: 78.7 ± 2.4 (n=300)&#10;include_de_gen: 70.8 ± 3.7 (n=154)&#10;">76.0</td>
<td class="num" data-value="43.69461761825484">44</td>
<td class="num" data-value="23.80832467007433" title="overall 59.7 t/s with 3 parallel requests, from 9 of 9 tasks">24</td>
<td class="num" data-value="987.8475690816915">988</td>
<td class="num" data-value="4077.5521710747853">4078</td>
<td data-value="2026-09-23">2026-09-23</td>
</tr>
<tr data-reasoning="off">
<td data-value="qwen3.8-27b-mtp">
<a href="runs/20.html" class="strong">qwen3.8-27b-mtp</a>
<div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="64.20301055669476">64.2</td>
<td class="num" data-value="69.85656565656565" title="mmlu_pro: 80.3 ± 2.3 (n=300)&#10;gpqa_diamond_gen: 63.6 ± 3.4 (n=198)&#10;musr_gen: 65.6 ± 3.0 (n=250)&#10;">69.9</td>
<td class="num" data-value="60.4" title="omni_math_gen: 60.4 ± 3.1 (n=250)&#10;">60.4</td>
<td class="num" data-value="49.498746867167924" title="ifbench: 38.3 ± 2.8 (n=300)&#10;ifeval: 83.3 ± 3.1 (n=150)&#10;">49.5</td>
<td class="num" data-value="65.0" title="livecodebench_gen: 65.0 ± 3.4 (n=200)&#10;">65.0</td>
<td class="num" data-value="76.25974025974027" title="mmlu_prox_de: 79.0 ± 2.4 (n=300)&#10;include_de_gen: 70.8 ± 3.7 (n=154)&#10;">76.3</td>
<td class="num" data-value="15.103805505442727">15</td>
<td class="num" data-value="22.383948355383822" title="overall 67.6 t/s with 3 parallel requests, from 5 of 9 tasks">22</td>
<td class="num" data-value="2550.5619640606633">2551</td>
<td class="num" data-value="1577.6915270835161">1578</td>
<td data-value="2026-09-18">2026-09-18</td>
</tr>
<tr data-reasoning="off">
<td data-value="occamy-1.0">
<a href="runs/27.html" class="strong">occamy-1.0</a>
<div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="64.03163768005874">64.0</td>
<td class="num" data-value="73.1057239057239" title="mmlu_pro: 84.0 ± 2.1 (n=300)&#10;gpqa_diamond_gen: 71.7 ± 3.2 (n=198)&#10;musr_gen: 63.6 ± 3.0 (n=250)&#10;">73.1</td>
<td class="num" data-value="62.4" title="omni_math_gen: 62.4 ± 3.1 (n=250)&#10;">62.4</td>
<td class="num" data-value="44.73182957393484" title="ifbench: 31.3 ± 2.7 (n=300)&#10;ifeval: 85.3 ± 2.9 (n=150)&#10;">44.7</td>
<td class="num" data-value="63.0" title="livecodebench_gen: 63.0 ± 3.4 (n=200)&#10;">63.0</td>
<td class="num" data-value="76.92063492063492" title="mmlu_prox_de: 79.7 ± 2.3 (n=300)&#10;include_de_gen: 71.4 ± 3.7 (n=154)&#10;">76.9</td>
<td class="num" data-value="36.719412085128695">37</td>
<td class="num" data-value="31.562617796089718" title="overall 94.1 t/s with 3 parallel requests, from 9 of 9 tasks">32</td>
<td class="num" data-value="5539.414736774487">5539</td>
<td class="num" data-value="726.4305330463685">726</td>
<td data-value="2026-09-21">2026-09-21</td>
</tr>
<tr data-reasoning="off">
<td data-value="qwen3.8-27b-abliterated">
<a href="runs/23.html" class="strong">qwen3.8-27b-abliterated</a>
<div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="61.428784587731954">61.4</td>
<td class="num" data-value="71.50235690235691" title="mmlu_pro: 81.0 ± 2.3 (n=300)&#10;gpqa_diamond_gen: 70.7 ± 3.2 (n=198)&#10;musr_gen: 62.8 ± 3.1 (n=250)&#10;">71.5</td>
<td class="num" data-value="57.99999999999999" title="omni_math_gen: 58.0 ± 3.1 (n=250)&#10;">58.0</td>
<td class="num" data-value="46.07518796992481" title="ifbench: 34.0 ± 2.7 (n=300)&#10;ifeval: 82.7 ± 3.1 (n=150)&#10;">46.1</td>
<td class="num" data-value="59.5" title="livecodebench_gen: 59.5 ± 3.5 (n=200)&#10;">59.5</td>
<td class="num" data-value="72.06637806637806" title="mmlu_prox_de: 74.3 ± 2.5 (n=300)&#10;include_de_gen: 67.5 ± 3.8 (n=154)&#10;">72.1</td>
<td class="num" data-value="15.903617023850842">16</td>
<td class="num" data-value="21.942627395574263" title="overall 67.0 t/s with 3 parallel requests, from 9 of 9 tasks">22</td>
<td class="num" data-value="2472.0116600865786">2472</td>
<td class="num" data-value="1627.824036986567">1628</td>
<td data-value="2026-09-20">2026-09-20</td>
</tr>
<tr data-reasoning="off">
<td data-value="ternary-bonsai-2-27b">
<a href="runs/26.html" class="strong">ternary-bonsai-2-27b</a>
<div class="sub">FP8 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="61.28656742867268">61.3</td>
<td class="num" data-value="69.5077441077441" title="mmlu_pro: 77.7 ± 2.4 (n=300)&#10;gpqa_diamond_gen: 65.7 ± 3.4 (n=198)&#10;musr_gen: 65.2 ± 3.0 (n=250)&#10;">69.5</td>
<td class="num" data-value="59.599999999999994" title="omni_math_gen: 59.6 ± 3.1 (n=250)&#10;">59.6</td>
<td class="num" data-value="46.32581453634085" title="ifbench: 34.3 ± 2.7 (n=300)&#10;ifeval: 82.7 ± 3.1 (n=150)&#10;">46.3</td>
<td class="num" data-value="56.49999999999999" title="livecodebench_gen: 56.5 ± 3.5 (n=200)&#10;">56.5</td>
<td class="num" data-value="74.4992784992785" title="mmlu_prox_de: 77.3 ± 2.4 (n=300)&#10;include_de_gen: 68.8 ± 3.7 (n=154)&#10;">74.5</td>
<td class="num" data-value="28.488199681723682">28</td>
<td class="num" data-value="19.303084023060574" title="overall 58.5 t/s with 3 parallel requests, from 9 of 9 tasks">19</td>
<td class="num" data-value="898.1292759650371">898</td>
<td class="num" data-value="4480.424041044898">4480</td>
<td data-value="2026-09-21">2026-09-21</td>
</tr>
<tr data-reasoning="off">
<td data-value="mimo-v2.6-flash (pool)">
<a href="runs/31.html" class="strong">mimo-v2.6-flash (pool)</a>
<div class="sub">FP8 + MXFP4 · Reasoning off · direct · 2× NVIDIA DGX Spark, je 128 GB RAM</div>
</td>
<td class="num strong" data-value="59.427507658033974">59.4</td>
<td class="num" data-value="69.69562289562289" title="mmlu_pro: 82.0 ± 2.2 (n=300)&#10;gpqa_diamond_gen: 68.7 ± 3.3 (n=198)&#10;musr_gen: 58.4 ± 3.1 (n=250)&#10;">69.7</td>
<td class="num" data-value="41.6" title="omni_math_gen: 41.6 ± 3.1 (n=250)&#10;">41.6</td>
<td class="num" data-value="49.819548872180455" title="ifbench: 37.0 ± 2.8 (n=300)&#10;ifeval: 88.7 ± 2.6 (n=150)&#10;">49.8</td>
<td class="num" data-value="61.5" title="livecodebench_gen: 61.5 ± 3.4 (n=200)&#10;">61.5</td>
<td class="num" data-value="74.52236652236651" title="mmlu_prox_de: 78.7 ± 2.4 (n=300)&#10;include_de_gen: 66.2 ± 3.8 (n=154)&#10;">74.5</td>
<td class="num" data-value="27.055489674547132">27</td>
<td class="num" data-value="21.7770272387994" title="overall 70.2 t/s with 3 parallel requests, from 9 of 9 tasks">22</td>
<td class="num" data-value="2177.560718839664">2178</td>
<td class="num" data-value="1847.9392860026564">1848</td>
<td data-value="2026-09-26">2026-09-26</td>
</tr>
<tr data-reasoning="off">
<td data-value="gemma-4-12b-it-heretic">
<a href="runs/17.html" class="strong">gemma-4-12b-it-heretic</a>
<div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="57.7166917293233">57.7</td>
<td class="num" data-value="63.70909090909091" title="mmlu_pro: 76.7 ± 2.4 (n=300)&#10;gpqa_diamond_gen: 56.1 ± 3.5 (n=198)&#10;musr_gen: 58.4 ± 3.1 (n=250)&#10;">63.7</td>
<td class="num" data-value="48.4" title="omni_math_gen: 48.4 ± 3.2 (n=250)&#10;">48.4</td>
<td class="num" data-value="46.81203007518797" title="ifbench: 33.0 ± 2.7 (n=300)&#10;ifeval: 88.7 ± 2.6 (n=150)&#10;">46.8</td>
<td class="num" data-value="62.0" title="livecodebench_gen: 62.0 ± 3.4 (n=200)&#10;">62.0</td>
<td class="num" data-value="67.66233766233766" title="mmlu_prox_de: 70.0 ± 2.7 (n=300)&#10;include_de_gen: 63.0 ± 3.9 (n=154)&#10;">67.7</td>
<td class="num" data-value="21.360681311630678">21</td>
<td class="num" data-value="21.121219019103503" title="overall 62.6 t/s with 3 parallel requests, from 5 of 9 tasks">21</td>
<td class="num" data-value="5258.027383235864">5258</td>
<td class="num" data-value="763.404164230451">763</td>
<td data-value="2026-09-18">2026-09-18</td>
</tr>
<tr data-reasoning="off">
<td data-value="nemotron-3.5-lightning-dspark">
<a href="runs/16.html" class="strong">nemotron-3.5-lightning-dspark</a>
<div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="53.13354497354497">53.1</td>
<td class="num" data-value="55.756902356902366" title="mmlu_pro: 71.0 ± 2.6 (n=300)&#10;gpqa_diamond_gen: 57.1 ± 3.5 (n=198)&#10;musr_gen: 39.2 ± 3.1 (n=250)&#10;">55.8</td>
<td class="num" data-value="40.400000000000006" title="omni_math_gen: 40.4 ± 3.1 (n=250)&#10;">40.4</td>
<td class="num" data-value="46.0" title="ifbench: 35.0 ± 2.8 (n=300)&#10;ifeval: 79.3 ± 3.3 (n=150)&#10;">46.0</td>
<td class="num" data-value="53.0" title="livecodebench_gen: 53.0 ± 3.5 (n=200)&#10;">53.0</td>
<td class="num" data-value="70.5108225108225" title="mmlu_prox_de: 72.0 ± 2.6 (n=300)&#10;include_de_gen: 67.5 ± 3.8 (n=154)&#10;">70.5</td>
<td class="num" data-value="83.7975869946743">84</td>
<td class="num" data-value="70.42131680616991" title="overall 213.6 t/s with 3 parallel requests, from 5 of 9 tasks">70</td>
<td class="num" data-value="7236.849600555504">7237</td>
<td class="num" data-value="554.1085170116276">554</td>
<td data-value="2026-09-17">2026-09-17</td>
</tr>
<tr data-reasoning="off">
<td data-value="qwen3.5-9b-heretic-v2-mtp">
<a href="runs/19.html" class="strong">qwen3.5-9b-heretic-v2-mtp</a>
<div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="51.662634870003295">51.7</td>
<td class="num" data-value="65.07138047138046" title="mmlu_pro: 76.3 ± 2.5 (n=300)&#10;gpqa_diamond_gen: 58.1 ± 3.5 (n=198)&#10;musr_gen: 60.8 ± 3.1 (n=250)&#10;">65.1</td>
<td class="num" data-value="35.6" title="omni_math_gen: 35.6 ± 3.0 (n=250)&#10;">35.6</td>
<td class="num" data-value="44.140350877192986" title="ifbench: 29.7 ± 2.6 (n=300)&#10;ifeval: 88.0 ± 2.7 (n=150)&#10;">44.1</td>
<td class="num" data-value="46.5" title="livecodebench_gen: 46.5 ± 3.5 (n=200)&#10;">46.5</td>
<td class="num" data-value="67.00144300144301" title="mmlu_prox_de: 69.3 ± 2.7 (n=300)&#10;include_de_gen: 62.3 ± 3.9 (n=154)&#10;">67.0</td>
<td class="num" data-value="31.530246994028012">32</td>
<td class="num" data-value="35.229241139471675" title="overall 106.0 t/s with 3 parallel requests, from 5 of 9 tasks">35</td>
<td class="num" data-value="6290.925546819461">6291</td>
<td class="num" data-value="639.6515059750527">640</td>
<td data-value="2026-09-18">2026-09-18</td>
</tr>
<tr data-reasoning="off">
<td data-value="zgcm-1-7b">
<a href="runs/29.html" class="strong">zgcm-1-7b</a>
<div class="sub">NVFP4 · Reasoning off · direct · NVIDIA DGX Spark, 128 GB RAM</div>
</td>
<td class="num strong" data-value="33.36777195514038">33.4</td>
<td class="num" data-value="34.68552188552189" title="mmlu_pro: 25.3 ± 2.5 (n=300)&#10;gpqa_diamond_gen: 32.3 ± 3.3 (n=198)&#10;musr_gen: 46.4 ± 3.2 (n=250)&#10;">34.7</td>
<td class="num" data-value="40.0" title="omni_math_gen: 40.0 ± 3.1 (n=250)&#10;">40.0</td>
<td class="num" data-value="44.33082706766917" title="ifbench: 33.0 ± 2.7 (n=300)&#10;ifeval: 78.7 ± 3.4 (n=150)&#10;">44.3</td>
<td class="num" data-value="23.0" title="livecodebench_gen: 23.0 ± 3.0 (n=200)&#10;">23.0</td>
<td class="num" data-value="24.82251082251082" title="mmlu_prox_de: 21.0 ± 2.4 (n=300)&#10;include_de_gen: 32.5 ± 3.8 (n=154)&#10;">24.8</td>
<td class="num" data-value="33.67868964254804">34</td>
<td class="num" data-value="34.0307978654491" title="overall 102.4 t/s with 3 parallel requests, from 9 of 9 tasks">34</td>
<td class="num" data-value="3957.900821130251">3958</td>
<td class="num" data-value="1022.5117260124534">1023</td>
<td data-value="2026-09-23">2026-09-23</td>
</tr>
</tbody>
</table>
</div>
<div class="chart-set">
<script type="application/json" class="chart-data" id="charts-v3-quick">{"categories": [{"key": "overall", "label": "Overall"}, {"key": "knowledge", "label": "Knowledge \u0026 reasoning"}, {"key": "math", "label": "Math"}, {"key": "instructions", "label": "Instruction following"}, {"key": "code", "label": "Code"}, {"key": "german", "label": "German"}], "missing_tps": false, "reasoning_mark": "\ud83d\udcad", "runs": [{"label": "gpt-oss-120b \u00b7 \ud83d\udcad", "run_id": 18, "scores": {"code": 83.0, "german": 77.4, "instructions": 66.1, "knowledge": 68.5, "math": 70.4, "overall": 73.1}, "tps": 19.6, "tps_source": "run"}, {"label": "qwen3.8-flash-next-abliterated (pool)", "run_id": 32, "scores": {"code": 77.0, "german": 81.4, "instructions": 55.2, "knowledge": 77.2, "math": 72.0, "overall": 72.5}, "tps": 37.2, "tps_source": "run"}, {"label": "qwen3.8-flash-next-ple4", "run_id": 15, "scores": {"code": 75.5, "german": 81.1, "instructions": 49.6, "knowledge": 77.9, "math": 73.2, "overall": 71.5}, "tps": 21.6, "tps_source": "run"}, {"label": "gemma-4-12b-it", "run_id": 21, "scores": {"code": 75.0, "german": 74.5, "instructions": 51.3, "knowledge": 67.5, "math": 60.8, "overall": 65.8}, "tps": 13.8, "tps_source": "run"}, {"label": "gemma-4-12b-it-heretic", "run_id": 22, "scores": {"code": 75.0, "german": 76.5, "instructions": 50.0, "knowledge": 68.4, "math": 59.2, "overall": 65.8}, "tps": 13.8, "tps_source": "run"}, {"label": "qwen3-coder-next", "run_id": 30, "scores": {"code": 70.5, "german": 76.0, "instructions": 45.9, "knowledge": 71.3, "math": 61.6, "overall": 65.1}, "tps": 23.8, "tps_source": "run"}, {"label": "qwen3.8-27b-mtp", "run_id": 20, "scores": {"code": 65.0, "german": 76.3, "instructions": 49.5, "knowledge": 69.9, "math": 60.4, "overall": 64.2}, "tps": 22.4, "tps_source": "run"}, {"label": "occamy-1.0", "run_id": 27, "scores": {"code": 63.0, "german": 76.9, "instructions": 44.7, "knowledge": 73.1, "math": 62.4, "overall": 64.0}, "tps": 31.6, "tps_source": "run"}, {"label": "qwen3.8-27b-abliterated", "run_id": 23, "scores": {"code": 59.5, "german": 72.1, "instructions": 46.1, "knowledge": 71.5, "math": 58.0, "overall": 61.4}, "tps": 21.9, "tps_source": "run"}, {"label": "ternary-bonsai-2-27b", "run_id": 26, "scores": {"code": 56.5, "german": 74.5, "instructions": 46.3, "knowledge": 69.5, "math": 59.6, "overall": 61.3}, "tps": 19.3, "tps_source": "run"}, {"label": "mimo-v2.6-flash (pool)", "run_id": 31, "scores": {"code": 61.5, "german": 74.5, "instructions": 49.8, "knowledge": 69.7, "math": 41.6, "overall": 59.4}, "tps": 21.8, "tps_source": "run"}, {"label": "gemma-4-12b-it-heretic", "run_id": 17, "scores": {"code": 62.0, "german": 67.7, "instructions": 46.8, "knowledge": 63.7, "math": 48.4, "overall": 57.7}, "tps": 21.1, "tps_source": "run"}, {"label": "nemotron-3.5-lightning-dspark", "run_id": 16, "scores": {"code": 53.0, "german": 70.5, "instructions": 46.0, "knowledge": 55.8, "math": 40.4, "overall": 53.1}, "tps": 70.4, "tps_source": "run"}, {"label": "qwen3.5-9b-heretic-v2-mtp", "run_id": 19, "scores": {"code": 46.5, "german": 67.0, "instructions": 44.1, "knowledge": 65.1, "math": 35.6, "overall": 51.7}, "tps": 35.2, "tps_source": "run"}, {"label": "zgcm-1-7b", "run_id": 29, "scores": {"code": 23.0, "german": 24.8, "instructions": 44.3, "knowledge": 34.7, "math": 40.0, "overall": 33.4}, "tps": 34.0, "tps_source": "run"}], "texts": {"overall": "Overall", "overall_title": "Overall score", "points": "points", "score_axis": "Score (0\u2013100)", "source_run": "measured during the run", "source_test": "from the speed test", "tps_axis": "Generation (tokens/s)", "vs": " vs. generation (tokens/s)"}}</script>
<p class="segmented" role="group" aria-label="Chart category"><button type="button" data-chart-category="overall" class="active" aria-pressed="true">Overall</button><button type="button" data-chart-category="knowledge" class="" aria-pressed="false">Knowledge &amp; reasoning</button><button type="button" data-chart-category="math" class="" aria-pressed="false">Math</button><button type="button" data-chart-category="instructions" class="" aria-pressed="false">Instruction following</button><button type="button" data-chart-category="code" class="" aria-pressed="false">Code</button><button type="button" data-chart-category="german" class="" aria-pressed="false">German</button></p>
<div class="charts">
<figure class="chart-card">
<figcaption><span data-chart-title="bar">Overall score</span> <span class="sub">(💭 = with reasoning)</span></figcaption>
<div class="chart-box" data-chart-box="bar" style="height: 520px"><canvas data-chart="bar" role="img" aria-label="Score per model variant; the values are also in the table"></canvas></div>
</figure>
<figure class="chart-card">
<figcaption><span data-chart-title="scatter">Overall score vs. generation (tokens/s)</span> <span class="sub">(💭 = with reasoning)</span></figcaption>
<div class="chart-box" style="height: 320px"><canvas data-chart="scatter" role="img" aria-label="Score against generation speed in tokens per second; the values are also in the table"></canvas></div>
<p class="hint">x-axis: generation per request measured during the run; hollow points only have the speed-test value.</p>
</figure>
</div>
</div>
</section>
<p class="hint">Scores run from 0 to 100; the best value per category is shown in bold. Hovering a category cell lists its tasks with standard error (±) and number of items (n). Speed: generation in tokens/s from the speed test and measured during the run (gen t/s run, per request), prompt processing at 4k context in tokens/s, time to first token at 4k context in ms. Click a column header to sort.</p>
<footer class="page-foot">As of 2026-10-10 · scores 0–100 · built with <code>bench publish</code></footer>
</main>
<script src="chart.umd.min.js"></script>
<script src="app.js"></script>
</body>
</html>