Match the grid to the InferGauge report: ms, report wording, P95 latency, throughput, cost/request
2138cbd verified Download article.html from Infergauge-load/llm-performance-and-load: direct link, hf CLI and curl.
- Browser
- Download file 24.2 kB
-
https://huggingface.co/spaces/Infergauge-load/llm-performance-and-load/resolve/main/article.html
- Command line
-
hf download hf://spaces/Infergauge-load/llm-performance-and-load/article.html
-
curl -L -o article.html https://huggingface.co/spaces/Infergauge-load/llm-performance-and-load/resolve/main/article.html
24.2 kB
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>What 22 open models do at 1, 25 and 100 concurrent users</title> | |
| <meta name="description" content="Twenty-two open models on Hugging Face Inference Providers, benchmarked at 1, 25 and 100 concurrent users: goodput, inter-token latency, cost, what a provider does with your settings, and how much the numbers move between runs."> | |
| <style> | |
| :root{ | |
| --bg:#ffffff; --panel:#f7f8fa; --line:#e3e6ea; --ink:#12161c; --muted:#5b6673; | |
| --accent:#2a78d6; --accent-soft:#e8efff; | |
| --warn:#8a6d1f; --warn-soft:#fdf5e0; | |
| --bad:#d03b3b; --bad-soft:#fbe0e0; | |
| --good:#0ca30c; | |
| --bar:#9db8f5; --mono:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace; | |
| } | |
| @media (prefers-color-scheme:dark){ | |
| :root:not([data-theme="light"]){ | |
| --bg:#0e1116; --panel:#161b22; --line:#262d38; --ink:#e7ebf0; --muted:#98a3b3; | |
| --accent:#3987e5; --accent-soft:#182236; | |
| --warn:#e3c26b; --warn-soft:#241f12; | |
| --bad:#e66767; --bad-soft:#3f1d1d; | |
| --good:#0ca30c; | |
| --bar:#31487f; | |
| } | |
| } | |
| *{box-sizing:border-box} | |
| html,body{margin:0;padding:0} | |
| body{ | |
| background:var(--bg); color:var(--ink); | |
| font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",Inter,Roboto,Helvetica,Arial,sans-serif; | |
| -webkit-font-smoothing:antialiased; | |
| } | |
| .wrap{max-width:1120px;margin:0 auto;padding:48px 16px 80px} | |
| header h1{font-size:clamp(28px,4vw,40px);line-height:1.15;margin:0 0 10px;letter-spacing:-.02em} | |
| header p.sub{font-size:18px;color:var(--muted);margin:0 0 24px;max-width:64ch} | |
| .cta{display:flex;gap:14px;align-items:center;flex-wrap:wrap;margin:0 0 28px} | |
| .btn{ | |
| display:inline-block;background:var(--accent);color:#fff;text-decoration:none; | |
| font-weight:600;font-size:14.5px;padding:9px 16px;border-radius:8px;white-space:nowrap; | |
| } | |
| .btn:hover{filter:brightness(1.08)} | |
| .cta code{font-size:13.5px;padding:8px 12px} | |
| .cta-note{font-size:13.5px;color:var(--muted)} | |
| .finding{ | |
| background:var(--panel);border:1px solid var(--line);border-left:3px solid var(--accent); | |
| border-radius:8px;padding:18px 20px;margin:0 0 20px; | |
| } | |
| .finding h2{margin:0 0 8px;font-size:19px;letter-spacing:-.01em} | |
| .finding p{margin:0 0 8px;font-size:15.5px;max-width:72ch} | |
| .finding p:last-child{margin:0} | |
| .note{ | |
| background:var(--warn-soft);border:1px solid var(--line);border-left:3px solid var(--warn); | |
| border-radius:8px;padding:14px 16px;margin:0 0 32px;font-size:14.5px; | |
| } | |
| .note strong{color:var(--warn)} | |
| nav.tabs{display:flex;gap:4px;border-bottom:1px solid var(--line);margin-bottom:24px;flex-wrap:wrap} | |
| nav.tabs button{ | |
| appearance:none;background:none;border:0;border-bottom:2px solid transparent; | |
| color:var(--muted);font:inherit;font-size:15px;font-weight:500; | |
| padding:10px 14px;cursor:pointer;margin-bottom:-1px; | |
| } | |
| nav.tabs button[aria-selected="true"]{color:var(--ink);border-bottom-color:var(--accent)} | |
| nav.tabs button:hover{color:var(--ink)} | |
| section[hidden]{display:none} | |
| .controls{display:flex;gap:18px;flex-wrap:wrap;align-items:center;margin-bottom:16px;font-size:14px} | |
| .controls label{display:inline-flex;gap:6px;align-items:center;color:var(--muted);cursor:pointer} | |
| .controls input{accent-color:var(--accent)} | |
| .views{display:inline-flex;gap:4px;align-items:center;color:var(--muted)} | |
| .views button{font:inherit;font-size:13px;color:var(--muted);background:transparent; | |
| border:1px solid var(--line);border-radius:6px;padding:3px 10px;cursor:pointer} | |
| .views button.on{color:var(--fg);border-color:var(--accent)} | |
| .empty{background:var(--panel);border:1px dashed var(--line);border-radius:10px;padding:28px 24px} | |
| .empty h2{margin:0 0 10px;font-size:19px} | |
| .empty p{margin:0 0 12px;color:var(--muted);font-size:15px;max-width:68ch} | |
| .tablewrap{overflow-x:auto;border:1px solid var(--line);border-radius:10px} | |
| table{border-collapse:collapse;width:100%;font-size:14px} | |
| th,td{padding:9px 12px;text-align:left;border-bottom:1px solid var(--line);white-space:nowrap} | |
| thead tr.groups th{ | |
| background:var(--panel);font-size:11.5px;letter-spacing:.06em;text-transform:uppercase; | |
| color:var(--muted);border-bottom:1px solid var(--line);text-align:center;font-weight:700; | |
| } | |
| thead tr.groups th.spacer{background:var(--panel)} | |
| th{background:var(--panel);font-weight:600;font-size:12.5px;letter-spacing:.03em; | |
| text-transform:uppercase;color:var(--muted);cursor:pointer;user-select:none} | |
| th[data-dir]:after{content:"";margin-left:5px;opacity:.7} | |
| th[data-dir="asc"]:after{content:"\2191"} | |
| th[data-dir="desc"]:after{content:"\2193"} | |
| td.sep,th.sep{border-left:1px solid var(--line)} | |
| tbody tr:last-child td{border-bottom:0} | |
| tbody tr:hover{background:var(--accent-soft)} | |
| td.num{font-variant-numeric:tabular-nums;font-family:var(--mono);font-size:13px} | |
| td.model{font-family:var(--mono);font-size:12.5px} | |
| .pill{display:inline-block;padding:1px 7px;border-radius:99px;font-size:11.5px; | |
| background:var(--accent-soft);color:var(--accent);font-weight:600} | |
| .delta-bad{color:var(--bad);font-weight:700} | |
| .delta-ok{color:var(--muted)} | |
| .err-bad{color:var(--bad);font-weight:700} | |
| tr.collapsed td{background:var(--bad-soft)} | |
| tr.collapsed td:first-child{box-shadow:inset 3px 0 0 var(--bad)} | |
| tr.collapsed:hover td{filter:brightness(1.12)} | |
| .meta{font-size:13.5px;color:var(--muted);margin-top:14px;max-width:80ch} | |
| .prose h2{font-size:21px;margin:34px 0 10px;letter-spacing:-.01em} | |
| .prose h3{font-size:16px;margin:26px 0 8px} | |
| .prose p,.prose li{font-size:15.5px} | |
| .prose p,.prose ul{max-width:72ch} | |
| .prose blockquote{margin:16px 0;padding:12px 16px;border-left:3px solid var(--accent); | |
| background:var(--panel);border-radius:0 8px 8px 0} | |
| .prose blockquote p{margin:0 0 8px} | |
| .prose blockquote p:last-child{margin:0} | |
| pre{background:var(--panel);border:1px solid var(--line);border-radius:8px; | |
| padding:14px 16px;overflow-x:auto;font-family:var(--mono);font-size:13.5px;line-height:1.55} | |
| code{font-family:var(--mono);font-size:.92em;background:var(--panel); | |
| padding:1px 5px;border-radius:4px;border:1px solid var(--line)} | |
| pre code{background:none;border:0;padding:0} | |
| a{color:var(--accent)} | |
| footer{margin-top:56px;padding-top:20px;border-top:1px solid var(--line); | |
| font-size:13.5px;color:var(--muted)} | |
| footer a{margin-right:14px} | |
| .backlink{font-size:14px;margin:0 0 20px} | |
| .backlink a{color:var(--muted);text-decoration:none} | |
| .backlink a:hover{color:var(--accent)} | |
| article{max-width:76ch} | |
| article h2{font-size:23px;margin:36px 0 12px;letter-spacing:-.01em; | |
| padding-top:20px;border-top:1px solid var(--line)} | |
| article h3{font-size:17.5px;margin:28px 0 10px;letter-spacing:-.01em} | |
| article p{font-size:16px;line-height:1.62;margin:0 0 14px} | |
| article ul{font-size:16px;line-height:1.62;padding-left:22px} | |
| article li{margin:0 0 6px} | |
| article .tablewrap{margin:18px 0 8px;max-width:100%} | |
| article table th,article table td{padding:9px 12px} | |
| article .meta{margin:0 0 18px} | |
| </style> | |
| </head> | |
| <body> | |
| <div class="wrap"> | |
| <p class="backlink"><a href="./index.html">← Leaderboard</a></p> | |
| <header> | |
| <h1>What 22 open models do at 1, 25 and 100 concurrent users</h1> | |
| <p class="sub">Measured 5 October 2026. 3,960 requests across three levels of load, one | |
| provider pinned per model, no client-side retries.</p> | |
| </header> | |
| <div class="note"> | |
| <strong>Disclosure.</strong> I work on InferGauge, the tool used to produce these measurements. | |
| The harness, the prompt set and the raw per-request results are published so the numbers can be | |
| checked independently. | |
| </div> | |
| <article> | |
| <p>Most published inference numbers are single-stream. One request at a time, best case, and often | |
| without stating the concurrency at all. That tells you what a model can do. It doesn't tell you | |
| what happens when your app has users.</p> | |
| <p>So I ran the same prompt set against twenty-two open models on Hugging Face Inference Providers | |
| at three levels of load: 1, 25 and 100 concurrent requests per model. Same harness, same prompts, | |
| three runs, sequential. 3,960 requests.</p> | |
| <p>The board grew from thirteen models to twenty-two on 5 October, adding nine models released in | |
| 2026. Most of them reason by default, and that turned out to matter more than their size.</p> | |
| <p>The leaderboard is on <a href="./index.html">the first tab of this Space</a>. Every row reads | |
| across the three levels.</p> | |
| <h2>Method</h2> | |
| <p>Putting this first, because it decides the numbers.</p> | |
| <p><strong>One provider pinned per model.</strong> The same model id can be served by up to ten | |
| providers at different quantisation, hardware and price. An unpinned benchmark measures the router, | |
| not the model. Every row names the provider it used.</p> | |
| <p><strong>No client-side retries.</strong> The OpenAI SDK retries by default, which quietly deletes | |
| your 429s. A 429 or a 5xx is recorded as an error.</p> | |
| <p><strong>TTFT measured on a streamed response</strong>, not derived from total time. Throughput is | |
| measured over the generation window after the first token, because a whole-request rate flatters | |
| models with slow queueing.</p> | |
| <p><strong>Inter-token latency measured separately.</strong> The gap between streamed content | |
| chunks, reported as a p95. High TTFT with healthy ITL means requests are queueing. Healthy TTFT | |
| with rising ITL means generation is compute-bound. Reporting only one of them hides which problem | |
| you have.</p> | |
| <p><strong>Goodput is the headline metric.</strong> A request counts only if it meets <em>every</em> | |
| threshold at once: first token under 2s, finished under 15s, ITL p95 under 80ms, format score at or | |
| above 0.8, no error. Those thresholds describe a responsive chat application. They're an editorial | |
| choice, not a law, which is why they're published in <code>slo.json</code> rather than buried.</p> | |
| <p><strong>Quality is deterministic format checking</strong>, not a judge model. Does the JSON parse | |
| with the required keys and types. Does the code compile and define the requested function. Does the | |
| summary respect the stated sentence limit. Graded partial credit: valid JSON wrapped in prose scores | |
| 0.8, code that parses but defines the wrong function scores 0.4. An optional LLM-as-judge rating is | |
| recorded in its own column and never blended in.</p> | |
| <p><strong>Outliers kept.</strong> Cold starts and queueing spikes are part of what a caller | |
| experiences.</p> | |
| <p><strong>Reasoning suppressed where the model allows it.</strong> Six of the nine 2026 models | |
| reason by default. Left alone they spend the whole token budget thinking and answer nothing, which | |
| would be a harness artefact rather than a finding. So the run asks them not to: | |
| <code>chat_template_kwargs.enable_thinking = false</code> for the Qwen 3.5/3.6/3.8 generation and | |
| granite-4.2, <code>reasoning_effort: "low"</code> for the GLM and gpt-oss pairs. Whether the | |
| provider honours that request is itself one of the results below.</p> | |
| <p><strong>Latency is quoted in milliseconds</strong>, the same unit an InferGauge report | |
| uses, except where a figure runs past ten seconds and milliseconds stop being readable.</p> | |
| <p><strong>MoE models list active parameters separately.</strong> gpt-oss-120b is 117B total and | |
| 5.1B active. Charting it against dense models by total size produces a wrong chart.</p> | |
| <h2>Results</h2> | |
| <h3>16 of 22 models never failed a single request</h3> | |
| <div class="tablewrap"><table> | |
| <thead><tr><th>Model</th><th>Provider</th><th class="num">Goodput 1 / 25 / 100</th><th class="num">p95 TTFT, ms — 1 / 25 / 100</th></tr></thead> | |
| <tbody> | |
| <tr><td class="model">gpt-oss-20b</td><td>groq</td><td class="num">100 / 100 / 100</td><td class="num">420 / 580 / 411</td></tr> | |
| <tr><td class="model">Qwen3-Coder-30B-A3B</td><td>scaleway</td><td class="num">100 / 100 / 100</td><td class="num">535 / 556 / 462</td></tr> | |
| <tr><td class="model">Llama-3.1-8B</td><td>nscale</td><td class="num">95 / 95 / 95</td><td class="num">701 / 698 / 722</td></tr> | |
| <tr><td class="model">Qwen3-8B</td><td>nscale</td><td class="num">95 / 95 / 95</td><td class="num">710 / 716 / 723</td></tr> | |
| <tr><td class="model">gpt-oss-120b</td><td>groq</td><td class="num">83.3 / 85 / 85</td><td class="num">406 / 394 / 491</td></tr> | |
| <tr><td class="model">Qwen2.5-Coder-7B</td><td>nscale</td><td class="num">85 / 85 / 85</td><td class="num">707 / 680 / 706</td></tr> | |
| </tbody></table></div> | |
| <p class="meta">Two models met every threshold on every request at all three levels. gpt-oss-120b | |
| holds a p95 first token between 394 ms and 491 ms from one caller to a hundred. Note what the first | |
| column does <em>not</em> predict: sixteen models returned no errors at any level, and only four of | |
| them clear 90% goodput at a hundred.</p> | |
| <h3>Failures concentrate under load, and the spread explodes</h3> | |
| <div class="tablewrap"><table> | |
| <thead><tr><th>Concurrency</th><th class="num">Models ≥90% goodput</th><th class="num">Models with errors</th><th class="num">p95 TTFT, fastest to slowest</th></tr></thead> | |
| <tbody> | |
| <tr><td class="num">1</td><td class="num">6 / 22</td><td class="num">1</td><td class="num">0.41–5.75s (14x)</td></tr> | |
| <tr><td class="num">25</td><td class="num">5 / 22</td><td class="num">4</td><td class="num">0.39–22.42s (57x)</td></tr> | |
| <tr><td class="num">100</td><td class="num">4 / 22</td><td class="num">6</td><td class="num">0.41–192.23s (468x)</td></tr> | |
| </tbody></table></div> | |
| <p>The fastest row barely moves across the whole range. Everything that changes happens at the | |
| other end. At a hundred callers GLM-5.3 posts a <strong>192-second</strong> p95 first token — | |
| not an error, not a timeout, just a request that is still waiting after three minutes. That one row | |
| is most of the 468x.</p> | |
| <p>If you only ever benchmark single-stream, your data contains almost no failure signal at all.</p> | |
| <h3>A provider can honour your settings, ignore them, or refuse them</h3> | |
| <p>This is the finding I did not expect, and it is invisible in any latency number.</p> | |
| <p>Six models were sent the same request: don't reason, just answer. Three providers did three | |
| different things with it.</p> | |
| <div class="tablewrap"><table> | |
| <thead><tr><th>Provider</th><th>What it did with <code>enable_thinking: false</code></th><th>How you find out</th></tr></thead> | |
| <tbody> | |
| <tr><td>DeepInfra</td><td>Honoured it</td><td>Short answers, normal goodput</td></tr> | |
| <tr><td>Scaleway</td><td>Silently ignored it</td><td>Nothing in the response says so</td></tr> | |
| <tr><td>ovhcloud</td><td>Refused the request outright</td><td>HTTP 400, 60 of 60 failed</td></tr> | |
| </tbody></table></div> | |
| <p>ovhcloud is the easy case. It returns | |
| <code>400 — 'feature extra arguments: {"chat_template_kwargs":{"enable_thinking":false}} is | |
| not currently supported'</code> and Qwen3.5-9B fails every request at every level. Loud, immediate, | |
| impossible to miss.</p> | |
| <p>Scaleway is the dangerous one. Qwen3.6-35B-A3B returns <strong>159 output tokens against a | |
| 60-token prompt, identically, on every single request at all three levels</strong> — the | |
| model reasoning away its entire budget and never reaching an answer. No error. No warning. A p95 | |
| first token of 654–975 ms, which looks excellent. Goodput of exactly zero.</p> | |
| <p>If you are tuning a reasoning model through a router, assume nothing about whether your | |
| parameters arrived. Check the output length.</p> | |
| <h3>The model that looks healthy and isn't</h3> | |
| <p>Two rows score zero goodput while returning no errors at all.</p> | |
| <p>Qwen3.6-35B-A3B is the one above — the ignored flag. <strong>DeepSeek-R1 on Novita</strong> | |
| is the other, and it fails for its own reasons: 1.7%, 0% and 0% goodput across the three levels, | |
| with a p95 first token that never exceeds 2,217 ms and zero errors anywhere.</p> | |
| <p>What fails is everything between those two metrics. Inter-token latency sits around 305–319ms | |
| p95 against an 80ms threshold, so the text visibly stutters. Output runs to 154–332 tokens | |
| against a 52-token prompt, pushing most requests past the 15s total-time ceiling. Format score sits | |
| between 0.36 and 0.41.</p> | |
| <p>This is the clearest argument I have for publishing goodput rather than a latency number. Both | |
| failures are real, both are invisible to latency and error rate, and one of them is a 36B model | |
| that a conventional benchmark would have called the fastest thing on the board.</p> | |
| <h3>Cost runs backwards from quality</h3> | |
| <div class="tablewrap"><table> | |
| <thead><tr><th>Model</th><th>Provider</th><th class="num">$/1M output</th><th class="num">Goodput at 100</th></tr></thead> | |
| <tbody> | |
| <tr><td class="model">Qwen2.5-Coder-7B</td><td>nscale</td><td class="num">0.03</td><td class="num">85</td></tr> | |
| <tr><td class="model">Llama-3.1-8B</td><td>nscale</td><td class="num">0.06</td><td class="num">95</td></tr> | |
| <tr><td class="model">Qwen3-8B</td><td>nscale</td><td class="num">0.18</td><td class="num">95</td></tr> | |
| <tr><td class="model">granite-4.2-8b</td><td>deepinfra</td><td class="num">0.25</td><td class="num">78.3</td></tr> | |
| <tr><td class="model">gpt-oss-20b</td><td>groq</td><td class="num">0.50</td><td class="num">100</td></tr> | |
| <tr><td class="model">gpt-oss-120b</td><td>groq</td><td class="num">0.75</td><td class="num">85</td></tr> | |
| <tr><td class="model">Qwen3-Coder-30B-A3B</td><td>scaleway</td><td class="num">0.912</td><td class="num">100</td></tr> | |
| <tr><td class="model">MiniMax-M3</td><td>deepinfra</td><td class="num">1.10</td><td class="num">3.3</td></tr> | |
| <tr><td class="model">Qwen3.6-35B-A3B</td><td>scaleway</td><td class="num">1.71</td><td class="num">0</td></tr> | |
| <tr><td class="model">DeepSeek-R1</td><td>novita</td><td class="num">2.50</td><td class="num">0</td></tr> | |
| <tr><td class="model">GLM-5.3</td><td>deepinfra</td><td class="num">4.00</td><td class="num">0</td></tr> | |
| </tbody></table></div> | |
| <p>The three most expensive rows on the board all score zero goodput at a hundred callers. The | |
| cheapest holds 85% throughout, at roughly a hundred and thirtieth of the price. The best row on the | |
| whole board costs $0.50.</p> | |
| <p>Rates come from the Hugging Face router's own catalogue, which is what these runs were billed. | |
| They can differ from a provider's public pricing page: Groq lists gpt-oss-20b at $0.30 per million | |
| output tokens, while the router billed $0.50. Six models are served by Featherless, which bills per | |
| subscription rather than per token, so they have no per-1M rate and their cost cell is blank rather | |
| than estimated.</p> | |
| <h3>Throughput tracks the provider, not the model size</h3> | |
| <div class="tablewrap"><table> | |
| <thead><tr><th>Provider</th><th class="num">Tok/s at one caller</th><th class="num">Rows</th></tr></thead> | |
| <tbody> | |
| <tr><td>groq</td><td class="num">482–498</td><td class="num">2</td></tr> | |
| <tr><td>nscale</td><td class="num">186–209</td><td class="num">3</td></tr> | |
| <tr><td>scaleway</td><td class="num">131–203</td><td class="num">2</td></tr> | |
| <tr><td>featherless-ai</td><td class="num">39–99</td><td class="num">6</td></tr> | |
| <tr><td>deepinfra</td><td class="num">37–89</td><td class="num">5</td></tr> | |
| <tr><td>novita</td><td class="num">16–45</td><td class="num">3</td></tr> | |
| </tbody></table></div> | |
| <p>A 117B model on Groq runs about thirty times faster than a 321B model on Novita, and about ten | |
| times faster than a 0.6B model on Featherless. Pick the provider before you pick the parameter | |
| count.</p> | |
| <h2>How much these numbers move</h2> | |
| <p>Before trusting any of the above, it is worth knowing how much of it is noise. On 25 September I | |
| ran the 1-user level twice, two hours apart, with an identical harness against the same endpoints. | |
| The two runs disagree by more than rounding.</p> | |
| <div class="tablewrap"><table> | |
| <thead><tr><th></th><th class="num">First run</th><th class="num">Second run</th></tr></thead> | |
| <tbody> | |
| <tr><td>Models clearing 90% goodput</td><td class="num">4</td><td class="num">6</td></tr> | |
| <tr><td>DeepSeek-R1 goodput</td><td class="num">1.7%</td><td class="num">13.3%</td></tr> | |
| <tr><td>TTFT spread at 1 / 25 / 100</td><td class="num">10x / 38x / 27x</td><td class="num">8x / 40x / 48x</td></tr> | |
| </tbody></table></div> | |
| <p>The spread moved in opposite directions at the top end between the two sets. That was a | |
| thirteen-model board, but nothing about the board changes the point.</p> | |
| <p>Shared inference endpoints are noisy neighbours by construction. You're measuring a service whose | |
| load you don't control, and each per-model figure here is 60 requests. Treat any single number as | |
| one sample.</p> | |
| <p>So read the findings above by how much movement they could survive. A model that fails every | |
| request on an HTTP 400 is not noise. A provider returning 159 identical tokens on all 180 requests | |
| is not noise. A 192-second p95 is not noise. A three-point difference in one model's goodput is, | |
| and shouldn't be quoted.</p> | |
| <p>I'd rather publish this than have someone else discover it by re-running me.</p> | |
| <h2>What this does not measure</h2> | |
| <p>Everything here is provider-side, from a datacenter, with a fixed prompt set. It describes how | |
| the provider behaves, not how your application will.</p> | |
| <p>Your network path isn't in it. Your prompt distribution isn't in it. Your real traffic shape | |
| isn't in it, and neither are your rate limits. Percentiles over sixty requests are thin.</p> | |
| <p>Two more honest limits. The prompt set caps output at 16 tokens on two of its twenty prompts, | |
| which starves a reasoning model — that was a footnote when one row reasoned and is systematic | |
| now that most of them do. And TTFT stops at the first token of any kind, including the first token | |
| of a chain of thought, so it flatters a model that thinks out loud. A time-to-first-<em>answer</em> | |
| column lands in the next run.</p> | |
| <p>If you want numbers that describe your own system, run the harness against your own endpoint.</p> | |
| <h2>Reproduce it</h2> | |
| <p>Everything needed is in this Space:</p> | |
| <ul> | |
| <li><a href="./leaderboard-c1.csv">leaderboard-c1.csv</a>, | |
| <a href="./leaderboard-c25.csv">-c25.csv</a>, | |
| <a href="./leaderboard-c100.csv">-c100.csv</a> — one row per model per level</li> | |
| <li><a href="./results/results.csv">results/results.csv</a> — one row per request</li> | |
| <li><a href="./runs/manifest.json">runs/manifest.json</a> — every weekly run, archived under its own date</li> | |
| <li><a href="./prompts.json">prompts.json</a> — the 20 prompts and their checks</li> | |
| <li><a href="./quality.py">quality.py</a> — the scoring code, so you can disagree with it</li> | |
| <li><a href="./slo.json">slo.json</a> — the thresholds that decide goodput</li> | |
| <li><a href="./model_shortlist.csv">model_shortlist.csv</a> — the models, their pinned | |
| providers, and the reasoning controls each one is sent</li> | |
| <li><a href="./job.py">job.py</a> and <a href="./RUN.md">RUN.md</a> — the runner, and how to run it on HF Jobs</li> | |
| </ul> | |
| <p>Disagree with the SLO thresholds and the goodput column changes. That's the point of publishing | |
| them.</p> | |
| <p>Two licence notes, since this set is no longer uniformly permissive. <strong>MiniMax-M3 is | |
| non-commercial by default</strong> under the MiniMax Community Licence: commercial use needs a | |
| visible "Built with MiniMax M3" credit and prior written authorisation above $20M revenue. | |
| <strong>GLM-5.3</strong> is MIT-like, with a security review required only of Model-as-a-Service | |
| operators above $10B revenue. Everything else on the board is Apache-2.0, MIT, or the Llama and | |
| Gemma community licences.</p> | |
| <p>The tool is InferGauge, source-available under BSL 1.1: you can read it, run it internally and in | |
| production, and gate CI with it. You can't resell it as a hosted service. It converts to Apache-2.0 | |
| in 2030. <a href="https://www.infergauge.com" target="_blank" rel="noopener">infergauge.com</a></p> | |
| <p><a href="https://console.infergauge.com/" target="_blank" rel="noopener"><strong>Run a free test | |
| with 25 users →</strong></a> — the middle column of this table, against your own | |
| endpoint.</p> | |
| </article> | |
| <p class="backlink"><a href="./index.html">← Back to the leaderboard</a></p> | |
| </div> | |
| </body> | |
| </html> | |