What 22 open models do at 1, 25 and 100 concurrent users

Most published inference numbers are single-stream. One request at a time, best case, and often without stating the concurrency at all. That tells you what a model can do. It doesn't tell you what happens when your app has users.

So I ran the same prompt set against twenty-two open models on Hugging Face Inference Providers at three levels of load: 1, 25 and 100 concurrent requests per model. Same harness, same prompts, three runs, sequential. 3,960 requests.

The board grew from thirteen models to twenty-two on 5 October, adding nine models released in 2026. Most of them reason by default, and that turned out to matter more than their size.

The leaderboard is on the first tab of this Space. Every row reads across the three levels.

Method

Putting this first, because it decides the numbers.

One provider pinned per model. The same model id can be served by up to ten providers at different quantisation, hardware and price. An unpinned benchmark measures the router, not the model. Every row names the provider it used.

No client-side retries. The OpenAI SDK retries by default, which quietly deletes your 429s. A 429 or a 5xx is recorded as an error.

TTFT measured on a streamed response, not derived from total time. Throughput is measured over the generation window after the first token, because a whole-request rate flatters models with slow queueing.

Inter-token latency measured separately. The gap between streamed content chunks, reported as a p95. High TTFT with healthy ITL means requests are queueing. Healthy TTFT with rising ITL means generation is compute-bound. Reporting only one of them hides which problem you have.

Goodput is the headline metric. A request counts only if it meets every threshold at once: first token under 2s, finished under 15s, ITL p95 under 80ms, format score at or above 0.8, no error. Those thresholds describe a responsive chat application. They're an editorial choice, not a law, which is why they're published in slo.json rather than buried.

Quality is deterministic format checking, not a judge model. Does the JSON parse with the required keys and types. Does the code compile and define the requested function. Does the summary respect the stated sentence limit. Graded partial credit: valid JSON wrapped in prose scores 0.8, code that parses but defines the wrong function scores 0.4. An optional LLM-as-judge rating is recorded in its own column and never blended in.

Outliers kept. Cold starts and queueing spikes are part of what a caller experiences.

Reasoning suppressed where the model allows it. Six of the nine 2026 models reason by default. Left alone they spend the whole token budget thinking and answer nothing, which would be a harness artefact rather than a finding. So the run asks them not to: chat_template_kwargs.enable_thinking = false for the Qwen 3.5/3.6/3.8 generation and granite-4.2, reasoning_effort: "low" for the GLM and gpt-oss pairs. Whether the provider honours that request is itself one of the results below.

Latency is quoted in milliseconds, the same unit an InferGauge report uses, except where a figure runs past ten seconds and milliseconds stop being readable.

MoE models list active parameters separately. gpt-oss-120b is 117B total and 5.1B active. Charting it against dense models by total size produces a wrong chart.

Results

16 of 22 models never failed a single request

Two models met every threshold on every request at all three levels. gpt-oss-120b holds a p95 first token between 394 ms and 491 ms from one caller to a hundred. Note what the first column does not predict: sixteen models returned no errors at any level, and only four of them clear 90% goodput at a hundred.

Failures concentrate under load, and the spread explodes

The fastest row barely moves across the whole range. Everything that changes happens at the other end. At a hundred callers GLM-5.3 posts a 192-second p95 first token — not an error, not a timeout, just a request that is still waiting after three minutes. That one row is most of the 468x.

If you only ever benchmark single-stream, your data contains almost no failure signal at all.

A provider can honour your settings, ignore them, or refuse them

This is the finding I did not expect, and it is invisible in any latency number.

Six models were sent the same request: don't reason, just answer. Three providers did three different things with it.

ovhcloud is the easy case. It returns 400 — 'feature extra arguments: {"chat_template_kwargs":{"enable_thinking":false}} is not currently supported' and Qwen3.5-9B fails every request at every level. Loud, immediate, impossible to miss.

Scaleway is the dangerous one. Qwen3.6-35B-A3B returns 159 output tokens against a 60-token prompt, identically, on every single request at all three levels — the model reasoning away its entire budget and never reaching an answer. No error. No warning. A p95 first token of 654–975 ms, which looks excellent. Goodput of exactly zero.

If you are tuning a reasoning model through a router, assume nothing about whether your parameters arrived. Check the output length.

The model that looks healthy and isn't

Two rows score zero goodput while returning no errors at all.

Qwen3.6-35B-A3B is the one above — the ignored flag. DeepSeek-R1 on Novita is the other, and it fails for its own reasons: 1.7%, 0% and 0% goodput across the three levels, with a p95 first token that never exceeds 2,217 ms and zero errors anywhere.

What fails is everything between those two metrics. Inter-token latency sits around 305–319ms p95 against an 80ms threshold, so the text visibly stutters. Output runs to 154–332 tokens against a 52-token prompt, pushing most requests past the 15s total-time ceiling. Format score sits between 0.36 and 0.41.

This is the clearest argument I have for publishing goodput rather than a latency number. Both failures are real, both are invisible to latency and error rate, and one of them is a 36B model that a conventional benchmark would have called the fastest thing on the board.

Cost runs backwards from quality

The three most expensive rows on the board all score zero goodput at a hundred callers. The cheapest holds 85% throughout, at roughly a hundred and thirtieth of the price. The best row on the whole board costs $0.50.

Rates come from the Hugging Face router's own catalogue, which is what these runs were billed. They can differ from a provider's public pricing page: Groq lists gpt-oss-20b at $0.30 per million output tokens, while the router billed $0.50. Six models are served by Featherless, which bills per subscription rather than per token, so they have no per-1M rate and their cost cell is blank rather than estimated.

Throughput tracks the provider, not the model size

A 117B model on Groq runs about thirty times faster than a 321B model on Novita, and about ten times faster than a 0.6B model on Featherless. Pick the provider before you pick the parameter count.

How much these numbers move

Before trusting any of the above, it is worth knowing how much of it is noise. On 25 September I ran the 1-user level twice, two hours apart, with an identical harness against the same endpoints. The two runs disagree by more than rounding.

The spread moved in opposite directions at the top end between the two sets. That was a thirteen-model board, but nothing about the board changes the point.

Shared inference endpoints are noisy neighbours by construction. You're measuring a service whose load you don't control, and each per-model figure here is 60 requests. Treat any single number as one sample.

So read the findings above by how much movement they could survive. A model that fails every request on an HTTP 400 is not noise. A provider returning 159 identical tokens on all 180 requests is not noise. A 192-second p95 is not noise. A three-point difference in one model's goodput is, and shouldn't be quoted.

I'd rather publish this than have someone else discover it by re-running me.

What this does not measure

Everything here is provider-side, from a datacenter, with a fixed prompt set. It describes how the provider behaves, not how your application will.

Your network path isn't in it. Your prompt distribution isn't in it. Your real traffic shape isn't in it, and neither are your rate limits. Percentiles over sixty requests are thin.

Two more honest limits. The prompt set caps output at 16 tokens on two of its twenty prompts, which starves a reasoning model — that was a footnote when one row reasoned and is systematic now that most of them do. And TTFT stops at the first token of any kind, including the first token of a chain of thought, so it flatters a model that thinks out loud. A time-to-first-answer column lands in the next run.

If you want numbers that describe your own system, run the harness against your own endpoint.

Reproduce it

Everything needed is in this Space:

• leaderboard-c1.csv, -c25.csv, -c100.csv — one row per model per level
• results/results.csv — one row per request
• runs/manifest.json — every weekly run, archived under its own date
• prompts.json — the 20 prompts and their checks
• quality.py — the scoring code, so you can disagree with it
• slo.json — the thresholds that decide goodput
• model_shortlist.csv — the models, their pinned providers, and the reasoning controls each one is sent
• job.py and RUN.md — the runner, and how to run it on HF Jobs

Disagree with the SLO thresholds and the goodput column changes. That's the point of publishing them.

Two licence notes, since this set is no longer uniformly permissive. MiniMax-M3 is non-commercial by default under the MiniMax Community Licence: commercial use needs a visible "Built with MiniMax M3" credit and prior written authorisation above $20M revenue. GLM-5.3 is MIT-like, with a security review required only of Model-as-a-Service operators above $10B revenue. Everything else on the board is Apache-2.0, MIT, or the Llama and Gemma community licences.

The tool is InferGauge, source-available under BSL 1.1: you can read it, run it internally and in production, and gate CI with it. You can't resell it as a hosted service. It converts to Apache-2.0 in 2030. infergauge.com

Run a free test with 25 users → — the middle column of this table, against your own endpoint.