Hugging Face's logo

Spaces:
Infergauge-load
/
llm-performance-and-load
Running

App Files Community
6

Weekly run 2026-10-05: 1, 25, 100 concurrent users

#4
by Infergauge - opened
Discussion
Infergauge
 InferGauge org Hugging Face CLI

Weekly run, 2026-10-05. Same harness, same 20 prompts, one provider pinned per model, no client-side retries. Levels measured sequentially, never in parallel.

Concurrency Models >=90% goodput Models with errors TTFT p95 range
1 6 / 22 1 0.41-5.75s (14x)
25 5 / 22 4 0.39-22.42s (57x)
100 4 / 22 6 0.41-192.23s (468x)

Notable:

  • At 1: worst error rate was Qwen/Qwen3.5-9B at 100.0%.
  • At 25: worst error rate was Qwen/Qwen3.5-9B at 100.0%.
  • At 100: worst error rate was Qwen/Qwen3.5-9B at 100.0%.

Per-model figures, all 3 levels. @1 means at one concurrent user, @100 at 100.

Overview
Model Class Provider $/1M out TTFT p95 @1 Err % @1 TTFT p95 @25 Err % @25 TTFT p95 @100 Err % @100 Goodput % @100 ฮ” p95 1โ†’100
openai/gpt-oss-20b large groq 0.50 0.42 0.0 0.58 0.0 0.41 0.0 100 -2%
Qwen/Qwen3-Coder-30B-A3B-Instruct large scaleway 0.91 0.54 0.0 0.56 0.0 0.46 0.0 100 -14%
openai/gpt-oss-120b large groq 0.75 0.41 0.0 0.39 0.0 0.49 0.0 85 +21%
Qwen/Qwen2.5-Coder-7B-Instruct mid nscale 0.03 0.71 0.0 0.68 0.0 0.71 0.0 85 -0%
meta-llama/Llama-3.1-8B-Instruct mid nscale 0.06 0.70 0.0 0.70 0.0 0.72 0.0 95 +3%
Qwen/Qwen3-8B mid nscale 0.18 0.71 0.0 0.72 0.0 0.72 0.0 95 +2%
ibm-granite/granite-4.2-8b mid deepinfra 0.25 0.58 0.0 0.62 0.0 0.79 0.0 78 +35%
Qwen/Qwen3.6-35B-A3B large scaleway 1.71 0.91 0.0 0.65 0.0 0.97 0.0 0 +8%
Qwen/Qwen3.8-27B large deepinfra 2.50 0.82 0.0 1.51 0.0 0.98 0.0 82 +19%
deepseek-ai/DeepSeek-V4.1-Flash xlarge deepinfra 0.60 0.69 0.0 1.02 0.0 1.13 0.0 75 +64%
google/gemma-4-26B-A4B-it large novita 0.40 1.25 0.0 1.25 0.0 1.52 0.0 58 +22%
deepseek-ai/DeepSeek-R1 xlarge novita 2.50 2.22 0.0 1.81 0.0 2.01 0.0 0 -9%
zai-org/GLM-5.3-Flash xlarge novita 0.50 5.75 0.0 6.39 0.0 2.31 0.0 42 -60%
TinyLlama/TinyLlama-1.1B-Chat-v1.0 small featherless-ai โ€” 1.92 0.0 11.00 73.3 3.27 5.0 28 +70%
meta-llama/Llama-3.2-3B-Instruct small featherless-ai โ€” 4.41 0.0 4.06 0.0 4.25 0.0 40 -3%
Qwen/Qwen2.5-7B-Instruct mid featherless-ai โ€” 3.25 0.0 1.25 0.0 4.83 6.7 62 +48%
Qwen/Qwen3-4B small featherless-ai โ€” 1.58 0.0 11.26 83.3 12.26 46.7 35 +674%
google/gemma-3-1b-it small featherless-ai โ€” 5.30 0.0 4.63 5.0 22.38 76.7 3 +323%
MiniMaxAI/MiniMax-M3 xlarge deepinfra 1.10 1.61 0.0 1.11 0.0 22.60 0.0 3 +1300%
Qwen/Qwen3-0.6B small featherless-ai โ€” 2.46 0.0 3.87 0.0 31.39 11.7 12 +1174%
zai-org/GLM-5.3 xlarge deepinfra 4.00 3.34 0.0 22.42 0.0 192.23 0.0 0 +5654%
Qwen/Qwen3.5-9B mid ovhcloud 0.18 โ€” 100.0 โ€” 100.0 โ€” 100.0 0 โ€”
Latency
Model Provider TTFT p95 @1 ITL p95 ms @1 Tok/s @1 TTFT p95 @25 ITL p95 ms @25 Tok/s @25 TTFT p95 @100 ITL p95 ms @100 Tok/s @100
openai/gpt-oss-20b groq 0.42 4 498 0.58 14 692 0.41 21 630
Qwen/Qwen3-Coder-30B-A3B-Instruct scaleway 0.54 37 203 0.56 38 206 0.46 38 202
openai/gpt-oss-120b groq 0.41 14 482 0.39 20 482 0.49 20 486
Qwen/Qwen2.5-Coder-7B-Instruct nscale 0.71 10 209 0.68 7 203 0.71 7 200
meta-llama/Llama-3.1-8B-Instruct nscale 0.70 9 186 0.70 8 184 0.72 8 184
Qwen/Qwen3-8B nscale 0.71 35 192 0.72 36 191 0.72 36 191
ibm-granite/granite-4.2-8b deepinfra 0.58 29 56 0.62 68 37 0.79 105 38
Qwen/Qwen3.6-35B-A3B scaleway 0.91 โ€” 131 0.65 โ€” 128 0.97 โ€” 60
Qwen/Qwen3.8-27B deepinfra 0.82 156 56 1.51 92 49 0.98 162 47
deepseek-ai/DeepSeek-V4.1-Flash deepinfra 0.69 224 89 1.02 189 90 1.13 239 101
google/gemma-4-26B-A4B-it novita 1.25 109 45 1.25 115 32 1.52 169 26
deepseek-ai/DeepSeek-R1 novita 2.22 305 25 1.81 312 20 2.01 319 23
zai-org/GLM-5.3-Flash novita 5.75 213 16 6.39 236 16 2.31 180 35
TinyLlama/TinyLlama-1.1B-Chat-v1.0 featherless-ai 1.92 14 99 11.00 9 25 3.27 60 82
meta-llama/Llama-3.2-3B-Instruct featherless-ai 4.41 28 45 4.06 46 38 4.25 43 21
Qwen/Qwen2.5-7B-Instruct featherless-ai 3.25 51 39 1.25 44 40 4.83 53 43
Qwen/Qwen3-4B featherless-ai 1.58 54 42 11.26 10 5 12.26 66 32
google/gemma-3-1b-it featherless-ai 5.30 17 43 4.63 12 43 22.38 39 4
MiniMaxAI/MiniMax-M3 deepinfra 1.61 257 37 1.11 172 54 22.60 695 10
Qwen/Qwen3-0.6B featherless-ai 2.46 38 51 3.87 48 46 31.39 43 4
zai-org/GLM-5.3 deepinfra 3.34 376 80 22.42 921 19 192.23 1101 0
Qwen/Qwen3.5-9B ovhcloud โ€” โ€” โ€” โ€” โ€” โ€” โ€” โ€” โ€”
Quality
Model Class Licence Quality @1 Goodput % @1 Err % @1 Quality @25 Goodput % @25 Err % @25 Quality @100 Goodput % @100 Err % @100
openai/gpt-oss-20b large apache-2.0 1.00 100 0.0 1.00 100 0.0 1.00 100 0.0
Qwen/Qwen3-Coder-30B-A3B-Instruct large apache-2.0 1.00 100 0.0 1.00 100 0.0 1.00 100 0.0
openai/gpt-oss-120b large apache-2.0 0.85 83 0.0 0.85 85 0.0 0.85 85 0.0
Qwen/Qwen2.5-Coder-7B-Instruct mid apache-2.0 0.89 85 0.0 0.89 85 0.0 0.89 85 0.0
meta-llama/Llama-3.1-8B-Instruct mid llama3.1 0.95 95 0.0 0.95 95 0.0 0.95 95 0.0
Qwen/Qwen3-8B mid apache-2.0 0.95 95 0.0 0.95 95 0.0 0.95 95 0.0
ibm-granite/granite-4.2-8b mid apache-2.0 0.88 83 0.0 0.83 78 0.0 0.90 78 0.0
Qwen/Qwen3.6-35B-A3B large apache-2.0 0.00 0 0.0 0.00 0 0.0 0.00 0 0.0
Qwen/Qwen3.8-27B large apache-2.0 1.00 85 0.0 0.97 77 0.0 1.00 82 0.0
deepseek-ai/DeepSeek-V4.1-Flash xlarge mit 0.97 75 0.0 1.00 73 0.0 0.97 75 0.0
google/gemma-4-26B-A4B-it large apache-2.0 0.97 90 0.0 1.00 82 0.0 0.95 58 0.0
deepseek-ai/DeepSeek-R1 xlarge mit 0.41 2 0.0 0.38 0 0.0 0.36 0 0.0
zai-org/GLM-5.3-Flash xlarge mit 0.98 27 0.0 1.00 17 0.0 0.99 42 0.0
TinyLlama/TinyLlama-1.1B-Chat-v1.0 small apache-2.0 0.54 35 0.0 0.54 0 73.3 0.55 28 5.0
meta-llama/Llama-3.2-3B-Instruct small llama3.2 0.95 80 0.0 0.93 60 0.0 0.84 40 0.0
Qwen/Qwen2.5-7B-Instruct mid apache-2.0 0.95 85 0.0 0.95 93 0.0 0.94 62 6.7
Qwen/Qwen3-4B small apache-2.0 0.95 93 0.0 1.00 2 83.3 0.91 35 46.7
google/gemma-3-1b-it small gemma 0.95 68 0.0 0.94 57 5.0 0.90 3 76.7
MiniMaxAI/MiniMax-M3 xlarge other 0.70 55 0.0 0.71 65 0.0 0.70 3 0.0
Qwen/Qwen3-0.6B small apache-2.0 0.95 82 0.0 0.95 80 0.0 0.93 12 11.7
zai-org/GLM-5.3 xlarge other 0.96 63 0.0 0.97 32 0.0 0.98 0 0.0
Qwen/Qwen3.5-9B mid apache-2.0 โ€” 0 100.0 โ€” 0 100.0 โ€” 0 100.0
Tokens & cost
Model Provider $/1M out In tok @1 Out tok @1 Tok/s @1 In tok @25 Out tok @25 Tok/s @25 In tok @100 Out tok @100 Tok/s @100
openai/gpt-oss-20b groq 0.50 119 89 498 119 89 692 119 89 630
Qwen/Qwen3-Coder-30B-A3B-Instruct scaleway 0.91 58 49 203 58 49 206 58 49 202
openai/gpt-oss-120b groq 0.75 119 92 482 119 93 482 119 93 486
Qwen/Qwen2.5-Coder-7B-Instruct nscale 0.03 79 52 209 79 51 203 79 52 200
meta-llama/Llama-3.1-8B-Instruct nscale 0.06 83 55 186 83 54 184 83 54 184
Qwen/Qwen3-8B nscale 0.18 61 48 192 61 48 191 61 48 191
ibm-granite/granite-4.2-8b deepinfra 0.25 65 75 56 65 80 37 65 77 38
Qwen/Qwen3.6-35B-A3B scaleway 1.71 60 159 131 60 159 128 60 159 60
Qwen/Qwen3.8-27B deepinfra 2.50 62 52 56 62 51 49 62 50 47
deepseek-ai/DeepSeek-V4.1-Flash deepinfra 0.60 53 45 89 53 45 90 53 44 101
google/gemma-4-26B-A4B-it novita 0.40 64 52 45 64 55 32 64 51 26
deepseek-ai/DeepSeek-R1 novita 2.50 52 332 25 52 154 20 52 154 23
zai-org/GLM-5.3-Flash novita 0.50 60 56 16 60 54 16 60 55 35
TinyLlama/TinyLlama-1.1B-Chat-v1.0 featherless-ai โ€” 70 120 99 70 154 25 70 125 82
meta-llama/Llama-3.2-3B-Instruct featherless-ai โ€” 82 58 45 80 56 38 64 70 21
Qwen/Qwen2.5-7B-Instruct featherless-ai โ€” 79 54 39 79 53 40 76 53 43
Qwen/Qwen3-4B featherless-ai โ€” 61 48 42 62 43 5 53 43 32
google/gemma-3-1b-it featherless-ai โ€” 60 56 43 60 58 43 45 38 4
MiniMaxAI/MiniMax-M3 deepinfra 1.10 224 175 37 224 191 54 224 182 10
Qwen/Qwen3-0.6B featherless-ai โ€” 61 53 51 61 53 46 62 52 4
zai-org/GLM-5.3 deepinfra 4.00 60 52 80 60 51 19 60 54 0
Qwen/Qwen3.5-9B ovhcloud 0.18 โ€” โ€” โ€” โ€” โ€” โ€” โ€” โ€” โ€”

Goodput counts a request only if it met every threshold at once: first token under 2s, finished under 15s, inter-token latency p95 under 80ms, format score at or above 0.8, no error. Thresholds are in slo.json.

These are shared endpoints carrying load nobody here controls, and each figure is 60 requests. Week-to-week movement of a few points is noise, not signal. The per-run CSVs are archived under runs/ so you can check the trend yourself.

Full matrix and method: https://huggingface.co/spaces/Infergauge-load/llm-performance-and-load

Sign up or log in to comment