Weekly run 2026-09-28: 1, 25, 100 concurrent users
#2
by Infergauge - opened
Weekly run, 2026-09-28. Same harness, same 20 prompts, one provider pinned per model, no client-side retries. Levels measured sequentially, never in parallel.
| Concurrency | Models >=90% goodput | Models with errors | TTFT p95 range |
|---|---|---|---|
| 1 | 6 / 13 | 1 | 0.39-5.72s (15x) |
| 25 | 5 / 13 | 4 | 0.37-6.68s (18x) |
| 100 | 5 / 13 | 5 | 0.38-20.65s (55x) |
Notable:
- At 1: worst error rate was TinyLlama/TinyLlama-1.1B-Chat-v1.0 at 1.7%.
- At 25: worst error rate was meta-llama/Llama-3.2-3B-Instruct at 25.0%.
- At 100: worst error rate was google/gemma-3-1b-it at 60.0%.
Goodput counts a request only if it met every threshold at once: first token under 2s, finished under 15s, inter-token latency p95 under 80ms, format score at or above 0.8, no error. Thresholds are in slo.json.
These are shared endpoints carrying load nobody here controls, and each figure is 60 requests. Week-to-week movement of a few points is noise, not signal. The per-run CSVs are archived under runs/ so you can check the trend yourself.
Full matrix and method: https://huggingface.co/spaces/Infergauge-load/llm-performance-and-load