---
license: mit
tags:
- llm-serving
- capacity-planning
- performance-model
- roofline
- queueing
- simulation
library_name: custom
datasets:
- thanatchon36/one-gpu-capacity
---
# One GPU Capacity: calibrated step-time and capacity model
This repository holds the **calibrated parameters** of the analytical model in *How Many Users Can One GPU Serve?*
(Thanatchon Chongmankhong, November 2026). It holds **no neural-network weights**: the "model" is a token roofline
for one engine iteration, a birth–death model of a closed population of users, and an iteration-level simulator
of continuous batching with paged KV memory, whose few efficiency parameters were fitted to public benchmarks.
- 📖 Paper online:
- 💻 Code (simulator in C, Python reference):
- 📊 Dataset with every result: [`datasets/thanatchon36/one-gpu-capacity`](https://huggingface.co/datasets/thanatchon36/one-gpu-capacity)
## Fitted step-time parameters
Step time of one iteration = t₀ + a·(sequences) + smooth-max(bytes / (η_β·bandwidth), FLOPs / (η_c·compute)).
| engine | η_β (bandwidth eff.) | η_c (compute eff.) | t₀ (ms) | a (µs/seq) | rows | leave-one-GPU-out MAPE |
|---|---:|---:|---:|---:|---:|---:|
| vLLM | 0.77 | 0.61 | 2.2 | 81 | 188 | 10.4% |
| TensorRT-LLM | 0.78 | 0.56 | 1.4 | 19 | 184 | 8.7% |
Fitted jointly across GPUs and models on single-GPU batch latencies from LLM-Inference-Bench (A100 40 GB, H100 SXM,
GH200; four 7–8B models). Compute efficiency is weakly identified, because those benchmarks are mostly
bandwidth-bound; the paper propagates its uncertainty with a cluster bootstrap.
## Validation
| layer | test | result |
|---|---|---|
| V1 | simulator vs exact queueing results | all identities within the 95% CI |
| V2 | leave-one-GPU-out batch latency | MAPE 10.4% (vLLM), 8.7% (TensorRT-LLM) |
| V3 | consumer-card decode speed, fitted on datacenter cards only | Spearman ρ = 0.99 |
| V4 | 48 MLPerf Inference systems, after the calibration was frozen | 90% within 0.67–1.5× |
## Files
- `calibration.json`: every fit (Vidur operator profiles, functional forms, engine-level fits, llama.cpp transfer)
- `gpus.csv`, `models.csv`: the hardware and architecture catalogues the model is evaluated on
## Use
```bash
git clone https://github.com/thanatchon36/one-gpu-capacity && cd one-gpu-capacity
pip install -r requirements.txt
PYTHONPATH=src python -c "from ogc import capacity; help(capacity)"
```
## Limits
No GPU was measured. Predictions outside the calibrated range (other engines, speculative decoding, multi-GPU
tensor parallelism, heavy KV oversubscription) are extrapolations; see §12 of the paper.
## Citation
```bibtex
@misc{chongmankhong2026onegpu,
title = {How Many Users Can One GPU Serve? A Capacity Law for LLM Inference from Consumer Cards to Datacenter Accelerators},
author = {Chongmankhong, Thanatchon},
year = {2026},
note = {Preprint},
doi = {10.5281/zenodo.24243858},
url = {https://github.com/thanatchon36/one-gpu-capacity}
}
```