--- license: mit tags: - llm-serving - capacity-planning - performance-model - roofline - queueing - simulation library_name: custom datasets: - thanatchon36/one-gpu-capacity --- # One GPU Capacity: calibrated step-time and capacity model This repository holds the **calibrated parameters** of the analytical model in *How Many Users Can One GPU Serve?* (Thanatchon Chongmankhong, November 2026). It holds **no neural-network weights**: the "model" is a token roofline for one engine iteration, a birth–death model of a closed population of users, and an iteration-level simulator of continuous batching with paged KV memory, whose few efficiency parameters were fitted to public benchmarks. - 📖 Paper online: - 💻 Code (simulator in C, Python reference): - 📊 Dataset with every result: [`datasets/thanatchon36/one-gpu-capacity`](https://huggingface.co/datasets/thanatchon36/one-gpu-capacity) ## Fitted step-time parameters Step time of one iteration = t₀ + a·(sequences) + smooth-max(bytes / (η_β·bandwidth), FLOPs / (η_c·compute)). | engine | η_β (bandwidth eff.) | η_c (compute eff.) | t₀ (ms) | a (µs/seq) | rows | leave-one-GPU-out MAPE | |---|---:|---:|---:|---:|---:|---:| | vLLM | 0.77 | 0.61 | 2.2 | 81 | 188 | 10.4% | | TensorRT-LLM | 0.78 | 0.56 | 1.4 | 19 | 184 | 8.7% | Fitted jointly across GPUs and models on single-GPU batch latencies from LLM-Inference-Bench (A100 40 GB, H100 SXM, GH200; four 7–8B models). Compute efficiency is weakly identified, because those benchmarks are mostly bandwidth-bound; the paper propagates its uncertainty with a cluster bootstrap. ## Validation | layer | test | result | |---|---|---| | V1 | simulator vs exact queueing results | all identities within the 95% CI | | V2 | leave-one-GPU-out batch latency | MAPE 10.4% (vLLM), 8.7% (TensorRT-LLM) | | V3 | consumer-card decode speed, fitted on datacenter cards only | Spearman ρ = 0.99 | | V4 | 48 MLPerf Inference systems, after the calibration was frozen | 90% within 0.67–1.5× | ## Files - `calibration.json`: every fit (Vidur operator profiles, functional forms, engine-level fits, llama.cpp transfer) - `gpus.csv`, `models.csv`: the hardware and architecture catalogues the model is evaluated on ## Use ```bash git clone https://github.com/thanatchon36/one-gpu-capacity && cd one-gpu-capacity pip install -r requirements.txt PYTHONPATH=src python -c "from ogc import capacity; help(capacity)" ``` ## Limits No GPU was measured. Predictions outside the calibrated range (other engines, speculative decoding, multi-GPU tensor parallelism, heavy KV oversubscription) are extrapolations; see §12 of the paper. ## Citation ```bibtex @misc{chongmankhong2026onegpu, title = {How Many Users Can One GPU Serve? A Capacity Law for LLM Inference from Consumer Cards to Datacenter Accelerators}, author = {Chongmankhong, Thanatchon}, year = {2026}, note = {Preprint}, doi = {10.5281/zenodo.24243858}, url = {https://github.com/thanatchon36/one-gpu-capacity} } ```