|
Download README.md from thanatchon36/one-gpu-capacity: direct link, hf CLI and curl.
- Browser
- Download file 3.14 kB
-
https://huggingface.co/thanatchon36/one-gpu-capacity/resolve/main/README.md
- Command line
-
hf download hf://thanatchon36/one-gpu-capacity/README.md
-
curl -L -o README.md https://huggingface.co/thanatchon36/one-gpu-capacity/resolve/main/README.md
3.14 kB
| license: mit | |
| tags: | |
| - llm-serving | |
| - capacity-planning | |
| - performance-model | |
| - roofline | |
| - queueing | |
| - simulation | |
| library_name: custom | |
| datasets: | |
| - thanatchon36/one-gpu-capacity | |
| # One GPU Capacity: calibrated step-time and capacity model | |
| This repository holds the **calibrated parameters** of the analytical model in *How Many Users Can One GPU Serve?* | |
| (Thanatchon Chongmankhong, November 2026). It holds **no neural-network weights**: the "model" is a token roofline | |
| for one engine iteration, a birth–death model of a closed population of users, and an iteration-level simulator | |
| of continuous batching with paged KV memory, whose few efficiency parameters were fitted to public benchmarks. | |
| - 📖 Paper online: <https://thanatchon36.github.io/one-gpu-capacity/paper/> | |
| - 💻 Code (simulator in C, Python reference): <https://github.com/thanatchon36/one-gpu-capacity> | |
| - 📊 Dataset with every result: [`datasets/thanatchon36/one-gpu-capacity`](https://huggingface.co/datasets/thanatchon36/one-gpu-capacity) | |
| ## Fitted step-time parameters | |
| Step time of one iteration = t₀ + a·(sequences) + smooth-max(bytes / (η_β·bandwidth), FLOPs / (η_c·compute)). | |
| | engine | η_β (bandwidth eff.) | η_c (compute eff.) | t₀ (ms) | a (µs/seq) | rows | leave-one-GPU-out MAPE | | |
| |---|---:|---:|---:|---:|---:|---:| | |
| | vLLM | 0.77 | 0.61 | 2.2 | 81 | 188 | 10.4% | | |
| | TensorRT-LLM | 0.78 | 0.56 | 1.4 | 19 | 184 | 8.7% | | |
| Fitted jointly across GPUs and models on single-GPU batch latencies from LLM-Inference-Bench (A100 40 GB, H100 SXM, | |
| GH200; four 7–8B models). Compute efficiency is weakly identified, because those benchmarks are mostly | |
| bandwidth-bound; the paper propagates its uncertainty with a cluster bootstrap. | |
| ## Validation | |
| | layer | test | result | | |
| |---|---|---| | |
| | V1 | simulator vs exact queueing results | all identities within the 95% CI | | |
| | V2 | leave-one-GPU-out batch latency | MAPE 10.4% (vLLM), 8.7% (TensorRT-LLM) | | |
| | V3 | consumer-card decode speed, fitted on datacenter cards only | Spearman ρ = 0.99 | | |
| | V4 | 48 MLPerf Inference systems, after the calibration was frozen | 90% within 0.67–1.5× | | |
| ## Files | |
| - `calibration.json`: every fit (Vidur operator profiles, functional forms, engine-level fits, llama.cpp transfer) | |
| - `gpus.csv`, `models.csv`: the hardware and architecture catalogues the model is evaluated on | |
| ## Use | |
| ```bash | |
| git clone https://github.com/thanatchon36/one-gpu-capacity && cd one-gpu-capacity | |
| pip install -r requirements.txt | |
| PYTHONPATH=src python -c "from ogc import capacity; help(capacity)" | |
| ``` | |
| ## Limits | |
| No GPU was measured. Predictions outside the calibrated range (other engines, speculative decoding, multi-GPU | |
| tensor parallelism, heavy KV oversubscription) are extrapolations; see §12 of the paper. | |
| ## Citation | |
| ```bibtex | |
| @misc{chongmankhong2026onegpu, | |
| title = {How Many Users Can One GPU Serve? A Capacity Law for LLM Inference from Consumer Cards to Datacenter Accelerators}, | |
| author = {Chongmankhong, Thanatchon}, | |
| year = {2026}, | |
| note = {Preprint}, | |
| doi = {10.5281/zenodo.24243858}, | |
| url = {https://github.com/thanatchon36/one-gpu-capacity} | |
| } | |
| ``` | |