thanatchon36 commited on
Commit
0890af4
·
verified ·
1 Parent(s): f07aa27

Publish model card

Browse files
Files changed (4) hide show
  1. README.md +76 -0
  2. calibration.json +1919 -0
  3. gpus.csv +18 -0
  4. models.csv +10 -0
README.md ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ tags:
4
+ - llm-serving
5
+ - capacity-planning
6
+ - performance-model
7
+ - roofline
8
+ - queueing
9
+ - simulation
10
+ library_name: custom
11
+ datasets:
12
+ - thanatchon36/one-gpu-capacity
13
+ ---
14
+
15
+ # One GPU Capacity: calibrated step-time and capacity model
16
+
17
+ This repository holds the **calibrated parameters** of the analytical model in *How Many Users Can One GPU Serve?*
18
+ (Thanatchon Chongmankhong, November 2026). It holds **no neural-network weights**: the "model" is a token roofline
19
+ for one engine iteration, a birth–death model of a closed population of users, and an iteration-level simulator
20
+ of continuous batching with paged KV memory, whose few efficiency parameters were fitted to public benchmarks.
21
+
22
+ - 📖 Paper online: <https://thanatchon36.github.io/one-gpu-capacity/paper/>
23
+ - 💻 Code (simulator in C, Python reference): <https://github.com/thanatchon36/one-gpu-capacity>
24
+ - 📊 Dataset with every result: [`datasets/thanatchon36/one-gpu-capacity`](https://huggingface.co/datasets/thanatchon36/one-gpu-capacity)
25
+
26
+ ## Fitted step-time parameters
27
+
28
+ Step time of one iteration = t₀ + a·(sequences) + smooth-max(bytes / (η_β·bandwidth), FLOPs / (η_c·compute)).
29
+
30
+ | engine | η_β (bandwidth eff.) | η_c (compute eff.) | t₀ (ms) | a (µs/seq) | rows | leave-one-GPU-out MAPE |
31
+ |---|---:|---:|---:|---:|---:|---:|
32
+ | vLLM | 0.77 | 0.61 | 2.2 | 81 | 188 | 10.4% |
33
+ | TensorRT-LLM | 0.78 | 0.56 | 1.4 | 19 | 184 | 8.7% |
34
+
35
+ Fitted jointly across GPUs and models on single-GPU batch latencies from LLM-Inference-Bench (A100 40 GB, H100 SXM,
36
+ GH200; four 7–8B models). Compute efficiency is weakly identified, because those benchmarks are mostly
37
+ bandwidth-bound; the paper propagates its uncertainty with a cluster bootstrap.
38
+
39
+ ## Validation
40
+
41
+ | layer | test | result |
42
+ |---|---|---|
43
+ | V1 | simulator vs exact queueing results | all identities within the 95% CI |
44
+ | V2 | leave-one-GPU-out batch latency | MAPE 10.4% (vLLM), 8.7% (TensorRT-LLM) |
45
+ | V3 | consumer-card decode speed, fitted on datacenter cards only | Spearman ρ = 0.99 |
46
+ | V4 | 48 MLPerf Inference systems, after the calibration was frozen | 90% within 0.67–1.5× |
47
+
48
+ ## Files
49
+
50
+ - `calibration.json`: every fit (Vidur operator profiles, functional forms, engine-level fits, llama.cpp transfer)
51
+ - `gpus.csv`, `models.csv`: the hardware and architecture catalogues the model is evaluated on
52
+
53
+ ## Use
54
+
55
+ ```bash
56
+ git clone https://github.com/thanatchon36/one-gpu-capacity && cd one-gpu-capacity
57
+ pip install -r requirements.txt
58
+ PYTHONPATH=src python -c "from ogc import capacity; help(capacity)"
59
+ ```
60
+
61
+ ## Limits
62
+
63
+ No GPU was measured. Predictions outside the calibrated range (other engines, speculative decoding, multi-GPU
64
+ tensor parallelism, heavy KV oversubscription) are extrapolations; see §12 of the paper.
65
+
66
+ ## Citation
67
+
68
+ ```bibtex
69
+ @misc{chongmankhong2026onegpu,
70
+ title = {How Many Users Can One GPU Serve? A Capacity Law for LLM Inference from Consumer Cards to Datacenter Accelerators},
71
+ author = {Chongmankhong, Thanatchon},
72
+ year = {2026},
73
+ note = {Preprint},
74
+ url = {https://github.com/thanatchon36/one-gpu-capacity}
75
+ }
76
+ ```
calibration.json ADDED
@@ -0,0 +1,1919 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "vidur": {
3
+ "max": {
4
+ "a100": {
5
+ "eta_bw": 0.7310774976236001,
6
+ "eta_c": 0.6654064678288377,
7
+ "mape": 0.057558084397660726,
8
+ "n": 140,
9
+ "attn_bw_frac": 0.5991907381812237
10
+ },
11
+ "a40": {
12
+ "eta_bw": 0.7536243228217148,
13
+ "eta_c": 0.6,
14
+ "mape": 0.05581423683930286,
15
+ "n": 64,
16
+ "attn_bw_frac": 0.7680605528836582
17
+ },
18
+ "h100": {
19
+ "eta_bw": 0.8114088587438508,
20
+ "eta_c": 0.6,
21
+ "mape": 0.064630345092196,
22
+ "n": 64,
23
+ "attn_bw_frac": 0.7094333446982612
24
+ }
25
+ },
26
+ "sum": {
27
+ "a100": {
28
+ "eta_bw": 0.8154750554470435,
29
+ "eta_c": 1.2,
30
+ "mape": 0.06514696945554,
31
+ "n": 140,
32
+ "attn_bw_frac": 0.5991907381812237
33
+ },
34
+ "a40": {
35
+ "eta_bw": 0.8202894751055775,
36
+ "eta_c": 1.2,
37
+ "mape": 0.045614744672416406,
38
+ "n": 64,
39
+ "attn_bw_frac": 0.7680605528836582
40
+ },
41
+ "h100": {
42
+ "eta_bw": 0.8671763154735798,
43
+ "eta_c": 1.199999999999998,
44
+ "mape": 0.06913968586498931,
45
+ "n": 64,
46
+ "attn_bw_frac": 0.7094333446982612
47
+ }
48
+ },
49
+ "smooth": {
50
+ "a100": {
51
+ "eta_bw": 0.7500135866949069,
52
+ "eta_c": 0.6454558433375267,
53
+ "mape": 0.03124893839795496,
54
+ "n": 140,
55
+ "attn_bw_frac": 0.5991907381812237
56
+ },
57
+ "a40": {
58
+ "eta_bw": 0.7675289933621684,
59
+ "eta_c": 0.496990333497917,
60
+ "mape": 0.04068866140227775,
61
+ "n": 64,
62
+ "attn_bw_frac": 0.7680605528836582
63
+ },
64
+ "h100": {
65
+ "eta_bw": 0.8219516543325195,
66
+ "eta_c": 0.43142716896141786,
67
+ "mape": 0.06512007416540311,
68
+ "n": 64,
69
+ "attn_bw_frac": 0.7094333446982612
70
+ }
71
+ }
72
+ },
73
+ "forms": {
74
+ "max": {
75
+ "mape_logo": 0.1050090307005881,
76
+ "mape_lomo": 0.07892129707615032,
77
+ "mape_in": 0.062107935683446164
78
+ },
79
+ "sum": {
80
+ "mape_logo": 0.10471184600439247,
81
+ "mape_lomo": 0.07636385614385872,
82
+ "mape_in": 0.062130243653889514
83
+ },
84
+ "smooth": {
85
+ "mape_logo": 0.09579984855265163,
86
+ "mape_lomo": 0.07895041723090473,
87
+ "mape_in": 0.062217436246071
88
+ }
89
+ },
90
+ "chosen_form": "smooth",
91
+ "system": {
92
+ "vLLM": {
93
+ "stack": {
94
+ "name": "vLLM",
95
+ "eta_bw": 0.7744444064711272,
96
+ "eta_c": 0.6078207723971903,
97
+ "t0": 0.00224528080712071,
98
+ "a": 8.05735049544692e-05,
99
+ "combine": "smooth",
100
+ "k": 4.0,
101
+ "compute_mult": 1.0
102
+ },
103
+ "n": 188,
104
+ "mape_in": 0.06090956456365651,
105
+ "median_ape_in": 0.03841512643929196,
106
+ "mape_logo": 0.10408696257168513,
107
+ "per_logo": {
108
+ "Nvidia A100 GPU": 0.17154376693971685,
109
+ "Nvidia GH200 GPU": 0.05919784316579272,
110
+ "Nvidia H100 GPU": 0.09292727708869816
111
+ },
112
+ "mape_lomo": 0.07562555735366443,
113
+ "per_lomo": {
114
+ "llama2-7b": 0.0837342216690668,
115
+ "llama31-8b": 0.05565613538345523,
116
+ "mistral-7b": 0.10507827738524675,
117
+ "qwen2-7b": 0.06622822233467722
118
+ },
119
+ "boot": [
120
+ [
121
+ 0.8353708264730914,
122
+ 0.47105157215421306,
123
+ 0.0027255484484701832,
124
+ 6.170605122120394e-05
125
+ ],
126
+ [
127
+ 0.8560417085047962,
128
+ 0.47256250640760716,
129
+ 0.002961409476108579,
130
+ 6.0721319163930246e-05
131
+ ],
132
+ [
133
+ 0.7192319561221389,
134
+ 0.9999999999999767,
135
+ 0.001763079940410119,
136
+ 0.0001033392349283213
137
+ ],
138
+ [
139
+ 0.6732147229666561,
140
+ 0.7499969451943471,
141
+ 0.0015078635543028958,
142
+ 8.77581078651131e-05
143
+ ],
144
+ [
145
+ 0.7187804293985574,
146
+ 0.6709995183143378,
147
+ 0.00193480600409867,
148
+ 7.802649296399806e-05
149
+ ],
150
+ [
151
+ 0.6336665233933497,
152
+ 0.9999999999999203,
153
+ 0.001000113501736386,
154
+ 0.00010146864654072308
155
+ ],
156
+ [
157
+ 0.762585915760647,
158
+ 0.7304400278917808,
159
+ 0.0020895845707670675,
160
+ 8.277739669134237e-05
161
+ ],
162
+ [
163
+ 0.7637152296831576,
164
+ 0.5510780691373647,
165
+ 0.0021197671051368526,
166
+ 8.221404141596497e-05
167
+ ],
168
+ [
169
+ 0.8057501240953764,
170
+ 0.43555030428069275,
171
+ 0.0024921521654544887,
172
+ 5.704186190180971e-05
173
+ ],
174
+ [
175
+ 0.7953730920952746,
176
+ 0.7505219748400214,
177
+ 0.0022670700475717924,
178
+ 8.176009474276736e-05
179
+ ],
180
+ [
181
+ 0.762408366890393,
182
+ 1.0,
183
+ 0.0018911328362419529,
184
+ 0.00012324202310205502
185
+ ],
186
+ [
187
+ 0.8394238578831958,
188
+ 0.6337529071861983,
189
+ 0.0027684042093055923,
190
+ 7.21363356895421e-05
191
+ ],
192
+ [
193
+ 0.7845454745649316,
194
+ 0.9999999999999997,
195
+ 0.002226954989729916,
196
+ 9.343958838078387e-05
197
+ ],
198
+ [
199
+ 0.7460343115476997,
200
+ 0.9999999999999916,
201
+ 0.0019186310567728057,
202
+ 0.00010155208638303724
203
+ ],
204
+ [
205
+ 0.8124863414651788,
206
+ 0.618531884685702,
207
+ 0.002482982728969926,
208
+ 8.881419000108848e-05
209
+ ],
210
+ [
211
+ 0.7240993082499051,
212
+ 0.45273853142594567,
213
+ 0.0017701806621178843,
214
+ 6.595019320135386e-05
215
+ ],
216
+ [
217
+ 0.7437229377466746,
218
+ 0.39850170399872864,
219
+ 0.0022475527577927377,
220
+ 5.3515794506564e-05
221
+ ],
222
+ [
223
+ 0.7752307832959314,
224
+ 0.9999999999999993,
225
+ 0.002245687453103287,
226
+ 8.886954308417157e-05
227
+ ],
228
+ [
229
+ 0.7795571635310605,
230
+ 0.5286024949660089,
231
+ 0.0022372891319340053,
232
+ 7.872142605615057e-05
233
+ ],
234
+ [
235
+ 0.7680527664305951,
236
+ 0.6099180212644428,
237
+ 0.0022950893175969008,
238
+ 7.572534932209138e-05
239
+ ],
240
+ [
241
+ 0.761410225168744,
242
+ 0.6272120926483548,
243
+ 0.001993807502278899,
244
+ 7.762453840081338e-05
245
+ ],
246
+ [
247
+ 0.8622161899149313,
248
+ 0.49811478146613464,
249
+ 0.0030328311383799058,
250
+ 6.796722329839808e-05
251
+ ],
252
+ [
253
+ 0.7446388399982375,
254
+ 0.654725305909102,
255
+ 0.0018499409577999804,
256
+ 8.797375812580262e-05
257
+ ],
258
+ [
259
+ 0.8322357888797082,
260
+ 0.4231799241132008,
261
+ 0.002748950427211342,
262
+ 6.474846511780691e-05
263
+ ],
264
+ [
265
+ 0.8182308050526271,
266
+ 0.6246315987283173,
267
+ 0.002486184161428468,
268
+ 9.229583610962463e-05
269
+ ],
270
+ [
271
+ 0.8118260722446263,
272
+ 0.45159914137064233,
273
+ 0.0025363271072304113,
274
+ 6.039399649669963e-05
275
+ ],
276
+ [
277
+ 0.8685223966631739,
278
+ 0.47116001273017516,
279
+ 0.0030926640072680145,
280
+ 6.0699402388866954e-05
281
+ ],
282
+ [
283
+ 0.6571595307877908,
284
+ 0.9999999999999994,
285
+ 0.0011582699142094382,
286
+ 8.513281425522218e-05
287
+ ],
288
+ [
289
+ 0.7278670230804808,
290
+ 0.9999999999995702,
291
+ 0.0017357916610749636,
292
+ 9.582268678246998e-05
293
+ ],
294
+ [
295
+ 0.7368254823291168,
296
+ 0.7129205115395021,
297
+ 0.001798680710615292,
298
+ 9.58517902652646e-05
299
+ ],
300
+ [
301
+ 0.8486488254152718,
302
+ 0.4446517267140769,
303
+ 0.0029366332198873194,
304
+ 6.828702180806769e-05
305
+ ],
306
+ [
307
+ 0.804593219322101,
308
+ 0.34121645646403054,
309
+ 0.0023635973057661635,
310
+ 3.793596918700034e-05
311
+ ],
312
+ [
313
+ 0.8649213488536851,
314
+ 0.35848340339718276,
315
+ 0.0029338133914583652,
316
+ 5.380104179468968e-05
317
+ ],
318
+ [
319
+ 0.8287529615783064,
320
+ 0.42574119654010273,
321
+ 0.002701130706631773,
322
+ 5.885317282745769e-05
323
+ ],
324
+ [
325
+ 0.7874533953898625,
326
+ 0.9999999999999949,
327
+ 0.0023692546945181443,
328
+ 9.065213289650417e-05
329
+ ],
330
+ [
331
+ 0.8145593396763582,
332
+ 0.33520137779190157,
333
+ 0.0024431075639002233,
334
+ 3.7507030225938545e-05
335
+ ],
336
+ [
337
+ 0.7431869139144773,
338
+ 0.9999999999999996,
339
+ 0.0018308236077473816,
340
+ 0.0001035999954949547
341
+ ],
342
+ [
343
+ 0.8373484042291884,
344
+ 0.41537752970705893,
345
+ 0.0026703153410025026,
346
+ 5.299019278415294e-05
347
+ ],
348
+ [
349
+ 0.7036476176192261,
350
+ 0.9999999999626071,
351
+ 0.0016260211024298845,
352
+ 0.00010666818772398499
353
+ ],
354
+ [
355
+ 0.6128299265298791,
356
+ 0.9999999999993938,
357
+ 0.0009190937009442153,
358
+ 0.000100747340240261
359
+ ],
360
+ [
361
+ 0.7627861163692861,
362
+ 1.0,
363
+ 0.0020470553495592144,
364
+ 9.984065357712759e-05
365
+ ],
366
+ [
367
+ 0.7179290564225299,
368
+ 0.9999999999999881,
369
+ 0.0016265030761617122,
370
+ 0.00010890743390330733
371
+ ],
372
+ [
373
+ 0.751097475373388,
374
+ 1.0,
375
+ 0.001961121436625514,
376
+ 0.00011847043552063087
377
+ ],
378
+ [
379
+ 0.745551248332292,
380
+ 1.0,
381
+ 0.0019077165496051928,
382
+ 0.00010219799443282238
383
+ ],
384
+ [
385
+ 0.8165922055364906,
386
+ 0.4392150842204382,
387
+ 0.002633622389216885,
388
+ 5.623422822369551e-05
389
+ ],
390
+ [
391
+ 0.7157958911969412,
392
+ 0.9999999999999709,
393
+ 0.001447086476485912,
394
+ 0.00012964889929157686
395
+ ],
396
+ [
397
+ 0.8252952156096992,
398
+ 0.8037686958692518,
399
+ 0.002653637437900192,
400
+ 7.3372367875923e-05
401
+ ],
402
+ [
403
+ 0.7483286284513943,
404
+ 0.9999999999995427,
405
+ 0.001931809832949121,
406
+ 0.00010594651432441188
407
+ ],
408
+ [
409
+ 0.7616234179358646,
410
+ 0.4345319046492305,
411
+ 0.0019220165327800236,
412
+ 6.233636153154775e-05
413
+ ],
414
+ [
415
+ 0.7668977975115893,
416
+ 0.9999999999999997,
417
+ 0.002030100975435476,
418
+ 9.803899154798448e-05
419
+ ],
420
+ [
421
+ 0.7313087138205849,
422
+ 0.6262403053719064,
423
+ 0.0019925301126118666,
424
+ 7.695607319257882e-05
425
+ ],
426
+ [
427
+ 0.8335318018607543,
428
+ 0.8431573698607245,
429
+ 0.002602883146166928,
430
+ 9.606342885439337e-05
431
+ ],
432
+ [
433
+ 0.702750393739161,
434
+ 0.7193248097810666,
435
+ 0.0017167931539233707,
436
+ 8.193025270493273e-05
437
+ ],
438
+ [
439
+ 0.8114576744123023,
440
+ 0.5794606766352621,
441
+ 0.0024743261548782524,
442
+ 8.333277949324936e-05
443
+ ],
444
+ [
445
+ 0.8336212760159437,
446
+ 0.43946072679429166,
447
+ 0.0026990316077750197,
448
+ 6.211519357393451e-05
449
+ ],
450
+ [
451
+ 0.8077019755454314,
452
+ 0.44596306849928063,
453
+ 0.002589685958769796,
454
+ 5.7638899585390255e-05
455
+ ],
456
+ [
457
+ 0.8093518137624087,
458
+ 0.4369821925777495,
459
+ 0.0026809553564767336,
460
+ 6.283914858591618e-05
461
+ ],
462
+ [
463
+ 0.7895076969641859,
464
+ 0.9999999999995145,
465
+ 0.0024872267137695733,
466
+ 9.775805700950685e-05
467
+ ],
468
+ [
469
+ 0.8289441481528866,
470
+ 0.4885924738367364,
471
+ 0.002909517178506294,
472
+ 6.462126736195849e-05
473
+ ],
474
+ [
475
+ 0.6806983667885629,
476
+ 0.2354614568986886,
477
+ 0.0018175080914621384,
478
+ 3.108332427879116e-05
479
+ ],
480
+ [
481
+ 0.7901713822281731,
482
+ 0.9999999999999996,
483
+ 0.0022842597039831555,
484
+ 9.627723001798349e-05
485
+ ],
486
+ [
487
+ 0.8292701345021662,
488
+ 0.4217254769703908,
489
+ 0.002682192884883911,
490
+ 5.84177041201604e-05
491
+ ],
492
+ [
493
+ 0.8050992509995542,
494
+ 0.9999999999966119,
495
+ 0.0025440847196868148,
496
+ 9.033324253165938e-05
497
+ ],
498
+ [
499
+ 0.7748350905770086,
500
+ 0.6761130904987867,
501
+ 0.0021473466731111307,
502
+ 8.56910469834146e-05
503
+ ],
504
+ [
505
+ 0.8610689971531106,
506
+ 0.5383361980554172,
507
+ 0.0029611932157315992,
508
+ 7.476419909586163e-05
509
+ ],
510
+ [
511
+ 0.805034205571647,
512
+ 0.5713597259576557,
513
+ 0.002507380147929796,
514
+ 8.425292972567204e-05
515
+ ],
516
+ [
517
+ 0.8118523090850409,
518
+ 0.9999999999296268,
519
+ 0.0024975023253337577,
520
+ 9.164177486735534e-05
521
+ ],
522
+ [
523
+ 0.7385630253698925,
524
+ 0.9999999999995822,
525
+ 0.0018563904926071125,
526
+ 7.483260928731443e-05
527
+ ],
528
+ [
529
+ 0.7357301703687016,
530
+ 0.6290900467531724,
531
+ 0.0018497842142061395,
532
+ 7.947504722989657e-05
533
+ ],
534
+ [
535
+ 0.7400445485327132,
536
+ 0.765249832904814,
537
+ 0.0017236662747003352,
538
+ 8.305813890766016e-05
539
+ ],
540
+ [
541
+ 0.7447748013099332,
542
+ 0.999999999993609,
543
+ 0.0019473445298185319,
544
+ 9.948531308470131e-05
545
+ ],
546
+ [
547
+ 0.8174676987931586,
548
+ 0.4220654478735278,
549
+ 0.002463385948851147,
550
+ 5.1052571288925954e-05
551
+ ],
552
+ [
553
+ 0.7791356346799273,
554
+ 0.5859463712452607,
555
+ 0.002244081315439596,
556
+ 9.213863530673593e-05
557
+ ],
558
+ [
559
+ 0.7605841645461007,
560
+ 0.7445174548852463,
561
+ 0.0020964306623522406,
562
+ 9.126751756419745e-05
563
+ ],
564
+ [
565
+ 0.7946798515668119,
566
+ 0.45851446596260326,
567
+ 0.0023996294041861437,
568
+ 7.315791955377272e-05
569
+ ],
570
+ [
571
+ 0.8111236612532428,
572
+ 0.4118507483848411,
573
+ 0.00245276392945004,
574
+ 4.9900623529653874e-05
575
+ ],
576
+ [
577
+ 0.7611940655504937,
578
+ 0.5499033917533854,
579
+ 0.002188298304902807,
580
+ 8.755407450365254e-05
581
+ ],
582
+ [
583
+ 0.7186789156176874,
584
+ 0.4831799218029921,
585
+ 0.001675574859959616,
586
+ 7.556260621040713e-05
587
+ ],
588
+ [
589
+ 0.7535823358380767,
590
+ 0.6431608500416047,
591
+ 0.001727367850641587,
592
+ 7.881795271914563e-05
593
+ ],
594
+ [
595
+ 0.6964303540335867,
596
+ 0.9999999999999988,
597
+ 0.0014114773169190916,
598
+ 9.554030654954195e-05
599
+ ],
600
+ [
601
+ 0.8622954195157714,
602
+ 0.42808983040346976,
603
+ 0.0029568213774586747,
604
+ 6.142656647115003e-05
605
+ ],
606
+ [
607
+ 0.6532902540678565,
608
+ 0.9091396803803754,
609
+ 0.0011555060786130388,
610
+ 7.976621590795323e-05
611
+ ],
612
+ [
613
+ 0.7737072384941597,
614
+ 0.788970032594236,
615
+ 0.0021948117595252357,
616
+ 8.274181972773067e-05
617
+ ],
618
+ [
619
+ 0.6913857253090951,
620
+ 0.9999999999999989,
621
+ 0.0017955563282699466,
622
+ 0.00010050174936765267
623
+ ],
624
+ [
625
+ 0.7361599624177599,
626
+ 0.6971480138244244,
627
+ 0.0018686850365184032,
628
+ 9.33442892049338e-05
629
+ ],
630
+ [
631
+ 0.7651850789572118,
632
+ 0.4968597510282423,
633
+ 0.002218988525801107,
634
+ 7.187077115291988e-05
635
+ ],
636
+ [
637
+ 0.8317015590712944,
638
+ 0.335980105480587,
639
+ 0.002794223650601909,
640
+ 5.1635127271601796e-05
641
+ ],
642
+ [
643
+ 0.705898435285506,
644
+ 0.6091458624739697,
645
+ 0.0016656074809989383,
646
+ 7.52033184046103e-05
647
+ ],
648
+ [
649
+ 0.7242327325689151,
650
+ 0.9999999999999823,
651
+ 0.0018379501960491915,
652
+ 0.00010109350235933514
653
+ ],
654
+ [
655
+ 0.7706957175900865,
656
+ 0.6443317128363204,
657
+ 0.0022246193673120434,
658
+ 7.468948147003183e-05
659
+ ],
660
+ [
661
+ 0.8512860626912084,
662
+ 0.4218019565341914,
663
+ 0.0028977410778572585,
664
+ 5.953210212922031e-05
665
+ ],
666
+ [
667
+ 0.706332070657838,
668
+ 0.9999999944515124,
669
+ 0.0015131808512747439,
670
+ 9.895078837013193e-05
671
+ ],
672
+ [
673
+ 0.7301575713074908,
674
+ 0.8618359808744479,
675
+ 0.0019391214088100457,
676
+ 8.172832042258141e-05
677
+ ],
678
+ [
679
+ 0.8141730966056073,
680
+ 0.4737851176021522,
681
+ 0.0024554167927473987,
682
+ 6.0073564933500756e-05
683
+ ],
684
+ [
685
+ 0.7842760970160346,
686
+ 0.3393907460739776,
687
+ 0.002358984204081886,
688
+ 4.4110643310761415e-05
689
+ ],
690
+ [
691
+ 0.723993961770651,
692
+ 0.9909100299481836,
693
+ 0.00199901670544235,
694
+ 9.219534857777653e-05
695
+ ],
696
+ [
697
+ 0.7514454909803882,
698
+ 0.4691294332652259,
699
+ 0.002307517481031161,
700
+ 7.66758301008121e-05
701
+ ],
702
+ [
703
+ 0.8275399512959896,
704
+ 0.9999999999999958,
705
+ 0.0026332768419447907,
706
+ 9.825805788544647e-05
707
+ ],
708
+ [
709
+ 0.7680779687604855,
710
+ 0.5330922293290868,
711
+ 0.0020988254960352163,
712
+ 8.129740948142792e-05
713
+ ],
714
+ [
715
+ 0.8420273314466622,
716
+ 0.31712452880032577,
717
+ 0.002859914285645426,
718
+ 4.220790785411018e-05
719
+ ],
720
+ [
721
+ 0.7417991233248284,
722
+ 0.9647496573111105,
723
+ 0.0017019898693873498,
724
+ 0.00010955535435716848
725
+ ],
726
+ [
727
+ 0.756580377041966,
728
+ 0.7706830282418149,
729
+ 0.0022377121362757964,
730
+ 8.354763202470197e-05
731
+ ],
732
+ [
733
+ 0.7449406057444884,
734
+ 0.9999999999820625,
735
+ 0.0021922727762014316,
736
+ 9.118138355914767e-05
737
+ ],
738
+ [
739
+ 0.7029170537006624,
740
+ 0.9999999999999976,
741
+ 0.0015753956910101261,
742
+ 8.754437151289621e-05
743
+ ],
744
+ [
745
+ 0.7296784074867149,
746
+ 0.9999999999999999,
747
+ 0.0020297735984073687,
748
+ 0.00010100941472724924
749
+ ],
750
+ [
751
+ 0.7407855359134962,
752
+ 0.9999999999985886,
753
+ 0.0018694943278335292,
754
+ 0.0001064929144036425
755
+ ],
756
+ [
757
+ 0.7133422731531179,
758
+ 0.9999999999997181,
759
+ 0.0016472569475700259,
760
+ 9.235927281590073e-05
761
+ ],
762
+ [
763
+ 0.8253355966144139,
764
+ 0.4041447387681608,
765
+ 0.002703288621616843,
766
+ 5.516155032660833e-05
767
+ ],
768
+ [
769
+ 0.8167212633173931,
770
+ 0.4766319842712363,
771
+ 0.002618706568117999,
772
+ 6.304278328280155e-05
773
+ ],
774
+ [
775
+ 0.7158788070740595,
776
+ 0.9999999999999968,
777
+ 0.0013906219408478344,
778
+ 0.00012274607843641716
779
+ ],
780
+ [
781
+ 0.8041450533891074,
782
+ 0.4092959536674991,
783
+ 0.0024846603250295463,
784
+ 5.259734973337097e-05
785
+ ],
786
+ [
787
+ 0.7412798760308232,
788
+ 0.9999999999988626,
789
+ 0.0017782044825136064,
790
+ 0.00011309770086046463
791
+ ],
792
+ [
793
+ 0.7137527705871421,
794
+ 0.9991264209171432,
795
+ 0.0015776641633933914,
796
+ 7.962165825284975e-05
797
+ ],
798
+ [
799
+ 0.7818082905647621,
800
+ 0.5713710361524574,
801
+ 0.0022885028130101574,
802
+ 7.39540333522652e-05
803
+ ],
804
+ [
805
+ 0.8477608436115328,
806
+ 0.42149751095011684,
807
+ 0.0028425123179948064,
808
+ 6.059951463762912e-05
809
+ ],
810
+ [
811
+ 0.6281083087184094,
812
+ 0.9999999999999984,
813
+ 0.0007587490349252905,
814
+ 0.00010949461564514614
815
+ ],
816
+ [
817
+ 0.8677153547247751,
818
+ 0.4014058929512122,
819
+ 0.002893777312471754,
820
+ 5.047659227148908e-05
821
+ ],
822
+ [
823
+ 0.8162801613692985,
824
+ 0.41903257575738434,
825
+ 0.002602130316998014,
826
+ 5.308643895688496e-05
827
+ ],
828
+ [
829
+ 0.8319358917083393,
830
+ 0.46940619107168374,
831
+ 0.0028687296164740094,
832
+ 6.859572607730729e-05
833
+ ],
834
+ [
835
+ 0.7669409677834198,
836
+ 0.5417765464816305,
837
+ 0.002084972134328352,
838
+ 7.670243641967474e-05
839
+ ]
840
+ ]
841
+ },
842
+ "TensorRT-LLM": {
843
+ "stack": {
844
+ "name": "TensorRT-LLM",
845
+ "eta_bw": 0.7796191116388631,
846
+ "eta_c": 0.5606647598235109,
847
+ "t0": 0.0013746118851451557,
848
+ "a": 1.8938656021816108e-05,
849
+ "combine": "smooth",
850
+ "k": 4.0,
851
+ "compute_mult": 1.0
852
+ },
853
+ "n": 184,
854
+ "mape_in": 0.06355373992158145,
855
+ "median_ape_in": 0.053024128247834035,
856
+ "mape_logo": 0.08733257988103045,
857
+ "per_logo": {
858
+ "Nvidia A100 GPU": 0.06048933463173345,
859
+ "Nvidia GH200 GPU": 0.10745149858776595,
860
+ "Nvidia H100 GPU": 0.09019329973399003
861
+ },
862
+ "mape_lomo": 0.08234755667069374,
863
+ "per_lomo": {
864
+ "llama2-7b": 0.13072963367151963,
865
+ "llama31-8b": 0.05876058975972311,
866
+ "mistral-7b": 0.057166160953662414,
867
+ "qwen2-7b": 0.11098690323387012
868
+ },
869
+ "boot": [
870
+ [
871
+ 0.7359219060193969,
872
+ 0.6227365856397531,
873
+ 0.001053278155594566,
874
+ 1.834942753793523e-05
875
+ ],
876
+ [
877
+ 0.7885445651941824,
878
+ 0.7002023798439808,
879
+ 0.001515009491165306,
880
+ 2.9313571695961898e-05
881
+ ],
882
+ [
883
+ 0.7432319243228959,
884
+ 0.5399354597491807,
885
+ 0.0011267602156766936,
886
+ 1.5776290419157943e-05
887
+ ],
888
+ [
889
+ 0.7499636116041716,
890
+ 0.5298761221743113,
891
+ 0.0011553055544849522,
892
+ 1.6507997215328654e-05
893
+ ],
894
+ [
895
+ 0.7858095594054194,
896
+ 0.49034586631762084,
897
+ 0.001415765333832417,
898
+ 1.9052781687999997e-05
899
+ ],
900
+ [
901
+ 0.6659882200791357,
902
+ 0.9999999999999986,
903
+ 0.0005264585746758356,
904
+ 3.3728417654820376e-05
905
+ ],
906
+ [
907
+ 0.8232976297749337,
908
+ 0.6712797966472696,
909
+ 0.0016645310458951142,
910
+ 2.6053189738141698e-05
911
+ ],
912
+ [
913
+ 0.758756186213661,
914
+ 0.7030367513781075,
915
+ 0.0013547209015168553,
916
+ 2.931548102677105e-05
917
+ ],
918
+ [
919
+ 0.8139827763083345,
920
+ 0.46182580864553724,
921
+ 0.001561587478925132,
922
+ 1.0802003097855554e-05
923
+ ],
924
+ [
925
+ 0.8526462814080589,
926
+ 0.5046045557860185,
927
+ 0.0018467323350649077,
928
+ 1.65781482233843e-05
929
+ ],
930
+ [
931
+ 0.7903879741691824,
932
+ 0.9965980397636426,
933
+ 0.0014619180521554238,
934
+ 3.2973836048759415e-05
935
+ ],
936
+ [
937
+ 0.8470741951105514,
938
+ 0.531680564869698,
939
+ 0.0016361563702348147,
940
+ 1.654043762483558e-05
941
+ ],
942
+ [
943
+ 0.8026086634168096,
944
+ 0.8200176822348986,
945
+ 0.0015270956422075031,
946
+ 2.781636558752014e-05
947
+ ],
948
+ [
949
+ 0.751799405712363,
950
+ 1.0,
951
+ 0.0012610272690656032,
952
+ 4.051760859376816e-05
953
+ ],
954
+ [
955
+ 0.8115508580888094,
956
+ 0.48813434111464693,
957
+ 0.0015579185860161429,
958
+ 6.978328689059467e-06
959
+ ],
960
+ [
961
+ 0.760334508444434,
962
+ 0.5826573516308334,
963
+ 0.0013268766161577646,
964
+ 2.320932391615385e-05
965
+ ],
966
+ [
967
+ 0.8435593242371586,
968
+ 0.44451850836012785,
969
+ 0.0017184877419096965,
970
+ 1.5701646099380192e-05
971
+ ],
972
+ [
973
+ 0.7765546474593993,
974
+ 0.8794054255782748,
975
+ 0.0013174421211071471,
976
+ 3.309285400480369e-05
977
+ ],
978
+ [
979
+ 0.768080705357406,
980
+ 0.5758941419278004,
981
+ 0.0013104836156245735,
982
+ 1.9745000172613383e-05
983
+ ],
984
+ [
985
+ 0.7630169244605285,
986
+ 0.5068682976507168,
987
+ 0.001340881463799197,
988
+ 1.6837494080019053e-05
989
+ ],
990
+ [
991
+ 0.7234398912724461,
992
+ 0.573109346315636,
993
+ 0.0010340838026915058,
994
+ 1.5308655680066116e-05
995
+ ],
996
+ [
997
+ 0.7971500897463571,
998
+ 0.4894992663635182,
999
+ 0.0014570099802011557,
1000
+ 9.640009955068527e-06
1001
+ ],
1002
+ [
1003
+ 0.834911800694022,
1004
+ 0.4934402044958195,
1005
+ 0.0017472790278948154,
1006
+ 1.7404888739696767e-05
1007
+ ],
1008
+ [
1009
+ 0.7827751995650781,
1010
+ 0.4659411928817351,
1011
+ 0.001330165036281363,
1012
+ 6.457778824963409e-07
1013
+ ],
1014
+ [
1015
+ 0.7915972128682512,
1016
+ 0.4775241517962862,
1017
+ 0.001448144443124696,
1018
+ 3.565784779119575e-06
1019
+ ],
1020
+ [
1021
+ 0.8128980212312822,
1022
+ 0.4135290833262471,
1023
+ 0.001510673190796661,
1024
+ 8.395170816124575e-07
1025
+ ],
1026
+ [
1027
+ 0.8091767385355841,
1028
+ 0.6304231157866583,
1029
+ 0.001685425855539217,
1030
+ 2.7179788072376374e-05
1031
+ ],
1032
+ [
1033
+ 0.783937619703882,
1034
+ 1.0,
1035
+ 0.0015171424426579912,
1036
+ 3.774421916117577e-05
1037
+ ],
1038
+ [
1039
+ 0.7613046815898579,
1040
+ 0.6576995850345853,
1041
+ 0.0012381535663196018,
1042
+ 2.282769243470272e-05
1043
+ ],
1044
+ [
1045
+ 0.7484657588115747,
1046
+ 0.6717938586675668,
1047
+ 0.0011313392822102808,
1048
+ 2.2891359986181538e-05
1049
+ ],
1050
+ [
1051
+ 0.7959984931409848,
1052
+ 0.4684957150209964,
1053
+ 0.0013900071165396403,
1054
+ 3.2879226228400483e-06
1055
+ ],
1056
+ [
1057
+ 0.8304343779840485,
1058
+ 0.470944294346478,
1059
+ 0.001836150755481388,
1060
+ 1.3122064174162856e-05
1061
+ ],
1062
+ [
1063
+ 0.7901993636286831,
1064
+ 0.413821033441648,
1065
+ 0.0014286473029545593,
1066
+ 1.0000000000000952e-07
1067
+ ],
1068
+ [
1069
+ 0.8100899255017717,
1070
+ 0.46216850403620807,
1071
+ 0.0015168179900140204,
1072
+ 1.0342775595904172e-05
1073
+ ],
1074
+ [
1075
+ 0.7365152872127152,
1076
+ 0.9999999999999887,
1077
+ 0.0010145055881416946,
1078
+ 3.292945153348023e-05
1079
+ ],
1080
+ [
1081
+ 0.8520093980350707,
1082
+ 0.4274670043681462,
1083
+ 0.001984447832755207,
1084
+ 5.617686048134292e-06
1085
+ ],
1086
+ [
1087
+ 0.7428640266570005,
1088
+ 0.4825978275469581,
1089
+ 0.001077680385525213,
1090
+ 8.834720183401542e-06
1091
+ ],
1092
+ [
1093
+ 0.7190263269667936,
1094
+ 0.5728198592661821,
1095
+ 0.000979490014570719,
1096
+ 1.4873861186753387e-05
1097
+ ],
1098
+ [
1099
+ 0.7112398326070942,
1100
+ 0.9999999999983566,
1101
+ 0.0008451237684925684,
1102
+ 3.4670860887818015e-05
1103
+ ],
1104
+ [
1105
+ 0.7065429802012511,
1106
+ 0.6325046718388212,
1107
+ 0.0008242410312853946,
1108
+ 1.719582782345495e-05
1109
+ ],
1110
+ [
1111
+ 0.8356567619051908,
1112
+ 0.9999999999924324,
1113
+ 0.0016384599095415357,
1114
+ 3.122209025222083e-05
1115
+ ],
1116
+ [
1117
+ 0.6979870098214077,
1118
+ 0.732133670432916,
1119
+ 0.0007525411928219548,
1120
+ 2.247419244634168e-05
1121
+ ],
1122
+ [
1123
+ 0.7065010164918067,
1124
+ 0.6172526914523867,
1125
+ 0.0007428289480710296,
1126
+ 1.6555994989421926e-05
1127
+ ],
1128
+ [
1129
+ 0.8056332470385152,
1130
+ 0.9999999999999751,
1131
+ 0.0016329476228324155,
1132
+ 3.977178354824445e-05
1133
+ ],
1134
+ [
1135
+ 0.7201199757358795,
1136
+ 0.99999999999971,
1137
+ 0.0009850838375090535,
1138
+ 3.777728157439849e-05
1139
+ ],
1140
+ [
1141
+ 0.69927732241733,
1142
+ 0.9999999999999643,
1143
+ 0.0007116986184844294,
1144
+ 3.4860400778507605e-05
1145
+ ],
1146
+ [
1147
+ 0.7803790957127047,
1148
+ 0.9999999999999574,
1149
+ 0.0014027531392505224,
1150
+ 3.326422778272373e-05
1151
+ ],
1152
+ [
1153
+ 0.7661527003246358,
1154
+ 0.7943431057516055,
1155
+ 0.001254319812643982,
1156
+ 3.1811972812687906e-05
1157
+ ],
1158
+ [
1159
+ 0.7760257670909669,
1160
+ 0.5750869364747103,
1161
+ 0.001404632921053081,
1162
+ 1.941069488591985e-05
1163
+ ],
1164
+ [
1165
+ 0.8898660567295134,
1166
+ 0.9999999999985952,
1167
+ 0.0022114878608227007,
1168
+ 3.855160120541064e-05
1169
+ ],
1170
+ [
1171
+ 0.7648704776473704,
1172
+ 0.5080390415265577,
1173
+ 0.001249024257025083,
1174
+ 1.820387944466332e-05
1175
+ ],
1176
+ [
1177
+ 0.8417112473731513,
1178
+ 0.5096572619601326,
1179
+ 0.0016782393920205892,
1180
+ 7.328228004955635e-06
1181
+ ],
1182
+ [
1183
+ 0.6895866145742238,
1184
+ 0.9999999999999999,
1185
+ 0.0007239117976723822,
1186
+ 3.404160436925011e-05
1187
+ ],
1188
+ [
1189
+ 0.8254367068419877,
1190
+ 0.5058148114887024,
1191
+ 0.0017098496589789986,
1192
+ 1.331734037226613e-05
1193
+ ],
1194
+ [
1195
+ 0.7848321372702044,
1196
+ 0.4356672433878565,
1197
+ 0.0012601637414879879,
1198
+ 1.0000000000160221e-07
1199
+ ],
1200
+ [
1201
+ 0.7269737173893612,
1202
+ 0.999999999998833,
1203
+ 0.0010293547324274762,
1204
+ 3.5910463604801426e-05
1205
+ ],
1206
+ [
1207
+ 0.7265894904279827,
1208
+ 0.5853620371839363,
1209
+ 0.0009325137882295526,
1210
+ 1.753779234576245e-05
1211
+ ],
1212
+ [
1213
+ 0.8254779861590845,
1214
+ 0.5181472157217868,
1215
+ 0.0014934517645343824,
1216
+ 1.2850286309177061e-05
1217
+ ],
1218
+ [
1219
+ 0.7537925020649257,
1220
+ 0.5314537955317353,
1221
+ 0.0011756104198489172,
1222
+ 1.980757319069345e-05
1223
+ ],
1224
+ [
1225
+ 0.8144320762928754,
1226
+ 0.45714579161213753,
1227
+ 0.0015053091358659852,
1228
+ 1.5810403112121225e-05
1229
+ ],
1230
+ [
1231
+ 0.7956377300741353,
1232
+ 1.0,
1233
+ 0.0014454961889524116,
1234
+ 3.5998686861516885e-05
1235
+ ],
1236
+ [
1237
+ 0.7711906525563296,
1238
+ 0.5312913210334158,
1239
+ 0.0012691796101071344,
1240
+ 1.3398300250400527e-05
1241
+ ],
1242
+ [
1243
+ 0.8399496518558034,
1244
+ 0.9999999999999056,
1245
+ 0.0016931163661550843,
1246
+ 3.286299120293595e-05
1247
+ ],
1248
+ [
1249
+ 0.7768209723035081,
1250
+ 0.9999999999999976,
1251
+ 0.0013966762342397117,
1252
+ 4.0216862140214555e-05
1253
+ ],
1254
+ [
1255
+ 0.8221925579807335,
1256
+ 0.437592342708557,
1257
+ 0.0014391233000413252,
1258
+ 1.0000000000000029e-07
1259
+ ],
1260
+ [
1261
+ 0.784011116539838,
1262
+ 0.4800298966977662,
1263
+ 0.0013010038449956973,
1264
+ 8.441174462496427e-06
1265
+ ],
1266
+ [
1267
+ 0.8775779936103942,
1268
+ 0.5083750624655717,
1269
+ 0.0017191795157205578,
1270
+ 1.2527105672826686e-05
1271
+ ],
1272
+ [
1273
+ 0.7997124543595263,
1274
+ 0.30233032110386293,
1275
+ 0.001541994971532104,
1276
+ 1.0000000000000029e-07
1277
+ ],
1278
+ [
1279
+ 0.7577805373665452,
1280
+ 0.720488710529132,
1281
+ 0.0012706445077702421,
1282
+ 3.151896458602727e-05
1283
+ ],
1284
+ [
1285
+ 0.7619044346287372,
1286
+ 0.9999999999981702,
1287
+ 0.0014283086987594237,
1288
+ 4.5271636415576855e-05
1289
+ ],
1290
+ [
1291
+ 0.7687546437843186,
1292
+ 0.6377422261231228,
1293
+ 0.0013249416467350087,
1294
+ 1.986404515905207e-05
1295
+ ],
1296
+ [
1297
+ 0.7941456263215518,
1298
+ 0.4574385640028715,
1299
+ 0.0014213239595069402,
1300
+ 1.0000000000002125e-07
1301
+ ],
1302
+ [
1303
+ 0.7598341949845947,
1304
+ 0.5079036734504417,
1305
+ 0.0011821323710949577,
1306
+ 7.648275036592302e-06
1307
+ ],
1308
+ [
1309
+ 0.7429308064473075,
1310
+ 0.9444429665529903,
1311
+ 0.001102928945144453,
1312
+ 3.5360096354974814e-05
1313
+ ],
1314
+ [
1315
+ 0.8190997348548839,
1316
+ 0.3884618741932314,
1317
+ 0.0015654652380597742,
1318
+ 1.0719586776082247e-06
1319
+ ],
1320
+ [
1321
+ 0.8059943331594777,
1322
+ 0.4424694045761098,
1323
+ 0.0015840076508554556,
1324
+ 6.050835303748424e-06
1325
+ ],
1326
+ [
1327
+ 0.719865245704601,
1328
+ 0.6636696789044534,
1329
+ 0.0009477702814159146,
1330
+ 2.121595255939478e-05
1331
+ ],
1332
+ [
1333
+ 0.7793834926070883,
1334
+ 0.4234041082720739,
1335
+ 0.0013640231636910761,
1336
+ 7.383401773389143e-06
1337
+ ],
1338
+ [
1339
+ 0.7525597368912674,
1340
+ 0.8543646551563538,
1341
+ 0.0012632938214623018,
1342
+ 3.468872916761552e-05
1343
+ ],
1344
+ [
1345
+ 0.7560839140913481,
1346
+ 0.8369319979774522,
1347
+ 0.0013494177505132812,
1348
+ 3.4931437317347206e-05
1349
+ ],
1350
+ [
1351
+ 0.8437059232715044,
1352
+ 0.5266934669738106,
1353
+ 0.0016777630829972164,
1354
+ 1.590160662198321e-05
1355
+ ],
1356
+ [
1357
+ 0.7634444706048117,
1358
+ 0.31492066861343165,
1359
+ 0.0012763461356467116,
1360
+ 1.0000000000015306e-07
1361
+ ],
1362
+ [
1363
+ 0.8380862083620546,
1364
+ 0.4472476417910155,
1365
+ 0.0016637926189417688,
1366
+ 1.2063864643635252e-05
1367
+ ],
1368
+ [
1369
+ 0.7363367602687672,
1370
+ 0.8289348752461639,
1371
+ 0.000986520662114427,
1372
+ 2.481618885728654e-05
1373
+ ],
1374
+ [
1375
+ 0.7520776999078977,
1376
+ 0.4837314440589353,
1377
+ 0.0011867935585451093,
1378
+ 9.353217317409197e-06
1379
+ ],
1380
+ [
1381
+ 0.8000823205279449,
1382
+ 0.45050350388382826,
1383
+ 0.001573639646426504,
1384
+ 1.2098739118247222e-05
1385
+ ],
1386
+ [
1387
+ 0.8252042734719364,
1388
+ 0.3962107744813243,
1389
+ 0.001602737738552992,
1390
+ 1.3722288375878344e-06
1391
+ ],
1392
+ [
1393
+ 0.7419650665394701,
1394
+ 0.706327245460553,
1395
+ 0.0012623146517083666,
1396
+ 2.9666503132031772e-05
1397
+ ],
1398
+ [
1399
+ 0.7896441665999822,
1400
+ 0.5584682213433155,
1401
+ 0.0014127322852043338,
1402
+ 2.2833947822427677e-05
1403
+ ],
1404
+ [
1405
+ 0.6674822656775086,
1406
+ 0.8793291412156449,
1407
+ 0.0005370436011334892,
1408
+ 2.8288532645785582e-05
1409
+ ],
1410
+ [
1411
+ 0.7346417727676701,
1412
+ 0.6337822945032506,
1413
+ 0.001020886307425816,
1414
+ 1.8396637439716057e-05
1415
+ ],
1416
+ [
1417
+ 0.7360743377164395,
1418
+ 0.5841923766414739,
1419
+ 0.0011089470944841836,
1420
+ 2.0417576782966276e-05
1421
+ ],
1422
+ [
1423
+ 0.7126160023652109,
1424
+ 0.6886051823551841,
1425
+ 0.0009077134322523333,
1426
+ 2.700976313924168e-05
1427
+ ],
1428
+ [
1429
+ 0.7714723359763004,
1430
+ 0.4937640147969343,
1431
+ 0.0011586129298749549,
1432
+ 3.883175743291744e-06
1433
+ ],
1434
+ [
1435
+ 0.8254486866465843,
1436
+ 0.39384736620931093,
1437
+ 0.0015919442657151939,
1438
+ 6.730213342437136e-06
1439
+ ],
1440
+ [
1441
+ 0.8253500061479037,
1442
+ 0.48485482846841094,
1443
+ 0.0015151849760112852,
1444
+ 1.1520243303471474e-05
1445
+ ],
1446
+ [
1447
+ 0.8415705456830406,
1448
+ 0.37811287884328404,
1449
+ 0.0015250092366529055,
1450
+ 5.119151740533511e-07
1451
+ ],
1452
+ [
1453
+ 0.8526500988934947,
1454
+ 0.9816467669098754,
1455
+ 0.0017217163936249266,
1456
+ 2.7795884187620053e-05
1457
+ ],
1458
+ [
1459
+ 0.7419023457574205,
1460
+ 0.7321025197721713,
1461
+ 0.0011681336520221288,
1462
+ 2.8179204994797988e-05
1463
+ ],
1464
+ [
1465
+ 0.7783633958463528,
1466
+ 0.5428464344308617,
1467
+ 0.0015035444762186783,
1468
+ 2.129950596243695e-05
1469
+ ],
1470
+ [
1471
+ 0.7447248999192965,
1472
+ 0.8778259706362875,
1473
+ 0.0011536950388628482,
1474
+ 3.696605766256025e-05
1475
+ ],
1476
+ [
1477
+ 0.7580758537312632,
1478
+ 0.4962393182420136,
1479
+ 0.0011882732389728377,
1480
+ 1.682546465730765e-05
1481
+ ],
1482
+ [
1483
+ 0.737455449051877,
1484
+ 0.7977881012995194,
1485
+ 0.0010779312834962255,
1486
+ 2.9420142280448206e-05
1487
+ ],
1488
+ [
1489
+ 0.9100792054522403,
1490
+ 0.8533589210609352,
1491
+ 0.002338550858508574,
1492
+ 3.693662305639224e-05
1493
+ ],
1494
+ [
1495
+ 0.74658189201559,
1496
+ 0.505100402976335,
1497
+ 0.0011131708095762519,
1498
+ 1.3066249722336136e-05
1499
+ ],
1500
+ [
1501
+ 0.7914905790736605,
1502
+ 0.48073497903346135,
1503
+ 0.0013836792155246785,
1504
+ 9.366211053315413e-06
1505
+ ],
1506
+ [
1507
+ 0.7989305870757035,
1508
+ 0.624486634705717,
1509
+ 0.0015808318922203085,
1510
+ 2.783806930671276e-05
1511
+ ],
1512
+ [
1513
+ 0.794128635252851,
1514
+ 0.4455928793903156,
1515
+ 0.0015450773629228508,
1516
+ 9.219772601006449e-06
1517
+ ],
1518
+ [
1519
+ 0.7465498943576939,
1520
+ 0.4764094050322832,
1521
+ 0.0011061823094758508,
1522
+ 8.63370136640782e-06
1523
+ ],
1524
+ [
1525
+ 0.7142669948736567,
1526
+ 0.6448001191183055,
1527
+ 0.0008943179654972464,
1528
+ 2.0758293853005927e-05
1529
+ ],
1530
+ [
1531
+ 0.8132616770214822,
1532
+ 0.41317089056816414,
1533
+ 0.001649124644236535,
1534
+ 5.888296839981448e-06
1535
+ ],
1536
+ [
1537
+ 0.7407232734625357,
1538
+ 0.6693768560532652,
1539
+ 0.0010886794117472467,
1540
+ 2.5947620197189985e-05
1541
+ ],
1542
+ [
1543
+ 0.7329436977209065,
1544
+ 0.9999999999999998,
1545
+ 0.0012336768050582373,
1546
+ 4.416409348610507e-05
1547
+ ],
1548
+ [
1549
+ 0.7834387879262319,
1550
+ 0.4615690136590909,
1551
+ 0.0014935015280803838,
1552
+ 1.4460166143241233e-05
1553
+ ],
1554
+ [
1555
+ 0.8422398528440034,
1556
+ 0.48881906719890417,
1557
+ 0.0016974229104824645,
1558
+ 9.307665500048196e-06
1559
+ ],
1560
+ [
1561
+ 0.7107783003182624,
1562
+ 1.0,
1563
+ 0.0009482983387662579,
1564
+ 3.800856299521965e-05
1565
+ ],
1566
+ [
1567
+ 0.8381838756926461,
1568
+ 0.7572702052173461,
1569
+ 0.0018686303657684154,
1570
+ 2.967223256463352e-05
1571
+ ],
1572
+ [
1573
+ 0.721901273874388,
1574
+ 1.0,
1575
+ 0.0010124715863267235,
1576
+ 3.7807850117186755e-05
1577
+ ],
1578
+ [
1579
+ 0.7885907248061383,
1580
+ 0.5431761652438593,
1581
+ 0.0013449329925741143,
1582
+ 1.5427865437288046e-05
1583
+ ],
1584
+ [
1585
+ 0.754995569387005,
1586
+ 0.47007721296885274,
1587
+ 0.0012243525776440935,
1588
+ 6.314859963572301e-06
1589
+ ]
1590
+ ]
1591
+ }
1592
+ },
1593
+ "system_max": {
1594
+ "vLLM": {
1595
+ "stack": {
1596
+ "name": "vLLM",
1597
+ "eta_bw": 0.7751549369880121,
1598
+ "eta_c": 0.5963988791049138,
1599
+ "t0": 0.002249833167186188,
1600
+ "a": 8.025375061343338e-05,
1601
+ "combine": "max",
1602
+ "k": 4.0,
1603
+ "compute_mult": 1.0
1604
+ },
1605
+ "n": 188,
1606
+ "mape_in": 0.06084518653047605,
1607
+ "median_ape_in": 0.03780072399981438,
1608
+ "mape_logo": 0.11926649440533145,
1609
+ "per_logo": {
1610
+ "Nvidia A100 GPU": 0.2266343546900922,
1611
+ "Nvidia GH200 GPU": 0.059156419509656936,
1612
+ "Nvidia H100 GPU": 0.09281669809412675
1613
+ },
1614
+ "mape_lomo": 0.07559524226767847,
1615
+ "per_lomo": {
1616
+ "llama2-7b": 0.08376126193382903,
1617
+ "llama31-8b": 0.055711396795361266,
1618
+ "mistral-7b": 0.10481594492100356,
1619
+ "qwen2-7b": 0.06622620466966624
1620
+ }
1621
+ },
1622
+ "TensorRT-LLM": {
1623
+ "stack": {
1624
+ "name": "TensorRT-LLM",
1625
+ "eta_bw": 0.7836849553838169,
1626
+ "eta_c": 0.527010598403318,
1627
+ "t0": 0.0014030260698517777,
1628
+ "a": 1.7489411415306152e-05,
1629
+ "combine": "max",
1630
+ "k": 4.0,
1631
+ "compute_mult": 1.0
1632
+ },
1633
+ "n": 184,
1634
+ "mape_in": 0.06339813590495912,
1635
+ "median_ape_in": 0.052708378327944005,
1636
+ "mape_logo": 0.09044162213269816,
1637
+ "per_logo": {
1638
+ "Nvidia A100 GPU": 0.07152040411015496,
1639
+ "Nvidia GH200 GPU": 0.10737842605540904,
1640
+ "Nvidia H100 GPU": 0.09023110029341469
1641
+ },
1642
+ "mape_lomo": 0.08231965742393678,
1643
+ "per_lomo": {
1644
+ "llama2-7b": 0.13074280667924645,
1645
+ "llama31-8b": 0.05834596296526117,
1646
+ "mistral-7b": 0.0575047794778978,
1647
+ "qwen2-7b": 0.1109051767434791
1648
+ }
1649
+ }
1650
+ },
1651
+ "v3": {
1652
+ "eta": 0.7859524301863509,
1653
+ "t0": 0.0043992205190368965,
1654
+ "n_train": 10,
1655
+ "n_test": 18,
1656
+ "spearman": 0.9896800825593396,
1657
+ "median_ape": 0.19641776665455046,
1658
+ "mape": 0.19738789249080585,
1659
+ "table": [
1660
+ {
1661
+ "gpu": "3070 8GB",
1662
+ "cfg": "8B Q4_K_M",
1663
+ "tier": "consumer",
1664
+ "bw": 448,
1665
+ "value": 70.94,
1666
+ "pred": 57.30337895944639,
1667
+ "eta_obs": 1.057837468781738
1668
+ },
1669
+ {
1670
+ "gpu": "3080 10GB",
1671
+ "cfg": "8B Q4_K_M",
1672
+ "tier": "consumer",
1673
+ "bw": 760,
1674
+ "value": 106.4,
1675
+ "pred": 82.69323238120502,
1676
+ "eta_obs": 1.2095465451063712
1677
+ },
1678
+ {
1679
+ "gpu": "3080 Ti 12GB",
1680
+ "cfg": "8B Q4_K_M",
1681
+ "tier": "consumer",
1682
+ "bw": 912,
1683
+ "value": 106.71,
1684
+ "pred": 92.50171726743433,
1685
+ "eta_obs": 1.0134905810408072
1686
+ },
1687
+ {
1688
+ "gpu": "4070 Ti 12GB",
1689
+ "cfg": "8B Q4_K_M",
1690
+ "tier": "consumer",
1691
+ "bw": 504,
1692
+ "value": 82.21,
1693
+ "pred": 62.49694296870206,
1694
+ "eta_obs": 1.1743167745803165
1695
+ },
1696
+ {
1697
+ "gpu": "4080 16GB",
1698
+ "cfg": "8B Q4_K_M",
1699
+ "tier": "consumer",
1700
+ "bw": 717,
1701
+ "value": 106.22,
1702
+ "pred": 79.65402450867047,
1703
+ "eta_obs": 1.2780141109037009
1704
+ },
1705
+ {
1706
+ "gpu": "4080 16GB",
1707
+ "cfg": "8B F16",
1708
+ "tier": "consumer",
1709
+ "bw": 717,
1710
+ "value": 40.29,
1711
+ "pred": 32.20750358208973,
1712
+ "eta_obs": 1.0256777871584486
1713
+ },
1714
+ {
1715
+ "gpu": "RTX 4000 Ada 20GB",
1716
+ "cfg": "8B Q4_K_M",
1717
+ "tier": "workstation",
1718
+ "bw": 360,
1719
+ "value": 58.59,
1720
+ "pred": 48.44630874510202,
1721
+ "eta_obs": 1.0076605390698568
1722
+ },
1723
+ {
1724
+ "gpu": "RTX 4000 Ada 20GB",
1725
+ "cfg": "8B F16",
1726
+ "tier": "workstation",
1727
+ "bw": 360,
1728
+ "value": 20.85,
1729
+ "pred": 17.398556839598413,
1730
+ "eta_obs": 0.9576111230853452
1731
+ },
1732
+ {
1733
+ "gpu": "3090 24GB",
1734
+ "cfg": "8B Q4_K_M",
1735
+ "tier": "consumer",
1736
+ "bw": 936,
1737
+ "value": 111.74,
1738
+ "pred": 93.93009181577683,
1739
+ "eta_obs": 1.0790559863278681
1740
+ },
1741
+ {
1742
+ "gpu": "3090 24GB",
1743
+ "cfg": "8B F16",
1744
+ "tier": "consumer",
1745
+ "bw": 936,
1746
+ "value": 46.51,
1747
+ "pred": 40.300840020216285,
1748
+ "eta_obs": 0.9381941408146872
1749
+ },
1750
+ {
1751
+ "gpu": "4090 24GB",
1752
+ "cfg": "8B Q4_K_M",
1753
+ "tier": "consumer",
1754
+ "bw": 1008,
1755
+ "value": 127.74,
1756
+ "pred": 98.03920051264187,
1757
+ "eta_obs": 1.3295122291030945
1758
+ },
1759
+ {
1760
+ "gpu": "4090 24GB",
1761
+ "cfg": "8B F16",
1762
+ "tier": "consumer",
1763
+ "bw": 1008,
1764
+ "value": 54.34,
1765
+ "pred": 42.816972424501,
1766
+ "eta_obs": 1.0639191901645226
1767
+ },
1768
+ {
1769
+ "gpu": "RTX 5000 Ada 32GB",
1770
+ "cfg": "8B Q4_K_M",
1771
+ "tier": "workstation",
1772
+ "bw": 576,
1773
+ "value": 89.87,
1774
+ "pred": 68.72574792609842,
1775
+ "eta_obs": 1.1858702586005574
1776
+ },
1777
+ {
1778
+ "gpu": "RTX 5000 Ada 32GB",
1779
+ "cfg": "8B F16",
1780
+ "tier": "workstation",
1781
+ "bw": 576,
1782
+ "value": 32.67,
1783
+ "pred": 26.615403733029922,
1784
+ "eta_obs": 0.9947540562687923
1785
+ },
1786
+ {
1787
+ "gpu": "RTX A6000 48GB",
1788
+ "cfg": "8B Q4_K_M",
1789
+ "tier": "workstation",
1790
+ "bw": 768,
1791
+ "value": 102.22,
1792
+ "pred": 83.24491571868943,
1793
+ "eta_obs": 1.1114993026267512
1794
+ },
1795
+ {
1796
+ "gpu": "RTX A6000 48GB",
1797
+ "cfg": "8B F16",
1798
+ "tier": "workstation",
1799
+ "bw": 768,
1800
+ "value": 40.25,
1801
+ "pred": 34.154200347894026,
1802
+ "eta_obs": 0.9564111428518414
1803
+ },
1804
+ {
1805
+ "gpu": "RTX 6000 Ada 48GB",
1806
+ "cfg": "8B Q4_K_M",
1807
+ "tier": "workstation",
1808
+ "bw": 960,
1809
+ "value": 130.99,
1810
+ "pred": 95.32851540948249,
1811
+ "eta_obs": 1.479804889536415
1812
+ },
1813
+ {
1814
+ "gpu": "RTX 6000 Ada 48GB",
1815
+ "cfg": "8B F16",
1816
+ "tier": "workstation",
1817
+ "bw": 960,
1818
+ "value": 51.97,
1819
+ "pred": 41.14714179812888,
1820
+ "eta_obs": 1.0539521864715458
1821
+ },
1822
+ {
1823
+ "gpu": "A40 48GB",
1824
+ "cfg": "8B Q4_K_M",
1825
+ "tier": "datacenter",
1826
+ "bw": 696,
1827
+ "value": 88.95,
1828
+ "pred": 78.12285339856344,
1829
+ "eta_obs": 0.9649044022737184
1830
+ },
1831
+ {
1832
+ "gpu": "A40 48GB",
1833
+ "cfg": "8B F16",
1834
+ "tier": "datacenter",
1835
+ "bw": 696,
1836
+ "value": 33.95,
1837
+ "pred": 31.39447008008646,
1838
+ "eta_obs": 0.8611623236650897
1839
+ },
1840
+ {
1841
+ "gpu": "L40S 48GB",
1842
+ "cfg": "8B Q4_K_M",
1843
+ "tier": "datacenter",
1844
+ "bw": 864,
1845
+ "value": 113.6,
1846
+ "pred": 89.55118219341863,
1847
+ "eta_obs": 1.2078750724185845
1848
+ },
1849
+ {
1850
+ "gpu": "L40S 48GB",
1851
+ "cfg": "8B F16",
1852
+ "tier": "datacenter",
1853
+ "bw": 864,
1854
+ "value": 43.42,
1855
+ "pred": 37.715129354229504,
1856
+ "eta_obs": 0.9329078264936257
1857
+ },
1858
+ {
1859
+ "gpu": "A100 PCIe 80GB",
1860
+ "cfg": "8B Q4_K_M",
1861
+ "tier": "datacenter",
1862
+ "bw": 1935,
1863
+ "value": 138.31,
1864
+ "pred": 134.7523157254057,
1865
+ "eta_obs": 0.8389489151337184
1866
+ },
1867
+ {
1868
+ "gpu": "A100 PCIe 80GB",
1869
+ "cfg": "8B F16",
1870
+ "tier": "datacenter",
1871
+ "bw": 1935,
1872
+ "value": 54.56,
1873
+ "pred": 70.05756470343105,
1874
+ "eta_obs": 0.5571801713133513
1875
+ },
1876
+ {
1877
+ "gpu": "A100 SXM 80GB",
1878
+ "cfg": "8B Q4_K_M",
1879
+ "tier": "datacenter",
1880
+ "bw": 2039,
1881
+ "value": 133.38,
1882
+ "pred": 137.6103638065023,
1883
+ "eta_obs": 0.7274829914101165
1884
+ },
1885
+ {
1886
+ "gpu": "A100 SXM 80GB",
1887
+ "cfg": "8B F16",
1888
+ "tier": "datacenter",
1889
+ "bw": 2039,
1890
+ "value": 53.18,
1891
+ "pred": 72.62000513081995,
1892
+ "eta_obs": 0.511302461145103
1893
+ },
1894
+ {
1895
+ "gpu": "H100 PCIe 80GB",
1896
+ "cfg": "8B Q4_K_M",
1897
+ "tier": "datacenter",
1898
+ "bw": 2039,
1899
+ "value": 144.49,
1900
+ "pred": 137.6103638065023,
1901
+ "eta_obs": 0.8937934329551209
1902
+ },
1903
+ {
1904
+ "gpu": "H100 PCIe 80GB",
1905
+ "cfg": "8B F16",
1906
+ "tier": "datacenter",
1907
+ "bw": 2039,
1908
+ "value": 67.79,
1909
+ "pred": 72.62000513081995,
1910
+ "eta_obs": 0.711464047338404
1911
+ }
1912
+ ]
1913
+ },
1914
+ "argonne_rows": {
1915
+ "total": 400,
1916
+ "infeasible": 4,
1917
+ "pressured": 24
1918
+ }
1919
+ }
gpus.csv ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ gpu,label,tier,vram_gb,bw_gbs,fp16_tflops,tdp_w,msrp_usd,price_central_usd,price_low_usd,price_high_usd,host_usd,form,source_specs,notes
2
+ rtx3060,RTX 3060 12GB,budget,12,360,25.5,170,329,300,250,350,900,pcie,gpu.json:NVIDIA_GA106-300-A1,FP16 tensor halved for FP32 accumulate (GeForce)
3
+ rtx4060ti,RTX 4060 Ti 16GB,budget,16,288,44.0,165,499,450,400,500,900,pcie,gpu.json:NVIDIA_AD106-351,FP16 tensor halved for FP32 accumulate (GeForce)
4
+ rtx5060ti,RTX 5060 Ti 16GB,budget,16,448,47.5,180,429,450,429,550,900,pcie,gpu.json:NVIDIA_GB206-300_2,FP16 tensor halved for FP32 accumulate (GeForce; assumed as Ada)
5
+ rtx3090,RTX 3090 24GB,enthusiast,24,936,71.2,350,1499,800,650,1000,1200,pcie,gpu.json:NVIDIA_GA102-300-A1,price is used-market assumption; FP16 halved for FP32 accumulate
6
+ rtx4090,RTX 4090 24GB,enthusiast,24,1008,165.2,450,1599,1800,1599,2500,1200,pcie,gpu.json:NVIDIA_AD102-300,FP16 halved for FP32 accumulate
7
+ rtx5090,RTX 5090 32GB,enthusiast,32,1792,209.5,575,1999,2500,1999,3500,1500,pcie,gpu.json:NVIDIA_GB202-300,FP16 halved for FP32 accumulate (assumed as Ada)
8
+ a6000,RTX A6000 48GB,workstation,48,768,154.8,300,4649,4500,3500,5000,2500,pcie,gpu.json:NVIDIA_GA102-875-A1,dense = sparse/2
9
+ rtx6000ada,RTX 6000 Ada 48GB,workstation,48,960,364.2,300,6799,7000,6000,7500,2500,pcie,gpu.json:NVIDIA_AD102-870-A1 + manual FP16,gpu.json FP16 field is FP8-sparse; FP16 dense from vendor datasheet (verify)
10
+ rtxpro6000,RTX PRO 6000 Blackwell 96GB,workstation,96,1792,312.5,600,8565,8565,7500,9500,2500,pcie,gpu.json:NVIDIA_GB202_3,dense = listed sparse/2 (verify)
11
+ t4,T4 16GB,dc-inference,16,320,64.8,70,,1500,1000,2500,4000,pcie,gpu.json:NVIDIA_nan_56,no MSRP; price assumed
12
+ l4,L4 24GB,dc-inference,24,300,121.0,72,,2500,2000,3500,4000,pcie,gpu.json:NVIDIA_nan_69,no MSRP; price assumed
13
+ a10,A10/A10G 24GB,dc-inference,24,600,125.0,150,,3000,2000,4000,4000,pcie,gpu.json:NVIDIA_nan_58,A10G (AWS) treated as A10; price assumed
14
+ l40s,L40S 48GB,dc-inference,48,864,362.0,350,,8000,7000,10000,4000,pcie,manual (vendor datasheet; verify),not in gpu.json; price assumed
15
+ a100,A100 SXM 80GB,flagship,80,2039,312.0,400,,12000,9000,17000,8000,sxm,manual bandwidth for 80GB SXM + gpu.json:NVIDIA_nan_62 compute,gpu.json lists 40GB PCIe (1555 GB/s); price assumed
16
+ h100,H100 SXM 80GB,flagship,80,3352,989.4,700,,25000,20000,35000,8000,sxm,gpu.json:NVIDIA_nan_64,price assumed
17
+ h200,H200 SXM 141GB,flagship,141,4800,989.4,700,,32000,28000,40000,8000,sxm,gpu.json:NVIDIA_nan_66 + manual bandwidth,gpu.json lists 3360 GB/s; vendor datasheet 4800 GB/s used (verify)
18
+ b200,B200 180GB,flagship,180,8000,2250.0,1000,,40000,30000,50000,8000,sxm,gpu.json:NVIDIA_nan_71 + manual,gpu.json lists 2x96 GB / 2x4100 GB/s; 180 GB usable and 8 TB/s per vendor DGX spec (verify)
models.csv ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ model,label,layers,d_model,d_ff,n_q,n_kv,head_dim,vocab,tied,source
2
+ qwen3-4b,Qwen3-4B,36,2560,9728,32,8,128,151936,1,model card (verify)
3
+ llama31-8b,Llama-3.1-8B,32,4096,14336,32,8,128,128256,0,vidur model_config.py (Llama3_8B)
4
+ qwen25-14b,Qwen2.5-14B,48,5120,13824,40,8,128,152064,0,model card (verify)
5
+ qwen25-32b,Qwen2.5-32B,64,5120,27648,40,8,128,152064,0,model card (verify)
6
+ llama31-70b,Llama-3.1-70B,80,8192,28672,64,8,128,128256,0,vidur model_config.py (Llama3_70B)
7
+ llama2-7b,Llama-2-7B,32,4096,11008,32,32,128,32000,0,vidur model_config.py (Llama2_7B)
8
+ llama2-70b,Llama-2-70B,80,8192,28672,64,8,128,32000,0,vidur model_config.py (Llama2_70B)
9
+ mistral-7b,Mistral-7B,32,4096,14336,32,8,128,32000,0,model card (verify)
10
+ qwen2-7b,Qwen2-7B,28,3584,18944,28,4,128,152064,0,model card (verify)