Spaces:
Running
Running
File size: 16,677 Bytes
13be905 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 | <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>We're back - and cloud GPUs! | SupraLabs Blog</title>
<style>
:root {
--bg: #0f0f0f;
--surface: #1a1a1a;
--border: #333;
--text: #e0e0e0;
--accent: #536bfe;
--muted: #888;
--font-mono: 'JetBrains Mono', 'Fira Code', monospace;
}
* { margin: 0; padding: 0; box-sizing: border-box; }
body {
background-color: var(--bg);
color: var(--text);
font-family: 'Inter', -apple-system, sans-serif;
line-height: 1.6;
padding: 2rem;
}
code, pre, .mono { font-family: var(--font-mono); }
.container { max-width: 900px; margin: 0 auto; }
header {
border-bottom: 2px solid var(--border);
padding-bottom: 2rem;
margin-bottom: 3rem;
display: flex;
justify-content: space-between;
align-items: flex-end;
}
.logo-area h1 {
font-size: 1.2rem;
text-transform: uppercase;
letter-spacing: 2px;
color: var(--accent);
line-height: 1;
display: flex;
align-items: center;
gap: 10px;
}
.logo-area a { text-decoration: none; color: inherit; }
.logo-area { display: flex; align-items: center; gap: 10px; font-weight: bold; font-size: 1.2rem; }
nav a {
color: var(--text);
text-decoration: none;
margin-left: 1.5rem;
font-size: 0.9rem;
border-bottom: 1px solid transparent;
}
nav a:hover { border-bottom: 1px solid var(--accent); }
.post-header { margin-bottom: 3rem; }
.post-header h2 { font-size: 3rem; line-height: 1.1; margin-bottom: 1rem; font-weight: 800; }
.post-meta { font-family: var(--font-mono); color: var(--accent); font-size: 0.9rem; margin-bottom: 2rem; }
.post-content {
background: var(--surface);
border: 1px solid var(--border);
padding: 3rem;
margin-bottom: 4rem;
}
.post-content h2 { font-size: 1.8rem; margin: 2.5rem 0 1rem 0; color: var(--accent); }
.post-content h2:first-child { margin-top: 0; }
.post-content p { margin-bottom: 1.5rem; font-size: 1.1rem; color: var(--text); }
.post-content ul { margin-bottom: 1.5rem; padding-left: 1.5rem; }
.post-content li { margin-bottom: 0.5rem; font-size: 1.1rem; }
.post-content strong { color: #fff; }
.post-content code {
background: #111;
border: 1px solid var(--border);
padding: 2px 6px;
border-radius: 3px;
font-size: 0.95em;
color: var(--accent);
}
.callout {
border-left: 3px solid var(--accent);
background: #111;
padding: 1rem 1.5rem;
margin: 2rem 0;
font-family: var(--font-mono);
font-size: 0.95rem;
color: #ccc;
}
.callout span { display: block; color: var(--muted); font-size: 0.8rem; margin-bottom: 0.4rem; }
/* IO example pairs */
.io-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 1rem; margin: 2rem 0; }
.io-pair {
background: #111;
border: 1px solid var(--border);
padding: 1.2rem;
}
.io-pair .io-label {
font-family: var(--font-mono);
font-size: 0.72rem;
color: var(--muted);
margin-bottom: 0.5rem;
}
.io-pair .io-user { color: #ccc; font-size: 0.95rem; margin-bottom: 0.8rem; line-height: 1.5; }
.io-pair .io-result {
font-family: var(--font-mono);
color: var(--accent);
font-size: 0.9rem;
font-weight: 700;
}
/* Flow diagram */
.flow {
display: flex;
align-items: center;
justify-content: center;
gap: 0.8rem;
margin: 2rem 0;
flex-wrap: wrap;
}
.flow-box {
background: #111;
border: 1px solid var(--border);
padding: 0.8rem 1.2rem;
font-family: var(--font-mono);
font-size: 0.82rem;
text-align: center;
color: #ccc;
}
.flow-box.accent { border-color: var(--accent); color: var(--accent); }
.flow-arrow { color: var(--accent); font-size: 1.2rem; }
.table-wrap { overflow-x: auto; margin: 2rem 0; }
table { width: 100%; border-collapse: collapse; font-family: var(--font-mono); font-size: 0.88rem; }
th { background: #111; color: var(--accent); padding: 0.75rem 1rem; text-align: left; border: 1px solid var(--border); }
td { padding: 0.7rem 1rem; border: 1px solid var(--border); color: var(--text); }
tr:nth-child(even) td { background: #111; }
.tags { display: flex; gap: 0.5rem; margin-top: 2rem; flex-wrap: wrap; }
.tag { font-family: var(--font-mono); font-size: 0.7rem; padding: 2px 8px; border: 1px solid var(--border); border-radius: 4px; color: var(--muted); }
footer { margin-top: 6rem; padding-bottom: 2rem; font-size: 0.8rem; color: var(--muted); text-align: center; }
@media (max-width: 600px) {
.post-header h2 { font-size: 2rem; }
.post-content { padding: 1.5rem; }
header { flex-direction: column; align-items: flex-start; gap: 1rem; }
nav a { margin-left: 0; margin-right: 1rem; }
.io-grid { grid-template-columns: 1fr; }
.flow { flex-direction: column; }
}
</style>
</head>
<body>
<div class="container">
<header>
<div class="logo-area" style="font-size: 1.5em;">
<a href="./index.html"><h1><img src="./image.png" style="height: 2em"> SupraLabs_</h1></a>
</div>
<nav>
<a href="./index.html#news">News</a>
<a href="https://huggingface.co/SupraLabs" target="blank">HuggingFace</a>
<a href="./index.html#hardware">Hardware</a>
</nav>
</header>
<article>
<div class="post-header">
<div class="post-meta">// 2026-08-29 | Research</div>
<h2>We're back.<br>And we audited<br>cloud GPU math.</h2>
</div>
<div class="post-content">
<p>This blog was dark for a while. Not dead - parked. We are back. First post after the pause is not a launch. It is a spreadsheet with teeth.</p>
<p>Cloud GPU listings mix <strong>dense</strong> and <strong>sparse</strong> BF16 numbers like they are the same unit. They are not. NVIDIA datasheets often lead with 2:4 sparsity (roughly 2x dense). Sort by that mix and you pick the wrong card for a long run 😭. We re-checked VRAM, memory bandwidth, architecture, and BF16 dense TFLOPS against vendor datasheets. Hourly prices were already trusted (in Runpod). Everything else got rebuilt.</p>
<h2>The dense / sparse trap</h2>
<p>Training almost never uses structured sparsity the way the marketing peak assumes. If you pay for FLOPs, you should compare <strong>BF16 dense (FP32 accumulate)</strong>. Sparse peaks belong in a footnote, not in the ranking column.</p>
<p>Biggest corrections from the first pass:</p>
<ul>
<li>RTX PRO 4000 is ~161 dense TFLOPS, not ~358. Bandwidth is 672 GB/s.</li>
<li>RTX 5090 419 is sparse; dense is 209.5.</li>
<li>H100 NVL is 835.5 dense, not the SXM 989 number copied across SKUs.</li>
<li>L40 dense is 181, not the L40S 362 figure.</li>
<li>H200 is 141 GB, not 143. SXM and NVL share the same die and HBM3e stack.</li>
<li>B200 bandwidth is 7.7 TB/s (often rounded to 8). B300 keeps ~2250 BF16 dense; the Ultra bump is mostly FP4 and 288 GB.</li>
<li>MI300X 1307 BF16 dense was already right - and that moves it to the top of $/FLOP once everyone else is densified.</li>
</ul>
<p>Ada workstation cards were the worst offenders: Tensor numbers in the PDF are often FP8-with-sparsity. Divide by 8 and you get a usable BF16 dense estimate (RTX 4000 Ada ~41, RTX 2000 Ada ~24).</p>
<h2>How to pick a GPU for a long run</h2>
<p>Fixed work W (tokens, epochs, whatever). Price p in $/h. Peak dense BF16 F. Bandwidth B. Arithmetic intensity I (FLOPs per byte). Utilization η (MFU, often 0.3-0.5 in training).</p>
<div class="callout">
<span>// cost</span>
cost = p · W / F_eff<br>
F_eff = η · min(F, I · B)<br>
pick argmin p / F_eff subject to VRAM ≥ model + opt + acts
</div>
<p>That collapses to two sorts:</p>
<ul>
<li><strong>Compute-bound</strong> (big-batch training): minimize <strong>$ per BF16 TFLOP</strong>.</li>
<li><strong>Memory-bound</strong> (decode, tiny batches): minimize <strong>$ per TB/s</strong>.</li>
</ul>
<p>If the model does not fit, multiply by a parallel tax. PCIe without NVLink is ugly: TP=2 on a 70B BF16 decode can lose 35-55% vs a single fat card. Consumer GDDR also has no ECC - a bit flip in a 48-hour run is a real failure mode. Checkpoint.</p>
<p>Peak is not MFU. L40S looks cheap on 362 TFLOPS sitting on 0.86 TB/s until the kernel is bandwidth-starved. MI300X wins on paper if ROCm actually delivers; measured FP16/BF16 is often 45-85% of peak depending on the stack.</p>
<h2>The table</h2>
<p>Hourly $ kept as-is. Specs renormalized to BF16 <strong>dense</strong>. Sorted by $ per BF16 TFLOP. Scroll sideways if your viewport is not a cinema screen ... lol.</p>
<div class="table-wrap">
<table>
<thead>
<tr><th>GPU</th><th>Architecture</th><th>VRAM</th><th>Data throughput in TB/s</th><th>BF16 TFLOPS</th><th>Price in $ per hour</th><th>Price in $ per TB data throughput</th><th>Price in $ per BF16 TFLOP</th></tr>
</thead>
<tbody>
<tr><td>MI300X</td><td>CDNA 3</td><td>192 GB</td><td>5.3</td><td>1307</td><td>2.39</td><td>0.000125</td><td>0.00000051</td></tr>
<tr><td>RTX A5000</td><td>Ampere</td><td>24 GB</td><td>1</td><td>111</td><td>0.27</td><td>0.000098</td><td>0.00000068</td></tr>
<tr><td>RTX A4500</td><td>Ampere</td><td>20 GB</td><td>0.64</td><td>~95</td><td>0.25</td><td>0.000109</td><td>0.00000073</td></tr>
<tr><td>L40S</td><td>Ada Lovelace</td><td>48 GB</td><td>1</td><td>362</td><td>0.99</td><td>0.000318</td><td>0.00000076</td></tr>
<tr><td>A40</td><td>Ampere</td><td>48 GB</td><td>1</td><td>150</td><td>0.44</td><td>0.000176</td><td>0.00000082</td></tr>
<tr><td>B200</td><td>Blackwell</td><td>180 GB</td><td>7.7</td><td>2250</td><td>6.79</td><td>0.000245</td><td>0.00000084</td></tr>
<tr><td>RTX A4000</td><td>Ampere</td><td>16 GB</td><td>0</td><td>~77</td><td>0.25</td><td>0.000155</td><td>0.00000091</td></tr>
<tr><td>H100 SXM</td><td>Hopper</td><td>80 GB</td><td>3.35</td><td>989</td><td>3.29</td><td>0.000273</td><td>0.00000092</td></tr>
<tr><td>RTX PRO 4500 (+SE)</td><td>Blackwell</td><td>32 GB</td><td>1</td><td>~215</td><td>0.72</td><td>0.000223</td><td>0.00000093</td></tr>
<tr><td>RTX A6000</td><td>Ampere</td><td>48 GB</td><td>1</td><td>155</td><td>0.53</td><td>0.000192</td><td>0.00000095</td></tr>
<tr><td>B300</td><td>Blackwell Ultra</td><td>288 GB</td><td>8</td><td>2250</td><td>7.89</td><td>0.000274</td><td>0.00000097</td></tr>
<tr><td>RTX PRO 4000</td><td>Blackwell</td><td>24 GB</td><td>1</td><td>~161</td><td>0.57</td><td>0.000236</td><td>0.00000098</td></tr>
<tr><td>RTX PRO 6000 WK</td><td>Blackwell</td><td>96 GB</td><td>1.79</td><td>~500</td><td>1.89</td><td>0.000293</td><td>0.00000105</td></tr>
<tr><td>H100 NVL</td><td>Hopper</td><td>94 GB</td><td>3.9</td><td>836</td><td>3.19</td><td>0.000227</td><td>0.00000106</td></tr>
<tr><td>H100 PCIe</td><td>Hopper</td><td>80 GB</td><td>2</td><td>756</td><td>2.89</td><td>0.000401</td><td>0.00000106</td></tr>
<tr><td>H200 NVL</td><td>Hopper</td><td>141 GB</td><td>4.8</td><td>989</td><td>3.79</td><td>0.000219</td><td>0.00000106</td></tr>
<tr><td>L4</td><td>Ada Lovelace</td><td>24 GB</td><td>0.3</td><td>121</td><td>0.49</td><td>0.000454</td><td>0.00000112</td></tr>
<tr><td>RTX PRO 6000 SE</td><td>Blackwell</td><td>96 GB</td><td>1.79</td><td>~500</td><td>2.09</td><td>0.000324</td><td>0.00000116</td></tr>
<tr><td>PRO 6000 MIG 48GB</td><td>Blackwell (MIG)</td><td>48 GB</td><td>~0.90</td><td>~250</td><td>1.09</td><td>0.000338</td><td>0.00000121</td></tr>
<tr><td>A100 PCIe</td><td>Ampere</td><td>80 GB</td><td>1.94</td><td>312</td><td>1.39</td><td>0.000200</td><td>0.00000124</td></tr>
<tr><td>RTX 4090</td><td>Ada Lovelace</td><td>24 GB</td><td>1.01</td><td>165</td><td>0.74</td><td>0.000204</td><td>0.00000124</td></tr>
<tr><td>L40</td><td>Ada Lovelace</td><td>48 GB</td><td>1</td><td>181</td><td>0.82</td><td>0.000264</td><td>0.00000126</td></tr>
<tr><td>RTX 6000 Ada</td><td>Ada Lovelace</td><td>48 GB</td><td>0.96</td><td>182.5</td><td>0.84</td><td>0.000243</td><td>0.00000128</td></tr>
<tr><td>H200 SXM</td><td>Hopper</td><td>141 GB</td><td>4.8</td><td>989</td><td>4.59</td><td>0.000266</td><td>0.00000129</td></tr>
<tr><td>PRO 6000 MIG 24GB</td><td>Blackwell (MIG)</td><td>24 GB</td><td>~0.45</td><td>~125</td><td>0.59</td><td>0.000366</td><td>0.00000131</td></tr>
<tr><td>RTX 5090</td><td>Blackwell</td><td>32 GB</td><td>1.79</td><td>209.5</td><td>0.99</td><td>0.000153</td><td>0.00000131</td></tr>
<tr><td>A100 SXM</td><td>Ampere</td><td>80 GB</td><td>2.04</td><td>312</td><td>1.59</td><td>0.000217</td><td>0.00000142</td></tr>
<tr><td>RTX 4000 Ada</td><td>Ada Lovelace</td><td>20 GB</td><td>0.36</td><td>~41</td><td>0.28</td><td>0.000216</td><td>0.00000190</td></tr>
<tr><td>RTX 3090</td><td>Ampere</td><td>24 GB</td><td>1</td><td>71</td><td>0.5</td><td>0.000148</td><td>0.00000196</td></tr>
<tr><td>RTX 2000 Ada</td><td>Ada Lovelace</td><td>16 GB</td><td>0</td><td>~24</td><td>0.24</td><td>0.000298</td><td>0.00000278</td></tr>
</tbody>
</table>
</div>
<h2>What we would actually rent</h2>
<p>Jobs that fit in 20-24 GB: A4500 / A5000 still crush $/FLOP. Real training: MI300X if the software stack is ROCm-shaped, otherwise B200 or H100 SXM among NVIDIA. H200 NVL is the sleeper - H100-SXM compute, 141 GB, 4.8 TB/s, almost the same $ per dense TFLOP as H100 PCIe/NVL.</p>
<p>Do not sort sparse marketing peaks and call it research. We did that once. Then we fixed it. Life is crazy 😭😂</p>
<div class="callout">
<span>// notes</span>
Prices --> as provided ($/h)<br>
FLOPS --> BF16 dense, vendor datasheets<br>
BW / VRAM --> NVIDIA / AMD product pages<br>
Formula --> roofline, not vibes
</div>
<div class="tags">
<span class="tag">#we're-back</span>
<span class="tag">#gpu</span>
<span class="tag">#bf16</span>
<span class="tag">#roofline</span>
<span class="tag">#cloud</span>
<span class="tag">#mi300x</span>
<span class="tag">#blackwell</span>
<span class="tag">#research</span>
</div>
</div>
</article>
<footer>
<p class="mono">© 2026 SupraLabs // Built for the community.</p>
</footer>
</div>
</body>
</html> |