measured and certified numbers
Browse files
README.md
CHANGED
|
@@ -132,22 +132,22 @@ SmolLM2-135M bf16:
|
|
| 132 |
|
| 133 |
| GPU | this kernel | eager | compiled static |
|
| 134 |
|---|---|---|---|
|
| 135 |
-
| RTX 6000 Ada (sm89) |
|
| 136 |
-
| H200 (sm90) |
|
| 137 |
-
| RTX PRO 6000 (sm120) |
|
| 138 |
|
| 139 |
Llama-3.2-1B bf16:
|
| 140 |
|
| 141 |
| GPU | this kernel | eager | compiled static |
|
| 142 |
|---|---|---|---|
|
| 143 |
-
| H200 (sm90) |
|
| 144 |
-
| RTX PRO 6000 (sm120) |
|
| 145 |
|
| 146 |
-
Steady-state decode at position 512 on Llama-3.2-1B: 1.
|
| 147 |
-
H200 (1.
|
| 148 |
-
3.
|
| 149 |
|
| 150 |
-
Prefill: a 2040-token prompt costs 0.
|
| 151 |
token (RTX 6000 Ada, SmolLM2), bitwise-identical logits either way. An
|
| 152 |
eager greedy generation of SmolLM2 is roughly 180 kernel launches per
|
| 153 |
token; here the entire generation is one.
|
|
|
|
| 132 |
|
| 133 |
| GPU | this kernel | eager | compiled static |
|
| 134 |
|---|---|---|---|
|
| 135 |
+
| RTX 6000 Ada (sm89) | 956 tok/s | 44 | n/a (Windows) |
|
| 136 |
+
| H200 (sm90) | 766 tok/s | 53 | 582 |
|
| 137 |
+
| RTX PRO 6000 (sm120) | 784 tok/s | 56 | 727 |
|
| 138 |
|
| 139 |
Llama-3.2-1B bf16:
|
| 140 |
|
| 141 |
| GPU | this kernel | eager | compiled static |
|
| 142 |
|---|---|---|---|
|
| 143 |
+
| H200 (sm90) | 534 tok/s | 79 | 609 |
|
| 144 |
+
| RTX PRO 6000 (sm120) | 386 tok/s | 93 | 400 |
|
| 145 |
|
| 146 |
+
Steady-state decode at position 512 on Llama-3.2-1B: 1.70 ms/token on
|
| 147 |
+
H200 (1.77 TB/s effective weight bandwidth), 2.23 ms on RTX PRO 6000,
|
| 148 |
+
3.76 ms on RTX 6000 Ada (0.80 TB/s, 83 percent of the card).
|
| 149 |
|
| 150 |
+
Prefill: a 2040-token prompt costs 0.73 s against 2.7 s consumed token by
|
| 151 |
token (RTX 6000 Ada, SmolLM2), bitwise-identical logits either way. An
|
| 152 |
eager greedy generation of SmolLM2 is roughly 180 kernel launches per
|
| 153 |
token; here the entire generation is one.
|