3.5: explain why the slowdown concentrates in prefill
Browse files- index.html +11 -1
index.html
CHANGED
|
@@ -757,7 +757,17 @@ same-size d4 sibling — the measured price of the quality its geometry
|
|
| 757 |
buys, and one that may differ on hardware with a different memory
|
| 758 |
hierarchy. Against affine,
|
| 759 |
VQ prefill remains ~0.5x at 35B scale even after the zero-copy lever;
|
| 760 |
-
decode is within 10–20%.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 761 |
between arms: we found decode throughput at ~100 GiB residency to be
|
| 762 |
bimodal on our hardware (the same artifact varying 40% run to run, with
|
| 763 |
swap, thermals and storage path each ruled out by measurement), we have
|
|
|
|
| 757 |
buys, and one that may differ on hardware with a different memory
|
| 758 |
hierarchy. Against affine,
|
| 759 |
VQ prefill remains ~0.5x at 35B scale even after the zero-copy lever;
|
| 760 |
+
decode is within 10–20%. The asymmetry is the signature of where each
|
| 761 |
+
phase's time goes. Decode generates one token at a time and is bound by
|
| 762 |
+
memory bandwidth — the cost of reading the weights — and a VQ artifact
|
| 763 |
+
has fewer bytes to read, so the extra arithmetic of in-kernel codebook
|
| 764 |
+
lookups hides behind the memory traffic and decode stays near parity.
|
| 765 |
+
Prefill processes the whole prompt as large matrix multiplies and is
|
| 766 |
+
bound by arithmetic throughput; there the same per-weight decode work is
|
| 767 |
+
added to a compute-saturated path with no bandwidth saving to pay for
|
| 768 |
+
it, and it surfaces as the 2x gap. This is an interpretation consistent
|
| 769 |
+
with the measured split rather than a profiled attribution; what is
|
| 770 |
+
measured is the pair of ratios. Speed numbers here are same-session ratios
|
| 771 |
between arms: we found decode throughput at ~100 GiB residency to be
|
| 772 |
bimodal on our hardware (the same artifact varying 40% run to run, with
|
| 773 |
swap, thermals and storage path each ruled out by measurement), we have
|