TheDrainFlorist commited on
Commit
cc221eb
·
verified ·
1 Parent(s): 8287e33

3.5: explain why the slowdown concentrates in prefill

Browse files
Files changed (1) hide show
  1. index.html +11 -1
index.html CHANGED
@@ -757,7 +757,17 @@ same-size d4 sibling — the measured price of the quality its geometry
757
  buys, and one that may differ on hardware with a different memory
758
  hierarchy. Against affine,
759
  VQ prefill remains ~0.5x at 35B scale even after the zero-copy lever;
760
- decode is within 10–20%. Speed numbers here are same-session ratios
 
 
 
 
 
 
 
 
 
 
761
  between arms: we found decode throughput at ~100 GiB residency to be
762
  bimodal on our hardware (the same artifact varying 40% run to run, with
763
  swap, thermals and storage path each ruled out by measurement), we have
 
757
  buys, and one that may differ on hardware with a different memory
758
  hierarchy. Against affine,
759
  VQ prefill remains ~0.5x at 35B scale even after the zero-copy lever;
760
+ decode is within 10–20%. The asymmetry is the signature of where each
761
+ phase's time goes. Decode generates one token at a time and is bound by
762
+ memory bandwidth — the cost of reading the weights — and a VQ artifact
763
+ has fewer bytes to read, so the extra arithmetic of in-kernel codebook
764
+ lookups hides behind the memory traffic and decode stays near parity.
765
+ Prefill processes the whole prompt as large matrix multiplies and is
766
+ bound by arithmetic throughput; there the same per-weight decode work is
767
+ added to a compute-saturated path with no bandwidth saving to pay for
768
+ it, and it surfaces as the 2x gap. This is an interpretation consistent
769
+ with the measured split rather than a profiled attribution; what is
770
+ measured is the pair of ratios. Speed numbers here are same-session ratios
771
  between arms: we found decode throughput at ~100 GiB residency to be
772
  bimodal on our hardware (the same artifact varying 40% run to run, with
773
  swap, thermals and storage path each ruled out by measurement), we have