Schroding18 commited on
Commit
981c4aa
·
verified ·
1 Parent(s): f5019ff

lead with what is new; closing novelty section; int2 strip

Browse files
Files changed (1) hide show
  1. index.html +9 -0
index.html CHANGED
@@ -21,6 +21,7 @@ svg .lab{font-size:12px;fill:#333} svg .val{font-size:12px;fill:#111;font-weight
21
  <h1>Run it twice while it's still in L2: cheap extra quality for LLMs</h1>
22
 
23
  <p><strong>When a layer fits in L2, running it again costs about half a fresh layer (0.49× on an RTX 5080). At equal parameters, that extra pass lowers loss by 0.05–0.08.</strong></p>
 
24
  <blockquote>
25
  <p><strong>L2Loop has evidence for trading compute for precision and memory, but not yet for trading compute for parameter diversity.</strong></p>
26
  </blockquote>
@@ -1074,6 +1075,14 @@ svg .lab{font-size:12px;fill:#333} svg .val{font-size:12px;fill:#111;font-weight
1074
  </tbody>
1075
  </table>
1076
  <p>Retrofitting the loop onto a finished model gives little so far.</p>
 
 
 
 
 
 
 
 
1077
  <h2>8. Takeaways</h2>
1078
  <ol>
1079
  <li><strong>Compute trades for precision and memory, not (yet) for parameter diversity.</strong> Bits and loops trade against each other. From scratch, looping twice gains 0.05–0.08 at equal parameters, the gain adds to GPTQ's, and int3 + 3 loops matches a larger non-looped int4 model at ~0.55× block bytes. Two passes are the sweet spot; beyond that, add blocks.</li>
 
21
  <h1>Run it twice while it's still in L2: cheap extra quality for LLMs</h1>
22
 
23
  <p><strong>When a layer fits in L2, running it again costs about half a fresh layer (0.49× on an RTX 5080). At equal parameters, that extra pass lowers loss by 0.05–0.08.</strong></p>
24
+ <p><strong>What is new.</strong> We treat the GPU's L2 cache as a design target. Quantize the repeated layers until they fit in L2, run them again while they are still there, and use that cheap second pass to win back the quality the low bit-width cost. We measure this loop end to end: a controlled sweep of bit-width × loop count at equal weight bytes (two seeds, int3–int8, 1–6 loops), the cache cliff on GPUs with different L2 sizes, and the same idea applied to MoE experts.</p>
25
  <blockquote>
26
  <p><strong>L2Loop has evidence for trading compute for precision and memory, but not yet for trading compute for parameter diversity.</strong></p>
27
  </blockquote>
 
1075
  </tbody>
1076
  </table>
1077
  <p>Retrofitting the loop onto a finished model gives little so far.</p>
1078
+ <h2>What is new here</h2>
1079
+ <p>Running a layer again while its weights sit in cache is cheap; that observation alone is not ours. What this post adds is turning it into a design rule and measuring the whole loop:</p>
1080
+ <ol>
1081
+ <li><strong>Quantize to fit L2, then loop to recover quality.</strong> Lower bits are used as the means to get the repeated unit into L2, and the cached repeat is used to win back what the low bit-width cost. int3 + looping matches a larger int4 model with 0.55× the weight bytes.</li>
1082
+ <li><strong>Bits and loops as one budget.</strong> A controlled sweep of bit-width × loop count at equal weight bytes, two seeds: the looped frontier lies below the standard one at every byte budget we tested, most of all when bytes are scarce.</li>
1083
+ <li><strong>The cache cliff, measured across GPUs.</strong> The same layer costs 0.49× a fresh one where it fits in L2 (RTX 5080) and 1.00× where it does not (RTX 5060 Ti); expert-level repeats on four GPUs follow the same rule.</li>
1084
+ <li><strong>MoE.</strong> Per-expert repeat cost on four GPUs, whole routed-layer latency for standard, repeated, half-expert, double-expert and wide-expert MoEs, and from-scratch quality runs.</li>
1085
+ </ol>
1086
  <h2>8. Takeaways</h2>
1087
  <ol>
1088
  <li><strong>Compute trades for precision and memory, not (yet) for parameter diversity.</strong> Bits and loops trade against each other. From scratch, looping twice gains 0.05–0.08 at equal parameters, the gain adds to GPTQ's, and int3 + 3 loops matches a larger non-looped int4 model at ~0.55× block bytes. Two passes are the sweet spot; beyond that, add blocks.</li>