Schroding18 commited on
Commit
c049c4f
·
verified ·
1 Parent(s): 31bd5ee

Pareto figure: equal-bytes loop vs standard frontier

Browse files
Files changed (2) hide show
  1. fig_pareto.png +0 -0
  2. index.html +2 -0
fig_pareto.png ADDED
index.html CHANGED
@@ -695,6 +695,8 @@ svg .lab{font-size:12px;fill:#333} svg .val{font-size:12px;fill:#111;font-weight
695
  <p>The 7900 GRE has a 6 MB L2 and a 64 MB Infinity Cache; the 35 MB expert is resident in the latter. RTX 5080/5060 Ti and GB10 measurements are running. Experts larger than the cache can be split along the FFN dimension and each slice repeated immediately; this is being measured.</p>
696
  </li>
697
  </ul>
 
 
698
  <p><strong>The hypothesis.</strong> At batch size 1, per-token cost is set by the bytes streamed from memory, not by the arithmetic. Once an expert has been streamed in, more computation on it costs about half price for as long as it stays in cache. Models are designed as if each weight should be used once per token; at batch 1 that allocation may not be optimal. We think the right axis is <strong>quality per byte read</strong>: a MoE that loads k experts and reuses each of them in cache may beat one that loads more distinct experts for the same memory traffic.</p>
699
  <p><strong>What would confirm it.</strong> Same hardware and the same active experts (k fixed), each expert run twice in cache versus once, compared on quality and per-token time; then the same at equal bytes read (fewer experts plus repeats versus more experts). A pretrained OLMoE-1B-7B is being tested now, training-free and with a gate-only fine-tune against a matched no-repeat control. If cache reuse wins at equal bytes, the next step is training small MoEs from scratch both ways.</p>
700
  <h2>8. Takeaways</h2>
 
695
  <p>The 7900 GRE has a 6 MB L2 and a 64 MB Infinity Cache; the 35 MB expert is resident in the latter. RTX 5080/5060 Ti and GB10 measurements are running. Experts larger than the cache can be split along the FFN dimension and each slice repeated immediately; this is being measured.</p>
696
  </li>
697
  </ul>
698
+ <p><img alt="At equal bytes read, looping the same weights beats loading more weights" src="https://huggingface.co/spaces/Schroding18/loop-in-L2/resolve/main/fig_pareto.png" /></p>
699
+ <p><em>Validation loss against block-weight bytes read per token, from-scratch models (seed 1; seed 0 has the same ordering). Gray: standard models with more distinct blocks; purple: the same 6 blocks run 2–4 times. Extra passes add no bytes, so each looped configuration is a vertical column. At equal bytes the loop frontier lies well below the standard frontier: 6 blocks int4 × 4 passes matches 12 blocks int4 with half the bytes. Extra passes cost time; at equal latency the two families tie (Section 4).</em></p>
700
  <p><strong>The hypothesis.</strong> At batch size 1, per-token cost is set by the bytes streamed from memory, not by the arithmetic. Once an expert has been streamed in, more computation on it costs about half price for as long as it stays in cache. Models are designed as if each weight should be used once per token; at batch 1 that allocation may not be optimal. We think the right axis is <strong>quality per byte read</strong>: a MoE that loads k experts and reuses each of them in cache may beat one that loads more distinct experts for the same memory traffic.</p>
701
  <p><strong>What would confirm it.</strong> Same hardware and the same active experts (k fixed), each expert run twice in cache versus once, compared on quality and per-token time; then the same at equal bytes read (fewer experts plus repeats versus more experts). A pretrained OLMoE-1B-7B is being tested now, training-free and with a gate-only fine-tune against a matched no-repeat control. If cache reuse wins at equal bytes, the next step is training small MoEs from scratch both ways.</p>
702
  <h2>8. Takeaways</h2>