Loop it in L2
Re-run L2-resident layers to undo low-bit quantization loss
int3 MoE beats full-precision MoE.
Run each MoE layer twice while its weights are still in the GPU's L2 cache. With int3 experts, the model beats a full-precision standard MoE of the same size, and reads only 22% of its expert bytes per token.
| result (8 layers, d768, 670M params, 524M tokens) | measured |
|---|---|
| 4.0% lower perplexity | at equal parameters (loss −0.041; standard MoE seed spread 0.007) |
| ≥ 2× wider experts | slightly better than a standard MoE with 2× wider experts (−0.037), with half its expert parameters |
| int3 > full precision | int3 version 0.024 below the full-precision standard MoE (2.4% lower perplexity) |
| 22% of the expert bytes | int3 vs bf16 expert weights read per token |
| 2nd pass hits L2 | RTX 5080: pass-2 weight reads drop to the L2-hot floor; optimised pass 2 costs 0.47× a fresh layer (Qwen3-30B-A3B layer shape, int4) |
| smaller model (6 layers, d512) | same picture, larger gaps: −6.0% / int3 −5.4% |
The catch: repeat attention, not just the experts. Re-running the same experts alone gives almost nothing (−0.010 at 8 layers, 3 seeds). The gain comes from a second round of token mixing, so the second pass must include attention.
The architecture, in four pieces.
Why it matters. At batch size 1, a GPU spends most of each token waiting for weights to arrive from VRAM. MoEs have huge total parameter counts, so they hit this wall hardest. This architecture spends the idle compute on a second pass over weights that are already on chip, and turns that into quality you can then trade for fewer bytes.
Loss gap to a standard MoE of the same size, through training, at two model sizes. Whole-layer repeat (purple) pulls below the standard MoE early and stays far outside its seed range to the end: −0.062 at 6 layers, −0.040 at 8 layers, where it also edges past a standard MoE with 2× wider experts (grey dashed). The int3 versions (lighter purple) stay below the full-precision baseline. Repeating only the experts (orange) gains about a quarter as much.
What is new. We treat the GPU's L2 cache as a design target. Quantize the repeated layers until they fit in L2, run them again while they are still there, and use that cheap second pass to win back the quality the low bit-width cost. We measure this loop end to end: a controlled sweep of bit-width × loop count at equal weight bytes (two seeds, int3–int8, 1–6 loops), the cache cliff on GPUs with different L2 sizes, and the same idea applied to MoE experts.
L2Loop has evidence for trading compute for precision and memory, but not yet for trading compute for parameter diversity.
Final val loss of whole-layer repeat minus a standard MoE of the same size, same data, steps and learning rate.
| model | training tokens | whole-layer repeat − standard MoE | int3 version − full-precision standard MoE |
|---|---|---|---|
| 6 layers, d512 (184M) | 101M | −0.063 / −0.077 (2 seeds) | −0.056 / −0.051 (2 seeds) |
| 6 layers, d512 | 303M (3×) | −0.066 | — |
| 8 layers, d768 (670M) | 524M | −0.041 | −0.024 |
| 12 layers, d1024 (1.5B) | 524M | −0.009 (A100) / −0.015 (GB10) | not run |
| 12 layers, d512 | 303M | −0.027 all layers (zero-init −0.025); −0.038 / −0.038 first half looped (2 seeds) | −0.020 (first half looped, int3) |
| 24 layers, d512 | 303M | −0.018 first half looped; −0.014 first quarter | not run |
| which layers run twice (12 layers, d512, 303M tokens) | extra layer passes | val loss | − standard MoE | perplexity |
|---|---|---|---|---|
| all 12 | 12 | 3.5810 | −0.027 | −2.7% |
| first 6 | 6 | 3.5699 | −0.038 | −3.8% |
| every other layer | 6 | 3.5776 | −0.031 | −3.0% |
| middle 6 | 6 | 3.5932 | −0.015 | −1.5% |
| last 6 | 6 | 3.6135 | +0.005 | +0.5% |
The second pass pays off in the early layers and hurts in the late ones. Repeating the first half beats repeating everything with half the extra compute, so in deep models the extra pass belongs in the early layers. Repeating the first half is confirmed with a second seed (−0.038 both times; the standard MoE's two seeds differ by 0.0025). Halving the learning rate, ramping the second pass in over training, and a learned per-layer gate do not beat it.
What the trained models rely on. Switching off one layer's second pass at a time in the trained models shows the same picture in all six of them: they put a strong second pass on one or two early layers (layer 2, 3 or 4 of 12; layers 4–5 of 24). Switching off that one layer costs 0.5–2.2 in loss, close to switching off every second pass; switching off any late layer costs 0.03–0.15. Even models allowed to repeat every layer end up relying on an early one. (This measures what a trained model depends on, not what a layer would add under different training.)
int3 at 12 layers. With the first half repeated and int3 experts, the model still beats the full-precision standard MoE (−0.020).
Our fused kernel, Qwen3-30B-A3B layer shape (int4), batch 1, all layers in one launch with distinct weights per layer (every layer starts cold), µs per token for the MoE stack:
| configuration | ctx 512 | ctx 4096 | weight bytes / token | extra VRAM |
|---|---|---|---|---|
| llama.cpp, standard 48 layers | 3559 | 4662 | — | — |
| ours, standard 48 layers | 3775 | 4383 | 1.58 GB | — |
| ours, 48 layers, first 24 repeated | 4918 | 6095 | 1.69 GB | ≈0.1 GB + a second KV cache for 24 layers |
| ours, standard 72 layers (+50% layers) | 5435 | 6329 | 2.37 GB | ≈9 GB |
| measured | |
|---|---|
| Cost of re-running a quantized Qwen3-1.7B layer that fits in L2 (RTX 5080, 64 MB L2) | 0.49× a fresh layer |
| Same layer when it does not fit (RTX 5060 Ti, 32 MB L2) | 1.00×: the benefit appears only when the unit fits |
| Loss recovered by looping, at equal parameters (from-scratch models, two seeds) | −0.05 to −0.08, which is 2–3× the int4 quantization penalty (≈0.03) |
| int3 + looping vs a larger non-looped int4 model | same loss with 0.55× the weight bytes |
The loop gain and the GPTQ gain add up: quantize harder, loop in cache, keep the quality.
The kernel.
Quantization makes it work. The repeated unit has to fit in L2. A Qwen3-1.7B layer is ≈100 MB in bf16 (fits no consumer L2), ≈50 MB in int8 (fits a 64 MB L2) and ≈25 MB in int4 (fits a 32 MB L2). Lower bits are what turn the second pass from a full-price layer into a cheap one.
The model change. The second pass enters through a gate initialised to zero, h₂ = h₁ + a·(f(h₁) − h₁). A model starts out identical to the original and learns how much of the second pass to use. A per-token router can restrict the repeat to the tokens that need it, so the average cost per token drops in proportion.
Why MoE. A MoE layer is large, but a token only touches a few experts, and each expert is small. With DeepSeek-V4.1-class shapes (hidden 5120, expert FFN 2304, 6 routed experts per token):
| unit | parameters | 4-bit size | fits 64 MB L2 (RTX 5080)? | fits 32 MB L2 (RTX 5060 Ti)? |
|---|---|---|---|---|
| one expert (gate, up, down) | 35.4M | ≈18 MB | yes | yes |
| the 6 experts a token uses | 212M | ≈106 MB | no | no |
| a dense 27B layer, for comparison | ≈330M | ≈165 MB | no | no |
A dense layer of a large model never fits, but every expert does. Three ways to use that:
What is measured and what is next. Sections 2–5 measure the mechanism on from-scratch models and four GPUs. Section 6 measures a pretrained 1.7B model, where a whole layer fits in L2 once quantized. Section 7 measures the cost of repeating MoE experts on four GPUs and the latency of whole routed MoE layers; for MoE quality, repeating the whole layer (attention and the same in-cache experts) beats a standard MoE by 0.063, and with int3 experts it still beats the full-precision standard MoE by 0.056 at 0.22× the expert bytes (Section 7).
Models. Both arms use d = 1024, 16 heads, FFN 4d, the GPT-2 vocabulary, context 1024, tied embeddings, and a product-key memory layer (65,536 slots, top-32, 4 heads). Both are trained on 1B tokens of FineWeb-Edu, two seeds per configuration.
| structure | distinct blocks | block params | |
|---|---|---|---|
| A (non-looped) | prelude + m distinct blocks + coda; m = 4/6/8/10, i.e. 6/8/10/12 blocks | 6–12 | 75.5M–151M |
| B (looped) | prelude + 4-block core looped L times + coda | 6 | 75.5M (+2.1M injection) |
Evaluation. Validation loss on 80 × 16 × 1024 held-out tokens; all numbers in a table come from the same script. Quantization is fake-quant: RTN int8 per row; RTN int4/int3 with group 32; GPTQ, group 32, 128 × 1024 calibration tokens; for B, GPTQ is calibrated at 2 loops.
Kernel. One persistent HIP kernel decodes a whole token: RMSNorm, int8/int4 GEMV with A8 activations, attention over the KV cache, PKM top-k and the LM-head argmax. Blocks synchronise through a grid barrier with a 50 ms watchdog. It is checked token by token against a PyTorch fake-quant reference (16/16 tokens match, max logit error < 0.05), and reduces A8 decode from 1.41 ms (PyTorch + HIP graph) to 0.45 ms per token. A CUDA port runs on the NVIDIA cards; it spills registers (64 per thread at 1024 threads per block), so its absolute times are pessimistic.
Loop count (B, validation loss, full precision):
| loops | 1 | 2 | 3 | 4 |
|---|---|---|---|---|
| seed 0 | 3.367 | 3.271 | 3.250 | 3.245 |
| seed 1 | 3.356 | 3.264 | 3.245 | 3.240 |
The second pass gives 0.09–0.10, the third 0.02, the fourth 0.005.
Equal parameters (same 6 distinct blocks):
| seed 0 | seed 1 | |
|---|---|---|
| A, 6 blocks, 1 pass | 3.347 | 3.311 |
| B, 6 blocks, core looped 2× | 3.271 | 3.264 |
| difference | −0.076 | −0.047 |
With quantization (int4). "A8" = A with 8 blocks.
| A8 RTN | A8 GPTQ | B L2 RTN | B L2 GPTQ | |
|---|---|---|---|---|
| seed 0 | 3.328 | 3.308 | 3.305 | 3.280 |
| seed 1 | 3.321 | 3.304 | 3.297 | 3.274 |
µs per token at ctx 64, int8, median of 3 runs; A10/A12 latencies measured with same-shape weights.
| config | latency | val loss (seed 1) | distinct block params |
|---|---|---|---|
| A, 8 blocks | 446 | 3.292 | 101M |
| B, loops 2 | 502 | 3.264 | 75.5M |
| A, 10 blocks | 523 | 3.262 | 126M |
| B, loops 3 | 624 | 3.245 | 75.5M |
| A, 12 blocks | 602 | 3.239 | 151M |
| one B loop pass, int8, ctx 64 | µs |
|---|---|
| real | 134 |
| weights forced L2-resident | 107 |
| L2-resident + no grid barriers | 96 |
| all dimensions shrunk to 1/8 (64× fewer weight bytes) | 99 |
Cost of a reused block relative to a fresh block:
| reuse pattern | 7900 GRE | RTX 5080 | RTX 5060 Ti | RTX PRO 6000 |
|---|---|---|---|---|
| whole-core loop | 0.60–0.75 | 1.03–1.20 | 1.16–1.22 | ≈1.0 |
| immediate repeat (b1 b1 b2 b2 …) | 0.73–0.76 | 0.81–0.85 | 0.79 | – |
For a larger model, per-token time is dominated by streaming weights, so the fixed per-stage latency matters less, and a layer re-run while its weights are still in L2 should be cheaper.
Layer microbenchmark (Qwen3-1.7B layer shapes, batch 1, ctx 512). Full decoder step with layers 10–17 repeated immediately; CUDA/HIP graphs.
| GPU (L2) | bits | fresh layer µs | repeated layer µs | ratio | GEMV-only ratio |
|---|---|---|---|---|---|
| RTX 5080 (64 MB) | int8 | 84.5 | 41.2 | 0.49 | 0.33 |
| RTX 5080 | int4 | 56.0 | 36.3 | 0.65 | 0.44 |
| RX 7900 GRE (6 MB L2 + 64 MB IC) | int8 | 142.5 | 78.2 | 0.55 | 0.40 |
| RTX 5060 Ti (32 MB) | int8 | 152.9 | 152.9 | 1.00 | – |
| RTX 5060 Ti | int4 | 94.1 | 56.9 | 0.60 | – |
The 5060 Ti rows isolate the cache effect: an int8 layer (50 MB) does not fit its 32 MB L2 and the repeat costs exactly as much as a fresh layer; the int4 layer (25 MB) fits and drops to 0.60.
End-to-end decode (real Qwen3-1.7B weights, 256 new tokens, median of 5). "L" repeats layers 10–17 for every token; "LR" repeats on 30% of decode steps.
| GPU | stack | repeated ÷ fresh layer | L overhead | LR overhead |
|---|---|---|---|---|
| RTX 5080 | exllamav3 EXL3 8 bpw | 0.81 | +20% | +6% |
| RTX 5080 | exllamav3 EXL3 4 bpw | 0.89 | +22% | +7% |
| RTX 5060 Ti | exllamav3 EXL3 4/8 bpw | 0.91–0.99 | +22–24% | +7% |
| all three | PyTorch bf16, whole-token graph | ≈1.0 | +25% | +7–8% |
| RX 7900 GRE | our int8 HIP kernel (random weights, real attention) | 0.57–0.64 | +13–16% | +1.4–4.5% |
Quality on a pretrained model. We continue pretraining Qwen3-1.7B-Base on 162M tokens of FineWeb-Edu (1,239 steps × 64 × 2048, lr 2e-5 cosine, full fine-tune bf16) in arms that see identical data in identical order:
Validation loss on the same 1M held-out tokens:
| step (tokens) | C | L | LH | C − L | C − LH |
|---|---|---|---|---|---|
| 0 | 2.42051 | 2.42051 | 2.42051 | 0 | 0 |
| 154 (20M) | 2.41721 | 2.41721 | 2.41700 | 0.00000 | 0.00021 |
| 308 (40M) | 2.41696 | 2.41663 | 2.41644 | 0.00033 | 0.00052 |
| 462 (61M) | 2.41532 | 2.41488 | 2.41456 | 0.00044 | 0.00076 |
| 616 (81M) | 2.41299 | 2.41243 | ⟨TBM⟩ | 0.00056 | ⟨TBM⟩ |
| 770 (101M) | 2.41034 | 2.40977 | ⟨TBM⟩ | 0.00057 | ⟨TBM⟩ |
| 924 (121M) | 2.40805 | 2.40731 | ⟨TBM⟩ | 0.00074 | ⟨TBM⟩ |
| 1078 (141M) | 2.40644 | 2.40581 | ⟨TBM⟩ | 0.00063 | ⟨TBM⟩ |
| 1239 (162M) | ⟨TBM⟩ | 2.40510 | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ |
| arm | val loss | int4 GPTQ | int3 GPTQ | HellaSwag | ARC-e | PIQA |
|---|---|---|---|---|---|---|
| base | 2.4205 | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ |
| C | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ |
| L | 2.4051 | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ |
| LH | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ | ⟨TBM⟩ |
Expert repeat cost (measured, batch 1). One routed expert (gate, up, down GEMV) repeated immediately after its first pass, relative to computing a fresh expert. Three real expert shapes; three runs per GPU, ranges over runs. GB10 ranges cover two DGX Spark units measured on different days (they agree within 0.02).
| expert shape (hidden × expert FFN), bits | expert size | GB10 (24 MB L2) | RTX 5080 (64 MB) | RTX 5060 Ti (32 MB) | RX 7900 GRE (6 MB L2 + 64 MB IC) |
|---|---|---|---|---|---|
| OLMoE-1B-7B (2048 × 1024), int8 | 6.3 MB | 0.42–0.43 | 0.78–0.84 | 0.64–0.65 | 0.68–0.79 |
| OLMoE-1B-7B, int4 | 3.2 MB | 0.55–0.58 | 1.0–1.2 | 0.88–0.90 | 0.91–1.03 |
| Qwen3-30B-A3B (2048 × 768), int8 | 4.7 MB | 0.47–0.49 | 0.92–0.98 | 0.73 | 0.78–0.87 |
| DeepSeek-V4.1 class (5120 × 2304), int4 | 18 MB | 0.28–0.34 | 0.59–0.70 | 0.50 | 0.62 |
| DeepSeek-V4.1 class, int8, unsplit | 35 MB | 0.95–0.98 (does not fit) | – | 0.78–0.81 (does not fit) | 0.46–0.49 |
| DeepSeek-V4.1 class, int8, split in 2 slices | 2 × 18 MB | 0.30–0.32 | – | 0.53 | – |
Bits × loops at equal bytes (from-scratch models, two seeds, measured). Every quantization of the standard family (6–12 distinct blocks, one pass) and of the looped family (6 distinct blocks; the 4-block core run 2–6 times), all evaluated with the same script.
Validation loss against block-weight bytes read per token (parameters × bits/8). Left seed 0, right seed 1. Gray: standard models, one pass; purple: the same 6 blocks with the core looped ×2/×3/×4/×6. Extra loops add no bytes, so each looped bit width is a vertical column. Dotted lines mark L2 sizes; the strip under each panel shows the looped core alone (4 blocks × 12.58M params × bits/8: int3 18.9, int4 25.2, int5 31.5, int6 37.7, int8 50.3 MB). Extra loops cost compute: on the RX 7900 GRE a loop pass costs about 0.6–0.75× a fresh block. int2 collapses (≈4.4 at every loop count) and is not drawn.
| looped bits (block MB) | seed | ×1 | ×2 | ×4 | best standard at ≤ same bytes | ×2 − standard | ×4 − standard |
|---|---|---|---|---|---|---|---|
| int3 (28.3) | 0 | 3.4267 | 3.3240 | 3.2992 | 6 blocks int3: 3.4041 | −0.080 | −0.105 |
| 1 | 3.4153 | 3.3176 | 3.2940 | 6 blocks int3: 3.3826 | −0.065 | −0.089 | |
| int4 (37.7) | 0 | 3.3783 | 3.2805 | 3.2553 | 8 blocks int3: 3.3567 | −0.076 | −0.101 |
| 1 | 3.3668 | 3.2743 | 3.2506 | 6 blocks int4: 3.3236 | −0.049 | −0.073 | |
| int5 (47.2) | 0 | 3.3699 | 3.2730 | 3.2476 | 8 blocks int3: 3.3567 | −0.084 | −0.109 |
| 1 | 3.3588 | 3.2667 | 3.2425 | 10 blocks int3: 3.3147 | −0.048 | −0.072 | |
| int6 (56.6) | 0 | 3.3681 | 3.2713 | 3.2458 | 8 blocks int4: 3.3084 | −0.037 | −0.063 |
| 1 | 3.3571 | 3.2650 | 3.2406 | 12 blocks int3: 3.2829 | −0.018 | −0.042 | |
| int8 (75.5) | 0 | 3.3675 | 3.2708 | 3.2453 | 8 blocks int6: 3.2986 | −0.028 | −0.053 |
| 1 | 3.3565 | 3.2645 | 3.2401 | 12 blocks int4: 3.2465 | +0.018 | −0.006 |
int3–int6 are GPTQ (group 32), int8 is RTN per row. Seed 0 has no 10- or 12-block standard models, so its standard frontier stops at 8 blocks; seed 1 has them, and there the gap closes at the largest budget: at 75.5 MB, 12 blocks int4 beats the looped int8 model at 2 loops and is within 0.006 of it at 4 loops. Looping gains most where bytes are scarce, which is also where the looped core fits the caches (int4 core 25 MB < 32 MB L2; int3 core 19 MB < 24 MB L2).
Routed MoE layer latency (measured, batch 1). The from-scratch MoE shape: hidden 768, 32 experts per layer, SwiGLU expert FFN 1024 (MW: 2048), 8 MoE layers per token, the routed experts of each layer chosen at random per token (the whole expert set, 151 MB int8 / 75 MB int4, is larger than every L2 here). Each arm: M0 top-4 once; MR top-4, each expert computed twice back to back (e1 e1 e2 e2 …); MH top-2, each twice; MK top-8 once; MW top-4 with 2×-wide experts once. Only the expert part of the layer is timed (RMSNorm + quantize, gate|up GEMV, SwiGLU, down GEMV; router and attention excluded; a re-norm before every second pass). 16 tokens per graph, median of 3 runs × 11 rounds.
µs per MoE layer (µs per token for 8 layers in parentheses):
| GPU | bits | M0 (4 × 1) | MR (4 × 2) | MH (2 × 2) | MK (8 × 1) | MW (4 wide × 1) | MH / M0 | MR / MK |
|---|---|---|---|---|---|---|---|---|
| GB10 (24 MB L2) | int8 | 64.6 (517) | 102.4 (820) | 52.5 (420) | 126.8 (1014) | 104.1 (833) | 0.81 | 0.81 |
| GB10 | int4 | 43.4 (347) | 76.5 (612) | 39.1 (312) | 84.4 (675) | 65.8 (527) | 0.90 | 0.91 |
| RTX 5060 Ti (32 MB) | int8 | 44.7 (358) | 77.1 (617) | 39.5 (316) | 87.2 (697) | 69.5 (556) | 0.88 | 0.88 |
| RTX 5060 Ti | int4 | 33.8 (270) | 64.2 (514) | 32.9 (263) | 66.1 (529) | 49.0 (392) | 0.97 | 0.97 |
| RTX 5080 (64 MB) | int8 | 31.7 (253) | 60.1 (481) | 30.7 (246) | 61.8 (494) | 46.5 (372) | 0.97 | 0.97 |
| RTX 5080 | int4 | 25.9 (207) | 53.6 (429) | 27.6 (221) | 50.7 (406) | 34.7 (278) | 1.07 | 1.06 |
| RX 7900 GRE (6 MB L2) | int8 | 58.6 (469) | 109.1 (873) | 55.7 (446) | 112.3 (898) | 81.0 (648) | 0.95 | 0.97 |
| RX 7900 GRE | int4 | 46.9 (375) | 96.8 (774) | 49.5 (396) | 89.8 (718) | 60.5 (484) | 1.06 | 1.08 |
Pretrained MoE, gate-only training (OLMoE-1B-7B, measured). All weights frozen; one scalar per layer (16 in total) trained on 20M tokens of FineWeb-Edu; validation on the same 1M held-out tokens. rep_all repeats all 8 selected experts once more on the MoE output (same routing) and learns how much to add; rep_top1 repeats only the top-1 expert; control has no repeat and learns a per-layer scale of the MoE output (same number of trainable scalars).
| arm | val loss | vs no training |
|---|---|---|
| no training | 2.22980 | – |
| control (scale the MoE output) | 2.22981 | +0.00001 |
| rep_all (repeat all 8 experts) | 2.22970 | −0.00010 |
| rep_top1 (repeat the top-1 expert) | 2.22989 | +0.00009 |
One seed. All four arms lie within 0.0002 of each other, below what one seed and a 1M-token validation set can resolve; the learned scalars stay small (|a| ≤ 0.06). With only the gates trained, repeating experts in a pretrained MoE gives no measurable gain over the control at equal trainable parameters.
The hypothesis. At batch size 1, per-token cost is set by the bytes streamed from memory, not by the arithmetic. Once an expert has been streamed in, more computation on it costs about half price for as long as it stays in cache. Models are designed as if each weight should be used once per token; at batch 1 that allocation may not be optimal. We think the right axis is quality per byte read: a MoE that loads k experts and reuses each of them in cache may beat one that loads more distinct experts for the same memory traffic.
Our MoE proposal: expert repeat with input re-injection. We propose running each selected expert a second time while its weights are still in L2, with the layer input re-injected before the second pass:
h2 = h1 + E(RMSNorm(inj([h1, x]))), inj initialized to 0.5·[I, I].
We claim this as the MoE form of L2Loop: a way to gain quality from cached compute, and through that to recover quantization loss at fewer bytes. Re-injection is the ingredient that matters.
From-scratch result (Colab, 524M tokens, two seeds each). Small MoE: d 768, 8 layers, 32 experts, top-4; same data order, steps and learning rate for every arm. Final validation loss, 4096 windows:
| arm | seed 0 | seed 1 | mean | seed spread |
|---|---|---|---|---|
| standard MoE | 3.3948 | 3.3986 | 3.3967 | 0.004 |
| expert repeat, no re-injection | 3.4034 | 3.3875 | 3.3955 | 0.016 |
| expert repeat + re-injection | 3.3858 | 3.3883 | 3.3871 | 0.003 |
Both re-injection seeds are below both standard-MoE seeds, by 0.010 on average; the seeds agree to within 0.003 (re-injection) and 0.004 (standard). Plain repeat ties the standard MoE on average and varies five times more between seeds. The mechanism shows in the cosine between the two expert updates: 0.82 without re-injection, 0.40–0.44 with it. The second pass stops copying the first.
Fewer bytes, same or better quality. The repeat buys precision. Expert repeat + re-injection quantized to GPTQ int4 beats the standard MoE at int8, while reading 0.56× the expert bytes per token:
| re-injection arm vs standard MoE | expert bytes | seed 0 | seed 1 |
|---|---|---|---|
| GPTQ int4 vs int8 | 0.56× | −0.008 | −0.004 |
| GPTQ int3 vs GPTQ int4 | 0.78× | −0.001 | +0.002 |
Expert weights quantized; negative = the repeat arm is better at fewer bytes. At every precision we tried (int8, GPTQ int4, GPTQ int3) the re-injection arm stays 0.006–0.010 below the standard MoE at the same bytes. Earlier small runs on two DGX Sparks (118M tokens) tied: −0.0002 and +0.037.
Repeat the attention, not just the experts. The decisive ingredient for MoE is a second round of token mixing. This is a new MoE architecture: each layer runs twice in place, the second pass locked to the first pass's experts so it reads weights still in L2, with experts trained at int3 so the whole layer fits there. Repeating each layer in place, attention and the same pass-1 experts, with the residual stream preserved (MAL2f: h2 = h1 + Attn(z) + MoE(z + a2), z = inj([ln(h1), x]), same experts as pass 1 so they are still in L2), gives the largest gain we have measured on an MoE:
| arm (from-scratch MoE, d512, 6 layers, 32 experts top-4, 101M tokens) | val | vs standard MoE (mean of 3 seeds) |
|---|---|---|
| standard MoE | 4.3899 | — |
| whole-layer repeat, same experts (MAL2f) | 4.3274 | −0.063 |
| core loop of 4 layers, 2 passes (ML2) | 4.3391 | −0.051 |
| attention-only repeat, experts once (MAR) | 4.3587 | −0.031 |
| expert-only repeat + re-injection (MRi) | 4.3997 | +0.010 |
Expert-only repeat does not help at this size; every arm that repeats attention does. The experts are computed once more from the same weights in L2, and the attention gets a second look at the updated context.
Compute buys precision on MoE. Trained with int3 experts (quantization-aware), whole-layer repeat keeps almost all of its gain, while the standard MoE pays the usual int3 tax:
| arm, int3 experts (QAT) | val | vs standard MoE in bf16 |
|---|---|---|
| standard MoE, int3 | 4.4104 / 4.4138 | +0.021 / +0.024 |
| whole-layer repeat, int3 (MAL2fq) | 4.3344 | −0.056 |
An int3 MoE with in-cache whole-layer repeat beats a full-precision standard MoE by 0.056, while reading 0.22× of its expert bytes. This is the L2Loop trade on MoE: quantize the experts until they fit, run the layer again while they are in cache, and come out ahead of the unquantized model.
Screen setting: one seed per repeat arm, three seeds for the standard MoE (spread 0.034); identical data order, steps and learning-rate schedule for every arm.
What is established and what is not.
| claim | status |
|---|---|
| A looped core trained from scratch gains quality at equal parameters, and bits trade for loops at equal bytes | measured (two seeds, Sections 3–4) |
| At equal block bytes, a looped model has lower loss than a standard one | measured, two seeds; clear at 28–57 MB, gap closes at 75 MB on seed 1 (Section 7) |
| An expert repeated while it is in L2 costs 0.3–0.6× a fresh one; slicing keeps large experts at that price | measured (Section 7, four GPUs) |
| A routed MoE layer that repeats fewer experts is faster than one that reads more experts | measured, only on slow memory: 0.81× on GB10 int8, 0.88× on 5060 Ti int8, ≈1.0× elsewhere (small 768 × 1024 experts) |
| A pretrained model can be looped as is | no: opening the second pass without training raises Qwen3-1.7B loss from 2.4205 to 2.5815 |
| A pretrained model with short continued training | small so far (−0.0007 after 162M tokens, Section 6) |
| Expert repeat improves a pretrained MoE with gate-only training | no measurable gain: all arms within 0.0002 (rep_all −0.00011, rep_top1 +0.00009, control +0.00001; OLMoE-1B-7B, 20M tokens, one seed) |
| Expert repeat and MoE quality when trained from scratch | with re-injection: below standard MoE on both seeds (−0.010 mean; every repeat seed below every standard seed, two seeds each), and at int4 it beats standard int8 with 0.56× the expert bytes; without re-injection: ties (mean +0.001). Larger scales running |
| Whole-layer repeat (attention + same experts) on MoE | beats the standard MoE by 0.063; with int3 experts it beats the full-precision standard MoE by 0.056 at 0.22× expert bytes (from-scratch, d512 L6) |
Retrofitting the loop onto a finished model gives little so far.
Running a layer again while its weights sit in cache is cheap; that observation alone is not ours. What this post adds is turning it into a design rule and measuring the whole loop:
Code, kernels (HIP + CUDA), training scripts, Colab notebooks and timing summaries: this repository (see README.md). Checkpoints: ⟨TBM⟩.
Hardware: RX 7900 GRE (ROCm 6.4, Windows), RTX 5080 and RTX 5060 Ti (CUDA 12.8; CuPy JIT and exllamav3 1.6.4), RTX PRO 6000 (Colab G4), NVIDIA GB10 / DGX Spark (CUDA, nvcc; MoE benchmarks).
Re-run L2-resident layers to undo low-bit quantization loss