thefinalboss commited on
Commit
5acd391
·
verified ·
1 Parent(s): 5238a94

Per-token MoE routing. Causal leak closed.

Browse files
Files changed (2) hide show
  1. docs/CLAUDE.md +7 -15
  2. fractus/continuous_engine.py +5 -3
docs/CLAUDE.md CHANGED
@@ -123,24 +123,16 @@ The model does not read r. It reads MoE gates. On this fresh smoke (8 experts, t
123
 
124
  ## Chunk routing
125
 
126
- `_tick_chunk_core_pure` routes the whole chunk with the last position's θ:
127
 
128
- ```python
129
- phases_last = theta[:, -1:, :]
130
- phases_in = phases_last.expand(-1, C, -1)
131
- ```
132
 
133
- Generation (`tick_single`) routes with `theta[:, 0:1]`, the current token. Same lines in the parent. Inherited, not introduced here.
 
 
 
134
 
135
- Structural consequences: one routing decision per chunk per block, and a possible causal leak, because position t is gated by a θ computed from h at C−1.
136
-
137
- Measured on a fresh smoke, both at default T and at T=2:
138
-
139
- - Changing only the last id moves the last position's logits (~0.02–0.03). Positions 0..C−2 stay at 0, or at ~1e-4, float noise.
140
- - `theta[:, 0]` and `theta[:, -1]` differ by ~1e-7. `_encode_from_hidden` uses the mean of a LayerNorm'd h, which is ~0, so phases barely depend on position until the clock has something to integrate.
141
- - Alive experts stayed 2/8. The "2 alive" count was this expand, not a T effect.
142
-
143
- The leak is in the code. It is inert while θ is flat. It becomes real when phases start depending on position. Do not switch to `phases_in = theta` (per-token, causal, aligned with generation) until a small train shows the leak is nonzero and the gate does not get worse. That switch also changes MoE cost. Measure ids/s if it is made.
144
 
145
  ## If you are unsure
146
 
 
123
 
124
  ## Chunk routing
125
 
126
+ Fixed. `_tick_chunk_core_pure` now passes `phases_in = theta`. Each position is gated by its own phase. Generation already did this (`theta[:, 0:1]`). The old expand of `theta[:, -1]` is gone.
127
 
128
+ Measured on a smoke body (64-d, 8 experts, top-2, Kuramoto fix on):
 
 
 
129
 
130
+ - Last-theta routing: after 400 steps the leak is real. Changing only the last id moved position-0 logits by **0.028**. Theta spread between pos 0 and pos last was 0.024.
131
+ - Per-token routing: the same probe stays at **0.000** at every step from 0 to 400. Last-position logits still move (the token changed). Earlier positions do not.
132
+ - Throughput on the same CPU smoke, 40 steps after warmup: last-theta **2,257 ids/s**, per-token **2,088 ids/s**. About 7% slower on this tiny body. Not a 1B number.
133
+ - Alive experts stayed 2/8 either way on this short run. The switch removes the leak. It does not by itself wake more experts.
134
 
135
+ Do not put the expand back. It is a causal leak once phases depend on position, and it disagrees with generation.
 
 
 
 
 
 
 
 
136
 
137
  ## If you are unsure
138
 
fractus/continuous_engine.py CHANGED
@@ -200,10 +200,12 @@ class CTEBlock(nn.Module):
200
  theta = self.kuramoto._encode_from_hidden(h_kur)
201
  theta = self.kuramoto._rk4_integrate(theta)
202
 
203
- # MoE.
 
 
 
204
  h_moe = self.norm_moe(h)
205
- phases_last = theta[:, -1:, :]
206
- phases_in = phases_last.expand(-1, C, -1)
207
  moe_out, lb_loss = self.moe(h_moe, phases_in)
208
  h = h + moe_out
209
 
 
200
  theta = self.kuramoto._encode_from_hidden(h_kur)
201
  theta = self.kuramoto._rk4_integrate(theta)
202
 
203
+ # MoE. Per-token phases: causal, aligned with tick_single.
204
+ # The old expand of theta[:, -1] routed the whole chunk by the last
205
+ # position and leaked future tokens into earlier gates once phases
206
+ # depended on position (measured after ~400 smoke steps).
207
  h_moe = self.norm_moe(h)
208
+ phases_in = theta
 
209
  moe_out, lb_loss = self.moe(h_moe, phases_in)
210
  h = h + moe_out
211