Download DESIGN.md from elyanlabs/elya-nes: direct link, hf CLI and curl.
- Browser
- Download file 27.4 kB
-
https://huggingface.co/elyanlabs/elya-nes/resolve/main/DESIGN.md
- Command line
-
hf download hf://elyanlabs/elya-nes/DESIGN.md
-
curl -L -o DESIGN.md https://huggingface.co/elyanlabs/elya-nes/resolve/main/DESIGN.md
NES transformer port - ROM layout and cost model
Written before the kernel, as the brief requires. Every number here is
either a measured primitive from FINDINGS.md or arithmetic on top of one.
Where it is a prediction, it is labelled as such and is checked against a
measurement later in FINDINGS.md.
0. The constraint that shapes everything
The 6502 has one accumulator and two index registers, and no ORA/ADC to
X or Y. A ternary dot product with the accumulator resident in A therefore
has exactly one shape:
ldy <stream>,x ; 4 fetch the next input index from the weight stream
adc <activations>,y ; 4 accumulate that activation
Both operands must be reached through an index register, so the weight
stream must be addressed by an index register too, which means its base has
to be an assembled absolute address. Anything else - lda (ptr),y to walk
the stream - puts the stream byte in A and evicts the accumulator, which is
what turns an 8-cycle inner op into a 19-cycle one.
This is the same structural fact that killed the 4-bit-weight LUT design in the prior run, and it is the single reason for the "page chain" layout below.
1. Carry: why activations are stored biased
A chain of adc without clc is 8 cycles per element (measured) but is
arithmetically wrong - each adc adds the previous unsigned carry. Adding
clc costs 2 and puts us back at 10.
Instead the activations are stored biased: actb[i] = act[i] + 7, so
every stored byte is in 0..14 and every partial sum is non-negative. With a
block of 16 the largest possible block sum is 16*14 = 224 < 256, so the
carry flag provably never sets and no clc is needed anywhere inside a
block. The bias is removed once per list:
sum = (biased_sum - 7 * n)
This is the same headroom argument as the prior run's 16*7 = 112 bound, just
shifted into unsigned space, and it is the reason block size is 16 and not
32: 32 would give 448 > 255 and the carry-free guarantee would be lost.
Block 32 is not "probably fine here" - it is provably wrong for this
formulation.
2. Model shape, chosen from the RAM budget
vocab V |
64 |
reference (host/ref.py V=64) and the committed weight count all
say 64. 98,304 layer weights + 64*64 embedding = 102,400, which
is the figure in FINDINGS; V=128 would give 106,496. -->
| d_model D | 64 |
| layers L | 3 |
| heads H / d_head | 2 / 32 |
| d_ff F | 128 |
| context T | 20 or 85 - a build parameter (NES_T / -DNCTX) |
T is the only shape knob that varies, and the two values are the two ends of
what the cartridge can hold: 20 fills one PRG-RAM bank, 85 fills all four.
Everything else below is independent of it.
Ternary weight count:
per layer Wq,Wk,Wv,Wo 4 x 64x64 = 16,384
W1 (F x D) 128x64 = 8,192
W2 (D x F) 64x128 = 8,192
-------
32,768
x 3 layers = 98,304
output head (V x D) 64x64 = 4,096
-------
TOTAL ternary weights 102,400
(This block previously read 128x64 = 8,192 and TOTAL 106,496, left over
from the same V=128 doc error the vocab row above was corrected for. The
committed weight count, rom/nn.s NVOCAB = 64 and host/ref.py V = 64
all say 102,400.)
(The prior run's extrapolation was quoted for 114,688 active weights; this shape is within 7% of that, so the two are directly comparable.)
The $6000 window arithmetic
This describes the LEGACY attention path only. The attention kernels replaced it: K and V no longer share one interleaved cache, they want opposite layouts and live in separate PRG-RAM banks. See "The attention kernels and their two caches" below. The layout described here is still assembled, and is selected automatically for any
Tthe kernels cannot address - which is what keeps theT = 85experiment buildable.
KV cache is L x T x 2 x D bytes, one byte per 4-bit activation:
T = 20: 3 x 20 x 2 x 64 = 7,680 bytes 93.75% of ONE 8 KB PRG-RAM bank
T = 85: 3 x 85 x 2 x 64 = 32,640 bytes 99.61% of ALL FOUR
Measured PRG-RAM is 32 KB in 4 banks at $5113 = 4,5,6,7 (declaring more
in the iNES header still yields 4 - reconfirmed this run), so
T_max = 32768 / (L x 2 x D) = 32768 / 384 = 85 positions
is a hard ceiling on the legacy path, not a preference. T = 20 was chosen
originally so that the cache fitted one bank and the attention loop never had
to switch mid-flight; T = 85 gives that up and pays for a bank register
write per KV row address.
The attention kernels have a much lower ceiling, and it is not capacity.
They reach their caches with assembled absolute constants, which is the whole
reason the accumulator can stay in A. That costs two bounds:
QK: ldx KTBASE + d*64, y y = l*T + t
y is one byte and the d stride is 64, so L*T <= 64 -> T <= 21
AV: ldx VBASE + t*256, y y = l*64 + d
the base is an address inside the $6000 window, so
VBASE + (T-1)*256 + 255 <= $7FFF -> T <= 32
T <= 21 binds. Raising it means giving the key cache a stride of L*T
instead of 64, which at T = 85 is 64 * 255 = 16,320 bytes - two PRG-RAM
banks - and an assembled absolute address cannot cross a bank. And even if it
could: at T = 85 the two caches together need all 32,768 bytes of PRG-RAM,
so there is no bank left for the writable kernels. Both bounds are .assert-ed
in rom/nn.s; a build above them selects the legacy path rather than emitting
something that looks right.
KV row alignment, and why banking is safe. Rows are D = 64 bytes, laid
out at row index
row = (l*2 + kv) * T + t layout [layer][k|v][t]
addr = $6000 + (row & 127) * 64, $5113 = 4 + (row >> 7)
8192 / 64 = 128 rows fill a bank exactly, so a row can never straddle a
bank; and every row is 64-byte aligned, so (kptr),y with y < 64 cannot
cross a page and the measured $6000 page-cross +1 never fires. Both
guarantees are the same ones the one-bank version relied on, and both survive
the extension unchanged - which is the entire reason 64 bytes was chosen over
full 256-byte alignment.
The layout is [layer][k|v][t] rather than the original [layer][t][k|v] so
that the score loop and the AV loop walk contiguous rows in t. Each such
loop therefore crosses a bank boundary at most once instead of alternating
between two banks on every position.
System RAM map (2 KB)
The four per-position arrays are T entries each: 128 bytes at T = 20,
340 at T = 85. That does not fit page 7 alongside the 16-bit AV
accumulators, so the map is derived from T:
| range | use |
|---|---|
$0000-$00FF |
zero page: stream pointers, counters, 16-bit accumulators |
$0100-$01FF |
stack |
$0200-$02FF |
BSS: res_tokens (96), res_ntok, P4HI (T) - linker-capped |
$0300 |
marker port (instrument) |
$0400-$04FF |
ACTB - biased activation page, page-aligned, the adc ACTB,y target |
$0500-$057F |
OUTB - signed matmul output, up to F = 128 entries |
$0580-$05BF |
XVEC - the residual stream |
$05C0-$05FF |
Q4HI - (q & 15) << 4 |
$0600-$063F |
ATTV - attention output |
$0640-$06FF |
free under the attention kernels; ACC8 / AVL / AVH in the legacy long-context path |
$0700-$07FF |
SCORL, SCORH, EXPE - T bytes each, laid end to end |
Every base is chosen so that base_lo + max_index <= 255: no indexed access
in this map crosses a page. That is a cycle per access, not a correctness
issue, but it is measurable and it was avoidable.
The attention rewrite deleted ACC8, AVL and AVH outright - the new AV
kernel keeps its accumulator in A and its 16-bit total in zero page, so the
three 64-byte block-accumulator arrays have no reader. They are still declared
in the legacy path that the long-context builds fall back to, which is why the
map above leaves $0640-$06FF for them rather than reusing it.
$0400 is a hard constant, not a BSS symbol, and the BSS memory area is
capped at $0200-$02FF in the linker config so that a result slot growing
into $0300 or $0400 is a link error, not a silent collision. (A silent
collision of exactly this kind previously made banks read back ROM signatures
and looked like an emulator fault.) P4HI is deliberately a BSS symbol rather
than a constant so that the cap covers it too.
The battery-backed result block ("ELYA", count, token ids) lives in the tail
of the last KV bank. At T = 85 the cache leaves 128 bytes there, and the
-DDEBUG snapshot pages have nowhere to go, so that build refuses to assemble
rather than quietly overwriting the cache.
The attention kernels and their two caches
QK sums over d for a fixed t; AV sums over t for a fixed d. Written
so the accumulator stays in A, the summation axis has to be a register
index and the other axis has to be unrolled into the instruction stream -
which means the two kernels want the cache laid out along opposite axes:
KT[d][l][t] = $6000 + d*64 + (l*20 + t) RAM bank 4 (QK: y = l*20+t)
V [t][l][d] = $6000 + t*256 + (l*64 + d) RAM bank 6 (AV: y = d)
Both strides are chosen so the indexed load can never cross a page (64 + 59
and 128 + 63 both stay under 256), which is worth a whole cycle per
multiply-add - the same argument that page-aligns the ternary gather chains.
The multiply row tbl_mul + (operand << 4) is written into the operand byte
of the adc, so the kernels live in PRG-RAM bank 5, mapped over the
weight-stream window at $8000 for the duration of attention (attention never
reads the weight stream) and copied there from the $A000 bank at reset.
RAMEXEC in FINDINGS is the measurement that says MMC5 PRG-RAM at $8000 is
writable, executable, and independent of the $6000 window.
3. PRG-ROM layout
96 KB PRG in twelve 8 KB banks, MMC5 PRG mode 3. A mixture build has a computed layout instead - see "Mixture of experts" at the end of this section.
| window | reg | contents |
|---|---|---|
$6000-$7FFF |
$5113 = 4 |
PRG-RAM: transposed key cache (bank 4) |
$6000-$7FFF |
$5113 = 6 |
PRG-RAM: value cache (bank 6) |
$8000-$9FFF |
$5114 |
weight stream window, slides 0..N; during attention $5114 = 5 maps PRG-RAM bank 5 here, holding the self-modified kernels |
$A000-$BFFF |
$5115 |
embedding and positional table while both fit one bank (they do at T = 20); otherwise one each, switched once per token |
$C000-$DFFF |
$5116 |
row headers + lookup tables (fixed) |
$E000-$FFFF |
$5117 |
fixed code bank: kernel, page chains, tables, vectors |
The embedding is V*D = 4,096 bytes and the positional table is T*D, which
at T = 85 is 5,440. Together that is 9,536 > 8,192, so they no longer share
a bank: embed_pos reads the embedding row, switches $5115, and adds the
positional row in place. Two bank writes per token, 12 cycles, against ~1.3
million. One spare bank keeps the image a whole number of 16 KB iNES units.
Weight stream format
The stream is written once, read strictly sequentially forward, exactly once per token, in the same order every token. Per matrix row:
byte n_pos real count of +1 weights
byte n_neg real count of -1 weights
n_pos bytes input indices with weight +1
n_neg bytes input indices with weight -1
Indices are one byte because the largest input dimension is F = 128.
Rows are emitted in fixed order: for each layer, Wq, Wk, Wv, Wo, W1, W2;
then the output head. 3 x (4x64 + 128 + 64) + 64 = 1,408 rows per token.
Expected size at 50% zeros (P(0)=0.5, P(+1)=P(-1)=0.25):
index bytes 0.5 x 102,400 = 51,200
row headers 4 x 1,408 = 5,632 (n_pos, n_neg, d7 lo, d7 hi)
------
stream 51,200 bytes = 6.25 banks of 8 KB; the image is padded to 7
Row-to-bank mapping and BANK CROSSINGS PER TOKEN
This is the number that decided the Genesis and N64 outcomes.
The packer emits rows contiguously and, when the next row would straddle an 8 KB bank boundary, pads to the boundary and starts the row in the next bank. So no row ever straddles a bank, and the reader only ever needs to switch when it steps from one bank to the next, in order.
bank crossings per token = ceil(stream_bytes / 8192) - 1 ~= 6
switch cost = 73 cycles (MEASURED IN THE LOOP, see below)
total ~= 511 cycles per token
Correction, from the MoE work. This section used to say the switch cost
was "6 cycles (measured, MMC5)" for a total of ~42 cycles a token. Six cycles
is the sta $5114 alone. What the loop actually pays is the whole sentinel
body - bump the bank, rebuild the OR mask, store it, reset the self-modified
chain pointer, step the header pointer, re-read the displaced header row - and
a -DBANKPROF build brackets that body in situ and reports 71 cycles
(increment sentinel) or 73 (absolute sentinel), identically on every one
of 114 and 247 samples respectively. The conclusion does not change: at 73
cycles and 7 switches this is 511 cycles of 1,117,063, i.e. 0.046%. But
the old figure was wrong by 12x and is corrected here rather than left
standing.
Against a measured 1,117,063 cycles per token that is 0.046%. Bank switching is a non-issue for this port, and the reason is structural, not lucky: the access pattern is a single forward sweep, never a random walk. (On MMC1 the same 7 switches would cost 210 cycles - still nothing. The mapper choice matters for code layout, not for throughput, and MMC5 is chosen because 6 < 30 and because it gives three independent windows plus 32 KB of banked PRG-RAM.)
Mixture of experts: a computed bank map
A mixture build keeps Wq/Wk/Wv/Wo, the embedding, the positional table and
the head shared, and gives W1/W2 of every layer N copies. The weights
STREAMED per token stay at 102,400; the weights ON THE CARTRIDGE become
53,248 + N x 49,152.
The bank map is computed by host/ref.py and written out as three generated
artifacts - moe.inc (assembler constants), moebanks.inc (the .incbin
lines) and nnmoe.cfg (the linker config) - because a bank map maintained by
hand in three places is how a ROM comes to run perfectly and say the wrong
thing.
bank 0 .. S-1 shared stream (attention rows + head)
bank S + e*K .. +K-1 expert e's feed-forward rows
bank S + N*K embedding (+ positions)
bank S + N*K + 1 positional table
bank S + N*K + 2 + e expert e's HEADER TABLE + a copy of the lookup
tables at $D700 (asserted identical addresses)
last bank code, chains, vectors
The copy at $D700 is 2,063 bytes under the power-of-two normaliser and
2,101 under the exact one, which adds tbl_e8lo, tbl_e8hi and
tbl_p4. Those three have to be in the per-expert copy rather than in the
fixed bank because the normaliser walks them inside softmax, with $C000
showing whichever bank the router chose. tbl_pv goes the other way: one copy
in the fixed bank, because that bank is mapped whichever expert won.
Two things make this work at all:
- The sentinel carries an absolute bank number.
$FF, bank, 0, 0. The old "increment" form cannot express a walk that leaves the shared region for an expert's banks and comes back three times a token. - Every region chunk starts at a bank boundary. The ROM's stream offset
is implicit -
chain_resetputs the gather index back to 0 on every switch - so the only place a walk can resume is the start of a bank. This wastes part of a bank per chunk and buys a sentinel with no resume offset in it.
The router is the whole of the routing machinery:
ldx curtok ; 2
lda routebank,x ; 4 which expert's header bank
sta MMC5_PRGC000 ; 4
Measured costs: 13 bank switches a token instead of 7, at 73 cycles, plus the 10-cycle router = 448 cycles a token, 0.040%. The switch count is 13 at N = 4, 8 and 16 alike. The ceiling is N = 16 (122 of the MMC5's 128 banks, 839,680 ternary weights); N = 17 needs 129 and the packer refuses.
The page chains
ldy <stream>,x needs an assembled absolute base. The $8000 window is 8 KB
= 32 pages, so the fixed bank contains 32 specialised gather chains, one
per page $8000, $8100, ... $9F00:
chain_p:
ldy $p00+0,x ; 4
adc ACTB,y ; 4
ldy $p00+1,x
adc ACTB,y
... 16 entries ...
jmp gather_fold
16 entries x 6 bytes + 3 = 99 bytes per chain, 32 chains = 3,168 bytes, plus a
64-byte pointer table. A row's list is walked by pointing chain_ptr at the
chain for the row's page and putting the row's low byte in X. Lists shorter
than a multiple of 16 enter the chain part-way (Duff's device) at
chain + (16-r)*6; nothing is padded, so no padding entry is ever
executed - padding would have cost 8 cycles each and been the dominant
overhead.
ldy $p00+k,x with k <= 15 and x <= 255 can reach into the next page.
That is correct - the window is contiguous - and only costs the measured
page-cross +1. Rows do not straddle banks; they may straddle pages.
4. Predicted cost model (to be confirmed by measurement)
Per MAC in the ternary kernel:
| cycles | |
|---|---|
ldy chain,x + adc ACTB,y |
8 |
| fold + X advance + loop, amortised over 16 | ~1.9 |
| predicted per MAC | ~10 |
Per token:
| stage | count | cycles each | total |
|---|---|---|---|
| ternary gathers | 51,200 | ~10 | ~512,000 |
| output requantise (16-bit -> 4-bit) | 1,408 | ~40 | ~56,000 |
attention MACs (QK + AV, full context, T = 20) |
7,680 | ~23 | ~177,000 |
| softmax / residual / misc | - | - | ~40,000 |
| bank switches | 6 | 6 | 42 |
| predicted total | ~810,000 |
= ~0.45 s per token at 1,789,772 Hz.
The prior run's extrapolation was ~0.8 s/token for 114,688 active weights, i.e. ~12.2 cycles per weight including zeros. The prediction here is lower because sign separation means a zero weight costs nothing at all - it is simply absent from both index lists - so the cost scales with nnz, not with the weight count. Whether that survives contact with a real measurement is exactly what the kernel measurement is for.
How that scales with T. Only the attention row moves, and it moves as
T^2: L*H*(p+1)*DH*2 MACs at position p, so 7,680 at T = 20's last
position and 32,640 at T = 85's. Everything else in the table is constant.
Since the attention term is the only one that grows, the mean cost per token
grows much less than the peak does - the mean sees (p+1) averaged over the
run, i.e. about half of full context either way. The measured figures are in
FINDINGS; this paragraph is the prediction they are checked against.
5. Exactness
Everything is integer and the host reference in host/ref.py is the
specification. Both sides implement, bit for bit:
- activations: signed 4-bit,
-7..+7, one per byte, biased by +7 in ACTB - ternary matmul:
acc16 = (sum_pos - 7*n_pos) - (sum_neg - 7*n_neg) - requantise 16-bit to 4-bit:
q = acc >> k(arithmetic, floor) clamped so thatacc > 8*2^k - 1 -> +7andacc < -7*2^k -> -7 - attention products via a 256-byte ROM table
mul[(nq<<4)|nk] = (q*k) >> 2(measured to be exactlyfloor(q*k/4)over the whole reachable -7..7 range; the table's saturation clamp never fires) - softmax: max-shift, 15-entry exp table, power-of-two normalisation, clamp
to
0..7. This is a quantised softmax, stated exactly, not an approximation of a float one - residual: 4-bit add then clamp to
-7..+7 - sampling: argmax over the 64 logits, ties to the lowest index (deterministic, so the ROM and the host must agree on the token id at every position)
Requantise shifts. K_SHIFT, W2_SHIFT, AV_SHIFT and SM_SHIFT are
now generated by host/ref.py into out/model/shifts.inc, which rom/nn.s
includes, so the specification and the kernel cannot hold different values.
AV_SHIFT was 4 in the first cut and is now 2: the attention accumulator
is provably bounded by 14, so a shift of 4 left it with two reachable output
levels, and the provably-correct shift of 1 measured worst of the ladder.
See FINDINGS. (This paragraph said "is now 1" until 2026-08-08; the shipped
value has been 2 since the ladder was run.) The context length T is stamped
into the npz the same way and for the same reason.
What the quantised softmax can express. The shipped normalisation now
picks p_t = min(e_t * SM_TARGET / S, PMAX) exactly; the power-of-two form
below is what shipped until 2026-08-09 and is still selectable with
NES_SM_NORM=pow2. Either way the budget is the same:
sum_t p_t <= 8 with every p_t an integer in 0..7
at most 8 of the T positions can carry any attention weight at all, and
that ceiling does not move when T does. Extending the window extends what
the model may look at, not how much it may look at. This is a property of the
kernel, stated here so that a context result can be read against it rather
than around it.
That property is real and it was NOT what was limiting the model. Raising
the budget to 16 and to 32 was measured, at two seeds each, and is worth
nothing (1.5279 / 1.5335 / 1.5267 nats/char - not even monotone). What the
old normaliser was costing is the difference between the budget and the
realised sum: kk is the smallest shift with S >> kk <= 8, so the
realised sum lands anywhere in (4, 8], and measured it landed on 4 in a
quarter of all softmax evaluations. Removing that waste - same nibble, same
budget - is worth 0.0239 nats/char at 60,000 steps and two seeds. See
FINDINGS.
The bar is a bit-exact match of every generated token id over 16+ tokens, not a 3-token spot check - a 3-token check passed two genuinely broken changes elsewhere in this project and only failed at positions 5 and 12.
6. The softmax probability budget (added when the integer softmax was revisited)
Section 5 ends with what the quantised softmax can express: sum_t p_t <= 8
with every p_t an integer in 0..7, so at most 8 of the T positions can
carry any weight. That is stated there as a property. This section is the
design for making it a parameter, written before the kernel changed.
The family
SM_TARGET is the sum the normaliser targets and PMAX = SM_TARGET - 1 the
per-position clamp. The attention accumulator is
acc = sum_t floor(p_t * v_t / 2^PMUL_SHIFT)
with sum_t p_t <= SM_TARGET and |v_t| <= 7
so |acc| <= SM_TARGET * 7 / 2^PMUL_SHIFT
Choosing
PMUL_SHIFT = 2 + log2(SM_TARGET / 8)
pins that bound at 14 for every member of the family. This is the whole
design. It means widening the probability does not move AV_SHIFT, does not
move the range of the attention output, and does not move where the
requantise table saturates - so the AV_SHIFT ladder that was measured at
SM_TARGET = 8 carries over instead of having to be re-run from scratch.
It also means the product table's entries stay in the same band, so the AV chain's carry-free block barely moves:
| SM_TARGET | PMUL_SHIFT | max table entry | carry-free block | |acc| bound |
|---:|---:|---:|---:|---:|
| 8 (shipped) | 2 | 25 | 10 | 14 |
| 16 | 3 | 27 | 9 | 14 |
| 32 | 4 | 27 | 9 | 14 |
Where the family stops being free
A probability is stored pre-shifted as p<<4: it is the low byte of that
probability's row in a page-aligned 256-byte product table, and av_patch
writes it into the AV chain's operand with a single sta. That is what makes
the AV inner loop 8 cycles instead of 9 (see the ATTNBENCH head-to-head).
SM_TARGET = 32 needs 32 rows = 512 bytes, so the row address no longer fits
one patched byte and av_patch grows a second store per live unit. The
packer refuses it. The trainer and the host reference still implement it, so
the question "would 32 have bought anything?" can be answered without paying
for it.
The knob that actually mattered was not in this family
Everything above is about how many bits a probability gets. Measured, that buys nothing. The normaliser - how much of the budget the kernel actually hands out - is worth 0.0239 nats/char for +0.37% cycles, and its design is in FINDINGS rather than here because it was found by the screen, not predicted by it. The honest record is that this section correctly predicted the cost of the family and correctly identified that it was nearly free, and was aimed at the wrong knob.
What it costs, predicted
Nothing in the softmax kernel: the kk loop compares against SM_TARGET + 1
instead of a literal 9, and tbl_p holds min(e>>kk, PMAX) << 4 instead of
min(e>>kk, 7) << 4. Same instructions, same table shapes.
Nothing in the AV inner loop: it is the same ldx VBASE+t*256,y ; adc tbl_pv,x pair against a different 256-byte table, measured at 8.00
cycles/MAC either way.
The cost is entirely that more positions carry a nonzero nibble, so more AV chain units run - the AV chain is built from the live positions only, and 87.5% of its potential multiply-adds are currently absent rather than zero. The prediction is therefore that the cost tracks the live-position count and nothing else, and that is what FINDINGS measures.
Where the table went
tbl_pv could not go in the $C000 table bank: that bank was already at
2,063 of its 2,304 usable bytes and 256 more overflowed it by exactly 256.
It lives in the fixed $E000 bank, which is mapped at all times, so the
PRG-RAM kernels reach it with a plain absolute adc. The overflow was
initially invisible because build.sh piped ld65 through grep and lost
its exit status; that is fixed, and it is recorded in FINDINGS because it
nearly produced a measurement of a stale ROM.
Update, from the merge. Two numbers here move once both branches are on one tree, and they move in opposite directions:
- The
$C000table bank now holds 2,101 bytes of lookup tables, not 2,063: the exact normaliser addstbl_e8lo,tbl_e8hi(EXP_Nbytes each) andtbl_p4(SM_TARGETbytes), 38 in all at the shipping configuration. The argument for evictingtbl_pvis unchanged and stronger. - Those 38 bytes are the one part of the exact normaliser that a mixture build
has to duplicate per expert. They are read inside
softmax, andsoftmaxruns with$C000showing the bank the router chose, so a single shared copy would simply not be mapped.tbl_pvis the opposite case and needs exactly one copy for the same reason it was evicted - the fixed bank is mapped whichever expert won.host/ref.pyemits both placements, and every per-expert copy is.assert-ed to land at the same address as expert 0's, which is the only thing standing between a routed token and reading its exp table out of thin air.