Router entropy dispersion, 61% softmax mass truncation, and O(1) state decoding on 6GB VRAM
Hi Vertex AGI team,
Pulling off a full-parameter SFT of a 1.03B fine-grained MoE (394M active params) on a single GTX 1660 Ti with 6GB VRAM using 8-bit AdamW and gradient checkpointing is incredible resourcefulness. Withholding the GGUF export after catching the 61% softmax mass truncation in llama.cpp's DeepSeek implementation shows serious technical integrity.
A few architectural observations on your routing dynamics and generation behavior:
Fine-grained expert entropy and 61% softmax truncation:
Your 24 routed experts use a narrow intermediate dimension of 640 (roughly 0.625x of hidden size 1024). At 1B scale, allocating parameters across 24 micro-experts often leads to high routing entropy, because the representational difference between narrow experts is small.
When top-6 experts capture only 61% of total softmax mass, the remaining 39% sits in an unselected long tail. Forcing post-hoc renormalization scales up whatever top-6 logits cleared the threshold, which amplifies small numerical fluctuations. This causes expert assignments to flicker erratically across adjacent tokens, directly explaining the subword tokenization artifacts (like "broccoli,iant cucumber") and rapid loss of sentence coherence in the evaluation transcripts.Full-sequence attention recomputation:
Because your generation loop currently recomputes full multi-head attention across the entire sequence at every decode step without a KV-cache, inference compute scales quadratically O(L^2). On a 6GB Turing GPU with limited memory bandwidth, recomputing 20 layers of attention from scratch on every token introduces heavy compute latency as sequence length grows toward 2,048 tokens.Constant O(1) state decoding in hybrid architectures:
In an open architecture project called Maba v2 (reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we avoid the quadratic decode penalty and routing dispersion through a 3:1 macro-topology:
We route 75% of depth through Decoupled Gated Delta Attention (DGDA) paired with MABA-SA sparse attention and NoPE.
Because linear recurrence updates an associative state matrix in constant O(1) memory, tokens generate without quadratic history recomputations or massive KV-cache allocations. This allows long sequences to run at flat decode latency even on budget 6GB consumer cards.
Additionally, combining persistent state transitions with MABA-SA prevents the representational dilution that typically causes compact decoders to drift off-topic after the first sentence.
If you are planning a Quartz v2 or building custom inference kernels, checking out the recurrent state formulation and hybrid layer layout in the Maba v2 repo might provide some useful structural ideas for lightweight GPUs.
Did you notice if fine-tuning on Dolly-15k concentrated router mass into specific experts compared to the base pretrain, or did the top-6 softmax share remain around 61% across both checkpoints?
Best,
Andrew