Request for Unsloth Qwen4-35B-A3B-Distill-FlashNext

#2
by userr99 - opened

How plausible is
Qwen4-35B-A3B-Distill-FlashNext using:

32–48 layers
~2560–3072 hidden size
Qwen Sparse Attention
Gated DeltaNet hybrid
4-stream Gated Residual / Hyper-Connections
smaller PLE / n-gram memory
maybe 128–256 experts rather than Flash-Next’s 512
~6–12 routed experts/token
~3–4B active/token
same tokenizer as Flash-Next
Flash-Next itself as the teacher

That would preserve the architectural advantages instead of trying to cram Flash-Next behavior and assumes the precedent that the architecture really can run independently since it's already been implemented in llama.cpp-compatible and even standalone CPU runtimes, including Gated Residual, GDN, QSA, PLE and the MoE routing machinery.

Sign up or log in to comment