crystal-ball: flashnext expert guessers

Small side files for Flash-Next SSD, a fork of llama.cpp that streams the Qwen3.8-Flash-Next experts from SSD. Each one scores all 512 experts of a layer ahead of time, so the fork can start fetching them early. They change which experts are fetched early, never which are used: output is unchanged.

file guesses switch size sha256
c1-guesser.gguf two layers ahead (layers 2-47, from layer n-2) LLAMA_MOE_STREAM_LOOKAHEAD_GUESSER 120,686,368 bytes c22af709d65b99bb2bc64aaa5b8b2d47c04c86367b0f9ba0fe452ab47ec053c7
c1-guesser-gap1.gguf one layer ahead (layers 1-47, from layer n-1) LLAMA_MOE_STREAM_GUESSER1 123,309,952 bytes 2df6a3d9c63f787286a387965aa1c89da3ee86c6b75829e3c9a1e2ca24690d94

Both are one linear layer per layer, 2560 -> 512 plus bias, f16 weights and f32 bias, started from the model's own router for the layer they guess. Only valid for Qwen3.8-Flash-Next (n_embd 2560, 512 experts); the engine refuses a shape mismatch.

c1-guesser.gguf: two layers ahead

  • Confidence: min(1, 10 * softmax(z)); the fork fetches experts above a cut-off (we use 0.2, LLAMA_MOE_STREAM_LOOKAHEAD_CONF=0.2).
  • Use it with: engine b4745dbc3 or later, with LLAMA_MOE_STREAM_LOOKAHEAD_ALL=1 ..._DEPTH2=4 ..._GUESSER=<this file> ..._CONF=0.2 ..._KEEP=1.
  • Quality: two-layer top-10 recall (mean of layers 2-47) 67.2% on held-out generic papers, 71-72% on held-out chat/code papers. On a Mac mini M5 Pro 64GB with the 28 GiB cache and the MTP draft: writing +4.2% (code, high effort) and +8.2% (chat, medium effort) against the same engine without it; the gain is +6-11% on papers up to long and about +1% past 100K tokens of context.

c1-guesser-gap1.gguf: one layer ahead

  • What it replaces: the engine's own one-layer guess, which borrows the next layer's router and reads it with this layer's input. Before training this file is exactly that guess; trained, it ranks the next layer's experts from this layer's input directly.
  • Use it with: engine a5290c843 or later, LLAMA_MOE_STREAM_GUESSER1=<this file>, beside the two-layer file and the keep rule above (the lookahead is on by default).
  • Quality: one-layer top-10 recall (mean of layers 1-47) 72.6% on held-out away papers against 65.0% for the borrowed guess, 76.7% against 60.8% on held-out chat/code/home papers, ahead on 43 and 47 of 47 layers. On the same Mac with the two-layer file already on: writing +7.5% (code, high effort) and +7.3% (chat, medium effort), on 12 conversations each, run base / guesser / base. Most of the gain is fewer wasted fetches: about 38 fewer guessed experts fetched per written token.

Trained on the router inputs recorded while the model wrote replies to benchmark and coding/chat prompts.

Downloads last month
40
GGUF
Model size
61.6M params
Architecture
Hardware compatibility
Log In to add your hardware

We're not able to determine the quantization variants.

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support