crystal-ball: flashnext expert guessers
Small side files for Flash-Next SSD, a fork of llama.cpp that streams the Qwen3.8-Flash-Next experts from SSD. Each one scores all 512 experts of a layer ahead of time, so the fork can start fetching them early. They change which experts are fetched early, never which are used: output is unchanged.
| file | guesses | switch | size | sha256 |
|---|---|---|---|---|
c1-guesser.gguf |
two layers ahead (layers 2-47, from layer n-2) | LLAMA_MOE_STREAM_LOOKAHEAD_GUESSER |
120,686,368 bytes | c22af709d65b99bb2bc64aaa5b8b2d47c04c86367b0f9ba0fe452ab47ec053c7 |
c1-guesser-gap1.gguf |
one layer ahead (layers 1-47, from layer n-1) | LLAMA_MOE_STREAM_GUESSER1 |
123,309,952 bytes | 2df6a3d9c63f787286a387965aa1c89da3ee86c6b75829e3c9a1e2ca24690d94 |
Both are one linear layer per layer, 2560 -> 512 plus bias, f16 weights and f32 bias, started from the model's own router for the layer they guess. Only valid for Qwen3.8-Flash-Next (n_embd 2560, 512 experts); the engine refuses a shape mismatch.
c1-guesser.gguf: two layers ahead
- Confidence:
min(1, 10 * softmax(z)); the fork fetches experts above a cut-off (we use 0.2,LLAMA_MOE_STREAM_LOOKAHEAD_CONF=0.2). - Use it with: engine b4745dbc3 or later, with
LLAMA_MOE_STREAM_LOOKAHEAD_ALL=1 ..._DEPTH2=4 ..._GUESSER=<this file> ..._CONF=0.2 ..._KEEP=1. - Quality: two-layer top-10 recall (mean of layers 2-47) 67.2% on held-out generic papers, 71-72% on held-out chat/code papers. On a Mac mini M5 Pro 64GB with the 28 GiB cache and the MTP draft: writing +4.2% (code, high effort) and +8.2% (chat, medium effort) against the same engine without it; the gain is +6-11% on papers up to long and about +1% past 100K tokens of context.
c1-guesser-gap1.gguf: one layer ahead
- What it replaces: the engine's own one-layer guess, which borrows the next layer's router and reads it with this layer's input. Before training this file is exactly that guess; trained, it ranks the next layer's experts from this layer's input directly.
- Use it with: engine a5290c843 or later,
LLAMA_MOE_STREAM_GUESSER1=<this file>, beside the two-layer file and the keep rule above (the lookahead is on by default). - Quality: one-layer top-10 recall (mean of layers 1-47) 72.6% on held-out away papers against 65.0% for the borrowed guess, 76.7% against 60.8% on held-out chat/code/home papers, ahead on 43 and 47 of 47 layers. On the same Mac with the two-layer file already on: writing +7.5% (code, high effort) and +7.3% (chat, medium effort), on 12 conversations each, run base / guesser / base. Most of the gain is fewer wasted fetches: about 38 fewer guessed experts fetched per written token.
Trained on the router inputs recorded while the model wrote replies to benchmark and coding/chat prompts.
- Downloads last month
- 40
Hardware compatibility
Log In to add your hardware
We're not able to determine the quantization variants.
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐ Ask for provider support