File size: 4,141 Bytes
32c4433 cbe0fea 32c4433 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | ---
license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: llama.cpp
tags:
- gguf
- llama.cpp
- moe
- mixture-of-experts
- qwen3
- inference
- streaming
---
# Qwen3-4B-Instruct-2507 — exact 8-expert MoE reformat
This repository contains a **format-level conversion** of
[Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)
from a dense GGUF to an 8-expert `qwen3moe` GGUF that llama.cpp can load.
It is **not a trained MoE**. The dense FFN is the sum of its hidden neurons:
```
FFN(x) = Σ_j W_down[:, j] · silu(W_gate[j] @ x) · (W_up[j] @ x)
```
The conversion partitions the 9,728 hidden neurons into 8 disjoint slices of
1,216 neurons. With all 8 experts selected and a uniform router, the result is
the same function. The exact model matches the source within F16 round-off.
## Files
| file | size | description |
|---|---:|---|
| `Qwen3-4B-MoE8-exact-f16.gguf` | 6.25 GB | exact full-fanout model; **use this one** |
| `Qwen3-4B-MoE8-ridge6-f16.gguf` | 6.25 GB | lossy top-6 variant, 75% of FFN bytes |
| `Qwen3-4B-MoE8-exact-f16.manifest.json` | 2.1 MB | tensor offsets, expert byte ranges, permutation hashes |
Checksums are in [`results/CHECKSUMS.txt`](https://github.com/DrMkdaddy/qwen-moe-stream/blob/main/results/CHECKSUMS.txt).
The non-FFN tensors (attention, embeddings, LM head) are copied bit-for-bit
from the source Q4_K_M file. The FFN experts are written F16 after
dequantization. The source is Q4_K_M, so the F16 conversion carries that
quantization's round-off.
## Run with llama.cpp
```bash
llama-cli -m Qwen3-4B-MoE8-exact-f16.gguf \
-p "The capital of France is" -n 16 -ngl 0 -t 8 --temp 0
```
`llama.cpp` build b11050 or newer is required for the `qwen3moe` graph. The
exact model is a normal dense-equivalent FFN, so it is slower than the Q4
source: the FFN is F16 and all 8 experts run.
## Measurements
Held-out text, 512-token context, CPU, `llama-perplexity`:
| model | PPL | ratio vs dense |
|---|---:|---:|
| dense Qwen3-4B Q4_K_M | 17.6203 | 1.000× |
| MoE8 exact (all 8 experts) | **17.6157** | 0.9997× |
| MoE8 top-6 ridge router | 20.8920 | 1.186× |
Logit comparison, exact vs dense:
| metric | value |
|---|---:|
| mean KL divergence | 0.001350 |
| same top token | 96.47% |
| PPL ratio | 0.99972 |
Streaming, 4 tokens × 36 layers, real expert tensors, `madvise` after use:
| schedule | peak RSS | bytes touched |
|---|---:|---:|
| resident, no eviction | 5.93 GB | 21.5 GB |
| streamed, evict after use | **0.78 GB** | 21.5 GB |
The 0.78 GB figure is the executor alone, not a full serving stack. The exact
model's FFN is 5.38 GB; the file is 6.25 GB.
## What top-6 means
The top-6 file is a **lossy** point on the quality/IO curve. It reads 75% of
the FFN bytes and raises PPL from 17.62 to 20.89. The experts are slices of a
dense FFN, not experts trained to produce complete updates; selecting a subset
drops real terms. The exact file is the faithful one. See the project README
for the oracle ceiling and the negative result on gradient-trained routers.
## Calibration and provenance
The ridge router was fitted on 384 token positions per layer from public project READMEs
and specification documents, dumped with a llama.cpp eval callback. No model
weights were trained. No private user data was used. The conversion scripts
and full methodology live at:
- GitHub: https://github.com/DrMkdaddy/qwen-moe-stream
- Source model: https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507
## Limitations
- The exact model is a reformat, not a new model, and not a speedup. All 8
experts run; only resident memory is improved by streaming.
- The top-6 model is lossy and should not be presented as equivalent.
- Only the FFN is MoE-ified. Attention, embeddings, and the LM head stay dense.
- Measurements are from one CPU host with `-ngl 0`; GPU offload will change
the memory and throughput numbers.
- F16 experts from a Q4_K_M source are larger than the source file.
## License
The conversion code is MIT. The weights are derived from
Qwen/Qwen3-4B-Instruct-2507 and are distributed under **Apache-2.0**.
|