plunderstruck commited on
Commit
1960cd8
Β·
verified Β·
1 Parent(s): 4679a39

Add card: Qwen3-Coder-Next ROCmFP4 (Q8 emb, code-weighted imatrix); measured -20% median KL on held-out code

Browse files
Files changed (1) hide show
  1. README.md +115 -0
README.md ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen3-Coder-Next
3
+ license: apache-2.0
4
+ library_name: gguf
5
+ tags:
6
+ - gguf
7
+ - rocmfp4
8
+ - qwen3next
9
+ - qwen3-coder-next
10
+ - coder
11
+ - moe
12
+ - imatrix
13
+ - strix-halo
14
+ - amd
15
+ - rocm
16
+ - vulkan
17
+ language:
18
+ - en
19
+ ---
20
+
21
+ # Qwen3-Coder-Next β€” ROCmFP4 STRIX (Q8 embeddings, imatrix)
22
+
23
+ Experimental **AMD Strix Halo (gfx1151)** quant of [**Qwen3-Coder-Next**](https://huggingface.co/Qwen/Qwen3-Coder-Next)
24
+ β€” Qwen's agentic coding model (80B total / **3B active** high-sparsity MoE, hybrid Gated-DeltaNet attention,
25
+ 262K context) β€” in the custom **ROCmFP4** 4-bit format, **imatrix-quantized** with a code-weighted
26
+ importance matrix.
27
+
28
+ > **⚠️ Ignore HuggingFace's auto-detected quant badge ("F16"/16-bit) β€” it's wrong.**
29
+ > HF can't read the custom ROCmFP4 tensor types and mislabels the file. **These are ~4.5 bpw 4-bit
30
+ > ROCmFP4 files, not 16-bit.** Pick by filename in *Files and versions*.
31
+
32
+ > ## Requires the ROCmFP4 fork (public) β€” not stock llama.cpp
33
+ > Uses the **ROCmFP4** tensor types (`q4_0_rocmfp4`, `q4_0_rocmfp4_fast`). **Stock llama.cpp, LM Studio,
34
+ > Ollama, etc. cannot load it.** Build/run with
35
+ > **[`charlie12345/rocmfp4-llama`](https://github.com/charlie12345/rocmfp4-llama)** (`mtp-rocmfp4-strix`).
36
+
37
+ ## Why this is a great fit for a single APU
38
+
39
+ - **Fast even without speculative decoding.** 3B active params + linear Gated-DeltaNet attention β†’
40
+ **~54 t/s** short-context decode on a Ryzen AI Max+ 395 (Vulkan0), and cheap long context (DeltaNet's
41
+ near-constant memory). No MTP needed.
42
+ - **Note on MTP:** Qwen3-Coder-Next ships **without** an MTP head, and the ROCmFP4 fork currently wires
43
+ MTP drafting only for the `qwen35`/`qwen35moe` archs, **not** `qwen3next`. So these are **no-MTP** builds.
44
+ In practice it doesn't matter much β€” it's already ~2Γ— the decode speed of the 27B/40B MTP models.
45
+
46
+ ## Two files (pick your trade-off)
47
+
48
+ Both share the same 4-bit ROCmFP4 body, **Q8 embeddings**, and the code-weighted imatrix. They differ only
49
+ in the **output head**:
50
+
51
+ | File | output head | best for |
52
+ |---|---|---|
53
+ | `…-STRIX-embQ8-imatrix.gguf` | 4-bit | **fastest** β€” recommended default |
54
+ | `…-STRIX-embQ8-imatrix-headQ6.gguf` | **Q6_K** | a notch more faithful output, ~5–7% slower decode |
55
+
56
+ - **Q8 embeddings** (not f16): the source is Q8_0, so Q8 matches its precision exactly β€” f16 would be
57
+ fake-f16 bloat for zero gain (embeddings are a lookup, not a matmul).
58
+ - **MoE experts + attention/SSM projections** β†’ 4-bit ROCmFP4; the DeltaNet-specific tensors
59
+ (`ssm_conv1d`, `ssm_a`, norms, router) stay **F32**.
60
+
61
+ ## The imatrix β€” code-weighted, and measured
62
+
63
+ Quantized **with an importance matrix** built from a **code-weighted** calibration mix (~2.6:1 code:general):
64
+ - **Code:** real multi-language source + code-analysis prompts, sampled from
65
+ [`eaddario/imatrix-calibration`](https://huggingface.co/datasets/eaddario/imatrix-calibration).
66
+ - **General:** Kalomaze's `groups_merged` (via [`froggeric/imatrix`](https://huggingface.co/datasets/froggeric/imatrix)).
67
+
68
+ *(Note on "dual imatrix": a plain merge of two imatrices is mathematically identical to concatenating the
69
+ corpora at the same ratio β€” the only real lever is the code:general ratio, which is what's set here. True
70
+ size-decoupled balancing would need normalized-merge tooling; not used.)*
71
+
72
+ **Measured benefit** β€” KL-divergence + perplexity vs the **Q8 reference** on a **held-out code** slice
73
+ (disjoint from calibration), imatrix vs no-imatrix:
74
+
75
+ | metric (vs Q8, held-out code) | no-imatrix | **imatrix** | change |
76
+ |---|---|---|---|
77
+ | **Median KLD** | 0.00597 | **0.00478** | **βˆ’20%** |
78
+ | 90th-pct KLD | 0.1342 | 0.1083 | βˆ’19% |
79
+ | **RMS Ξ”p** | 8.14% | **7.36%** | βˆ’10% |
80
+ | **Same top token as Q8** | 91.01% | **91.49%** | +0.48 pp |
81
+ | Mean PPL | 3.4556 | 3.4686 | +0.013 (within Β±0.077 noise β€” a wash) |
82
+
83
+ So the imatrix **measurably improves quantization fidelity to the full model on code** (median KL βˆ’20%,
84
+ the gold-standard metric), at **zero cost** (same size/speed). PPL is a statistical wash. Honest scope:
85
+ this is a fidelity-vs-Q8 measurement on ~20 K tokens of held-out code, **not** an absolute coding benchmark.
86
+
87
+ ## How it was built (reproducible)
88
+
89
+ ```bash
90
+ # code-weighted imatrix on the Q8 (single pass; ratio = the real lever)
91
+ llama-imatrix -m Qwen3-Coder-Next-Q8_0.gguf -f code-weighted-calib.txt -o coder-next.imatrix -c 512 -ngl 999
92
+
93
+ # quant -> ROCmFP4 with the imatrix (Q8 embeddings). headQ6 adds --output-tensor-type q6_K
94
+ llama-quantize --allow-requantize --token-embedding-type q8_0 --imatrix coder-next.imatrix \
95
+ Qwen3-Coder-Next-Q8_0.gguf Qwen3-Coder-Next-ROCmFP4-STRIX-embQ8-imatrix.gguf Q4_0_ROCMFP4_STRIX
96
+ ```
97
+
98
+ ## Usage
99
+
100
+ ```bash
101
+ env HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
102
+ llama-server -m Qwen3-Coder-Next-ROCmFP4-STRIX-embQ8-imatrix.gguf \
103
+ --alias qwen3-coder-next-rocmfp4 --host 0.0.0.0 --port 8080 \
104
+ -dev Vulkan0 -ngl 999 -fa on -c 131072 -b 2048 -ub 256 -t 16 -tb 16 \
105
+ -ctk f16 -ctv f16 --cache-reuse 256 \
106
+ --temp 0.7 --top-p 0.8 --top-k 20 --jinja --parallel 1 --metrics --no-mmap
107
+ ```
108
+
109
+ (No `--spec-type draft-mtp` β€” this arch has no MTP head; it's fast on its own.)
110
+
111
+ ## Credits & license
112
+
113
+ - **Model:** [`Qwen/Qwen3-Coder-Next`](https://huggingface.co/Qwen/Qwen3-Coder-Next) (Apache-2.0, Qwen team) β€” a derivative quantization; verify the base terms before redistribution/use.
114
+ - **Calibration:** [`eaddario/imatrix-calibration`](https://huggingface.co/datasets/eaddario/imatrix-calibration) (code), Kalomaze `groups_merged` via [`froggeric/imatrix`](https://huggingface.co/datasets/froggeric/imatrix) (general).
115
+ - **ROCmFP4 format & runtime:** [`charlie12345/rocmfp4-llama`](https://github.com/charlie12345/rocmfp4-llama) (based on llama.cpp, MIT).