plunderstruck commited on
Commit
abdce29
Β·
verified Β·
1 Parent(s): 3ffb142

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +12 -29
README.md CHANGED
@@ -85,14 +85,12 @@ Experimental **AMD Strix Halo (gfx1151)** quant of [**Qwen3-Coder-Next**](https:
85
  <th style="border:1px solid currentColor; padding:7px 10px; text-align:left; text-transform:uppercase; font-size:10px; letter-spacing:1px;">Pick if</th>
86
  </tr></thead>
87
  <tbody>
88
- <tr><td style="border:1px solid currentColor; padding:7px 10px;"><code>…-COHERENT-embQ8-headQ6.gguf</code> β˜…</td><td style="border:1px solid currentColor; padding:7px 10px;">Q6_K</td><td style="border:1px solid currentColor; padding:7px 10px;"><b>recommended</b> β€” all-dual-scale body, lowest measured KL vs Q8 (Β§04)</td></tr>
89
- <tr><td style="border:1px solid currentColor; padding:7px 10px;"><code>…-STRIX-embQ8-imatrix-headQ6.gguf</code></td><td style="border:1px solid currentColor; padding:7px 10px;">Q6_K</td><td style="border:1px solid currentColor; padding:7px 10px;">fast body β€” ~same fidelity, slightly smaller</td></tr>
90
- <tr><td style="border:1px solid currentColor; padding:7px 10px;"><code>…-STRIX-embQ8-imatrix.gguf</code></td><td style="border:1px solid currentColor; padding:7px 10px;">ROCmFP4 4-bit</td><td style="border:1px solid currentColor; padding:7px 10px;">smallest / fastest decode</td></tr>
91
  </tbody>
92
  </table>
93
  </div>
94
 
95
- All three share **Q8 embeddings** and the code-weighted imatrix (see Β§04). The **COHERENT** build (β˜…) puts every body tensor on the **dual-scale** `q4_0_rocmfp4` kernel β€” lowest measured KL vs the Q8 source at ~the same decode speed β€” vs the STRIX builds' faster single-scale `q4_0_rocmfp4_fast` bulk. The DeltaNet-specific tensors (`ssm_conv1d`, `ssm_a`, norms, router) stay **F32** in all three; MoE experts + attention/SSM projections are 4-bit ROCmFP4.
96
 
97
  <div style="border:1px solid currentColor; padding:8px 13px; font-family:ui-monospace,'SF Mono',Consolas,monospace; font-size:12px; margin:12px 0; opacity:0.85;">
98
  <b>NOTE //</b> <b>Q8 embeddings</b> (not f16): the source is Q8_0, so Q8 matches its precision exactly β€” f16 would be fake-f16 bloat for zero gain (embeddings are a lookup, not a matmul).
@@ -105,7 +103,7 @@ Run from the folder holding the `.gguf` (the Qwen ChatML template is baked in
105
  ```bash
106
  env HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
107
  llama-server \
108
- -m Qwen3-Coder-Next-ROCmFP4-COHERENT-embQ8-headQ6.gguf \
109
  --alias coder-next \
110
  --host 0.0.0.0 \
111
  --port 8080 \
@@ -177,30 +175,18 @@ Qwen3-Coder-Next is an **agentic coder** β€” built to call tools, not narrate co
177
  <tr><td style="border:1px solid currentColor; padding:8px 11px; width:42%;">DECODE Β· short context</td><td style="border:1px solid currentColor; padding:8px 11px; font-weight:700;">~54 t/s (Vulkan / Ryzen AI Max+ 395)</td></tr>
178
  <tr><td style="border:1px solid currentColor; padding:8px 11px;">SPECULATIVE DECODE</td><td style="border:1px solid currentColor; padding:8px 11px; font-weight:700;">none (no MTP head)</td></tr>
179
  <tr><td style="border:1px solid currentColor; padding:8px 11px;">LONG CONTEXT</td><td style="border:1px solid currentColor; padding:8px 11px;">cheap β€” DeltaNet near-constant memory</td></tr>
180
- <tr><td style="border:1px solid currentColor; padding:8px 11px;">QUANTIZATION</td><td style="border:1px solid currentColor; padding:8px 11px;">COHERENT body + code-weighted imatrix (measured wins β€” below)</td></tr>
181
  </tbody>
182
  </table>
183
  </div>
184
 
185
- **Recommended build = COHERENT (we measured it).** On top of the imatrix + Q8 emb + Q6 head, we swept the body kernel and rank by **KL divergence vs the Q8 source** on held-out code (lower = more faithful). The **all-dual-scale body** (COHERENT) beats the fast-body STRIX build on every robust metric at ~the same decode speed:
186
 
187
- <div style="overflow:hidden; border-radius:0;">
188
- <table style="width:100%; border-collapse:collapse; border-radius:0; font-family:ui-monospace,'SF Mono',Consolas,monospace; font-size:12.5px;">
189
- <thead><tr>
190
- <th style="border:1px solid currentColor; padding:7px 10px; text-align:left; text-transform:uppercase; font-size:10px; letter-spacing:1px;">Build (imatrix + embQ8 + headQ6)</th>
191
- <th style="border:1px solid currentColor; padding:7px 10px; text-align:left; text-transform:uppercase; font-size:10px; letter-spacing:1px;">Body</th>
192
- <th style="border:1px solid currentColor; padding:7px 10px; text-align:left; text-transform:uppercase; font-size:10px; letter-spacing:1px;">Mean KLD vs Q8 ↓</th>
193
- <th style="border:1px solid currentColor; padding:7px 10px; text-align:left; text-transform:uppercase; font-size:10px; letter-spacing:1px;">Median KLD ↓</th>
194
- <th style="border:1px solid currentColor; padding:7px 10px; text-align:left; text-transform:uppercase; font-size:10px; letter-spacing:1px;">Top-token</th>
195
- </tr></thead>
196
- <tbody>
197
- <tr><td style="border:1px solid currentColor; padding:7px 10px;"><code>COHERENT</code> β˜…</td><td style="border:1px solid currentColor; padding:7px 10px;">all-dual</td><td style="border:1px solid currentColor; padding:7px 10px;"><b>0.03866</b></td><td style="border:1px solid currentColor; padding:7px 10px;"><b>0.00352</b></td><td style="border:1px solid currentColor; padding:7px 10px;"><b>92.98%</b></td></tr>
198
- <tr><td style="border:1px solid currentColor; padding:7px 10px;"><code>STRIX</code></td><td style="border:1px solid currentColor; padding:7px 10px;">fast</td><td style="border:1px solid currentColor; padding:7px 10px;">0.04252</td><td style="border:1px solid currentColor; padding:7px 10px;">0.00395</td><td style="border:1px solid currentColor; padding:7px 10px;">92.32%</td></tr>
199
- </tbody>
200
- </table>
201
- </div>
202
 
203
- COHERENT is lower on mean KLD (βˆ’9%), median KLD (βˆ’11%), and RMS Ξ”p (6.58% vs 6.86%), and higher on same-top-token (+0.65 pp) β€” every robust metric, same direction. So it's the default; the STRIX builds remain as marginally smaller/faster fallbacks.
 
 
204
 
205
  **Fast even without speculative decoding.** 3B active params + linear Gated-DeltaNet attention β†’ ~54 t/s short-context decode on a Ryzen AI Max+ 395 (Vulkan0), and cheap long context. No MTP needed.
206
 
@@ -242,13 +228,10 @@ So the imatrix **measurably improves quantization fidelity to the full model on
242
  # code-weighted imatrix on the Q8 (single pass; ratio = the real lever)
243
  llama-imatrix -m Qwen3-Coder-Next-Q8_0.gguf -f code-weighted-calib.txt -o coder-next.imatrix -c 512 -ngl 999
244
 
245
- # quant -> ROCmFP4 with the imatrix (Q8 embeddings). headQ6 adds --output-tensor-type q6_K
246
- llama-quantize --allow-requantize --token-embedding-type q8_0 --imatrix coder-next.imatrix \
247
- Qwen3-Coder-Next-Q8_0.gguf Qwen3-Coder-Next-ROCmFP4-STRIX-embQ8-imatrix.gguf Q4_0_ROCMFP4_STRIX
248
-
249
- # RECOMMENDED: the COHERENT all-dual-scale body + Q6 head (the β˜… file) β€” lowest KL (Β§04)
250
  llama-quantize --allow-requantize --token-embedding-type q8_0 --output-tensor-type q6_K --imatrix coder-next.imatrix \
251
- Qwen3-Coder-Next-Q8_0.gguf Qwen3-Coder-Next-ROCmFP4-COHERENT-embQ8-headQ6.gguf Q4_0_ROCMFP4_COHERENT
252
  ```
253
 
254
  > Experimental research build for AMD Strix Halo β€” hardware/driver/prompt-sensitive, may not reproduce elsewhere. Not native FP4 tensor-core execution.
 
85
  <th style="border:1px solid currentColor; padding:7px 10px; text-align:left; text-transform:uppercase; font-size:10px; letter-spacing:1px;">Pick if</th>
86
  </tr></thead>
87
  <tbody>
88
+ <tr><td style="border:1px solid currentColor; padding:7px 10px;"><code>…-STRIX-embQ8-imatrix-headQ6.gguf</code> β˜…</td><td style="border:1px solid currentColor; padding:7px 10px;">Q6_K</td><td style="border:1px solid currentColor; padding:7px 10px;"><b>the one build</b> β€” best speed/quality balance: Q8 embeddings + Q6 output head on the fast single-scale body</td></tr>
 
 
89
  </tbody>
90
  </table>
91
  </div>
92
 
93
+ One file β€” the **best speed/quality balance** in ROCmFP4 for Strix Halo. It keeps the two quality levers that are actually *felt* β€” **Q8 token embeddings** (matching the Q8 source exactly) and a **Q6_K output head** β€” on the fast single-scale `q4_0_rocmfp4_fast` body + a code-weighted imatrix. Not the most faithful possible (see the fidelity link in Β§04) β€” it's the point where speed and quality meet best. The DeltaNet-specific tensors (`ssm_conv1d`, `ssm_a`, norms, router) stay **F32**; MoE experts + attention/SSM projections are 4-bit ROCmFP4.
94
 
95
  <div style="border:1px solid currentColor; padding:8px 13px; font-family:ui-monospace,'SF Mono',Consolas,monospace; font-size:12px; margin:12px 0; opacity:0.85;">
96
  <b>NOTE //</b> <b>Q8 embeddings</b> (not f16): the source is Q8_0, so Q8 matches its precision exactly β€” f16 would be fake-f16 bloat for zero gain (embeddings are a lookup, not a matmul).
 
103
  ```bash
104
  env HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
105
  llama-server \
106
+ -m Qwen3-Coder-Next-ROCmFP4-STRIX-embQ8-imatrix-headQ6.gguf \
107
  --alias coder-next \
108
  --host 0.0.0.0 \
109
  --port 8080 \
 
175
  <tr><td style="border:1px solid currentColor; padding:8px 11px; width:42%;">DECODE Β· short context</td><td style="border:1px solid currentColor; padding:8px 11px; font-weight:700;">~54 t/s (Vulkan / Ryzen AI Max+ 395)</td></tr>
176
  <tr><td style="border:1px solid currentColor; padding:8px 11px;">SPECULATIVE DECODE</td><td style="border:1px solid currentColor; padding:8px 11px; font-weight:700;">none (no MTP head)</td></tr>
177
  <tr><td style="border:1px solid currentColor; padding:8px 11px;">LONG CONTEXT</td><td style="border:1px solid currentColor; padding:8px 11px;">cheap β€” DeltaNet near-constant memory</td></tr>
178
+ <tr><td style="border:1px solid currentColor; padding:8px 11px;">QUANTIZATION</td><td style="border:1px solid currentColor; padding:8px 11px;">fast single-scale body + Q8 emb + Q6 head + code-weighted imatrix (measured win β€” below)</td></tr>
179
  </tbody>
180
  </table>
181
  </div>
182
 
183
+ **This is the best speed/quality balance in ROCmFP4 β€” by design, not the absolute fastest.** On top of the imatrix + Q8 emb + Q6 head, we swept the body kernel against the Q8 source by **KL divergence** (the right fidelity metric). An all-dual-scale body did edge the fast single-scale body on KL, but the gain sat inside the measurement noise while costing decode speed β€” so the **fast single-scale body + Q8 embeddings + Q6 head** is the right point, and the one file we ship.
184
 
185
+ This mirrors the fuller sweep on our [**Qwen3.6-27B sibling**](https://huggingface.co/plunderstruck/Qwen3.6-27B-MTP-ROCmFP4-GGUF), where every higher-precision body lever (all-dual-scale, selective Q5/Q6 bumps) bought a KL improvement inside the noise at a real speed cost β€” and where copying an entire dynamic-quant high-precision allocation onto ROCmFP4 *still* couldn't match a true dynamic K-quant, because FP4 is intrinsically less faithful than Q4_K's 4-bit. The same format limit applies here: within ROCmFP4, fast body + Q8 emb + Q6 head is the optimal balance; for maximum fidelity reach for a dynamic K-quant of the base (box below). *(Directional internal measurements β€” KL vs Q8 on held-out code; reproduce before citing.)*
 
 
 
 
 
 
 
 
 
 
 
 
 
 
186
 
187
+ <div style="border:1px solid currentColor; padding:8px 13px; font-family:ui-monospace,'SF Mono',Consolas,monospace; font-size:12px; margin:12px 0; opacity:0.9;">
188
+ <b>WANT MAXIMUM FIDELITY INSTEAD OF SPEED?</b> Grab a <b>Q6_K / Q8 dynamic GGUF of the base</b> from <a href="https://huggingface.co/Qwen/Qwen3-Coder-Next"><b>Qwen/Qwen3-Coder-Next</b></a> β€” higher-bit GGUFs run on this same fork. We optimize for throughput in ROCmFP4; if you want the last bit of fidelity over speed, that's the one to grab.
189
+ </div>
190
 
191
  **Fast even without speculative decoding.** 3B active params + linear Gated-DeltaNet attention β†’ ~54 t/s short-context decode on a Ryzen AI Max+ 395 (Vulkan0), and cheap long context. No MTP needed.
192
 
 
228
  # code-weighted imatrix on the Q8 (single pass; ratio = the real lever)
229
  llama-imatrix -m Qwen3-Coder-Next-Q8_0.gguf -f code-weighted-calib.txt -o coder-next.imatrix -c 512 -ngl 999
230
 
231
+ # quant -> ROCmFP4 with the imatrix (Q8 embeddings) + Q6 output head β€” the β˜… file (Β§01)
232
+ # fast single-scale body; --output-tensor-type q6_K raises the output head to Q6_K
 
 
 
233
  llama-quantize --allow-requantize --token-embedding-type q8_0 --output-tensor-type q6_K --imatrix coder-next.imatrix \
234
+ Qwen3-Coder-Next-Q8_0.gguf Qwen3-Coder-Next-ROCmFP4-STRIX-embQ8-imatrix-headQ6.gguf Q4_0_ROCMFP4_STRIX
235
  ```
236
 
237
  > Experimental research build for AMD Strix Halo β€” hardware/driver/prompt-sensitive, may not reproduce elsewhere. Not native FP4 tensor-core execution.