philtheriver commited on
Commit
3360642
·
verified ·
1 Parent(s): 67f2d2b

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +54 -87
README.md CHANGED
@@ -22,122 +22,89 @@ tags:
22
  - vision
23
  ---
24
 
25
- # Qwopus3.6-27B-Coder-MTP — ROCmFPX (`Q6_0_ROCMFPX_AGENT`)
26
 
27
- A **ROCmFPX** quantization of [`Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF`](https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF) built for **AMD Strix Halo** (Ryzen AI MAX, `gfx1151`, 128 GB unified). 27B dense, Apache-2.0, MTP head intact (`draft=2`), agent/tool-use tuned, multimodal (Qwen3-VL).
28
 
29
- > ⚠️ **Requires the [ROCmFPX llama.cpp fork](https://github.com/charlie12345/ROCmFPX), built at commit [`553504b`](https://github.com/charlie12345/ROCmFPX/commit/553504bd5d16fe1374e9b1508a1f395c9a35d746).**
30
- > `ROCmFPX` is a family of **custom AMD quant types**; this build uses **enum IDs 110–115**. Those IDs are **not upstream-stable** — a newer fork revision could renumber them and silently mis-decode these weights, so **build from the pinned commit above**. The file will **NOT load** in stock `llama.cpp`, LM Studio, Ollama, or any other runtime — only the fork's `llama-quantize`/`llama-server` understand it.
31
 
32
- > ℹ️ **The Hugging Face precision badge is wrong.** HF's GGUF parser can't read ROCmFPX block layouts and will mislabel this (e.g. "F16"). Pick the file by **filename**; the real footprint is **~6.5 bpw** (+ f16 token embeddings + Q6_K output head).
 
 
 
 
 
 
33
 
34
- ---
35
 
36
- ## What this is
37
 
38
- `Q6_0_ROCMFPX_AGENT` — the **agent-routed** ROCmFPX Q6 recipe: ~6.5 bpw weights with attention / FFN routing kept at higher precision for tool-call and JSON coherence, **f16 token embeddings** and a **Q6_K output head**, quantized with a **code-calibrated importance matrix**. More faithful than the published FP4 builds, while staying small enough to keep MTP speculative decoding fast on a 128 GB box.
 
 
 
 
 
39
 
40
- ## Recipe
41
 
42
- | Lever | Value |
43
- |-------|-------|
44
- | Base preset | `Q6_0_ROCMFPX_AGENT` (ROCmFPX enum 114 — *agent/tool-call coherent ROCmFPx Q6 routing*) |
45
- | Token embeddings | `f16` (`--token-embedding-type f16`) |
46
- | Output head | `Q6_K` (`--output-tensor-type q6_K`) |
47
- | Importance matrix | code-calibrated (~1.3 MB corpus: C#/.NET, Python, JavaScript, SQL) |
48
- | Source | `Qwopus3.6-27B-Coder-MTP-BF16.gguf` (54.66 GB BF16, MTP head spliced in) |
49
- | Output size | ~25 GiB |
50
 
51
- Exact build:
52
  ```bash
53
- llama-quantize \
54
- --imatrix qwopus-coder.imatrix \
55
- --token-embedding-type f16 --output-tensor-type q6_K \
56
- Qwopus3.6-27B-Coder-MTP-BF16.gguf \
57
- Qwopus3.6-27B-Coder-MTP-STRIX-embF16-headQ6-Q6_0_ROCMFPX_AGENT.gguf \
58
- Q6_0_ROCMFPX_AGENT
59
- ```
60
-
61
- ## Quant tiers (see the **Files** tab)
62
-
63
- Every tier carries the MTP head (`draft=2`) and pairs with the bundled `mmproj/mmproj-F32.gguf` (vision).
64
- f16 token embeddings throughout; `_AGENT` presets route attn/FFN at higher precision for tool-call /
65
- JSON coherence. Filenames are prefixed `Qwopus3.6-27B-Coder-MTP-STRIX-`.
66
-
67
- | File suffix | Preset | Enum | ~bpw | Size | Role |
68
- |-------------|--------|------|------|------|------|
69
- | `embF16-headQ6-Q6_0_ROCMFPX_AGENT.gguf` | `Q6_0_ROCMFPX_AGENT` | 114 | ~6.5 | 26 GB | **flagship** — f16 emb + Q6_K head + code imatrix |
70
- | `embF16-Q8_0_ROCMFPX_AGENT.gguf` | `Q8_0_ROCMFPX_AGENT` | 115 | ~8.25 | 30 GB | highest-fidelity agent |
71
- | `embF16-Q8_0_ROCMFPX.gguf` | `Q8_0_ROCMFPX` | 111 | 8.25 | 29 GB | highest-fidelity (straight) |
72
- | `embF16-Q6_0_ROCMFPX.gguf` | `Q6_0_ROCMFPX` | 110 | 6.50 | 24 GB | balanced (straight) |
73
- | `embF16-Q3_0_ROCMFPX_AGENT.gguf` | `Q3_0_ROCMFPX_AGENT` | 113 | ~3.5 | 21 GB | smallest agent |
74
- | `embF16-Q3_0_ROCMFPX.gguf` | `Q3_0_ROCMFPX` | 112 | 3.50 | 17 GB | smallest |
75
- | `embF16-Q4_0_ROCMFP4.gguf` | `Q4_0_ROCMFP4` | 100 | ~5.7¹ | 19 GB | fastest decode (4-bit body, ~13 tok/s raw) |
76
-
77
- > Tiers beyond the flagship are uploaded as they finish building — the **Files** tab is the source of truth for what's live.
78
- > ¹ `Q4_0_ROCMFP4` is a true 4-bit body; the ~5.7 *effective* bpw is inflated by the f16 token embeddings (a lookup — it doesn't slow the fast 4-bit decode).
79
-
80
- ## Running (ROCmFPX fork required)
81
-
82
- ```bash
83
- # 1) build the fork at the pinned commit (ROCm/HIP toolchain, gfx1151)
84
  git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
85
- git checkout 553504b # enum IDs 110-115 match this revision
86
- JOBS=16 scripts/build-strix-rocmfp4-mtp.sh # -> build-strix-rocmfp4/bin/
87
 
88
- # 2) serve with MTP speculative decoding + vision
89
- HSA_OVERRIDE_GFX_VERSION=11.5.1 \
90
- build-strix-rocmfp4/bin/llama-server \
91
  -m Qwopus3.6-27B-Coder-MTP-STRIX-embF16-headQ6-Q6_0_ROCMFPX_AGENT.gguf \
92
- --alias qwopus-coder \
93
  -dev ROCm0 -ngl 999 -fa on -c 32768 \
94
  --spec-type draft-mtp --spec-draft-ngl all --spec-draft-n-max 2 \
95
- --jinja --mmproj mmproj/mmproj-F32.gguf \
96
- --host 0.0.0.0 --port 8080
97
  ```
98
 
99
- - **MTP:** the head is built in (`draft=2`). Run with `--spec-type draft-mtp`; `--spec-draft-n-max 2` was optimal in testing.
100
- - **Vision (Qwen3-VL):** bundled `mmproj/mmproj-F32.gguf`; pass `--mmproj` (add `--image-min-tokens 1024` for high-detail images).
101
- - **Tool calls:** point your client at the **`qwen3_coder`** tool-call parser, or the model narrates code instead of emitting structured calls.
102
- - The chat template is embedded — `--jinja` reads it.
103
-
104
- ## Verification
105
 
106
- **Functional smoke** (`check-rocmfpx-agentic-smoke.sh`, ROCm0) — all gates pass:
 
107
 
108
- | Chat | Coding | Strict JSON | Tool-call JSON | Coherency |
109
- |:----:|:------:|:-----------:|:--------------:|:---------:|
110
- | ✅ | ✅ | ✅ | ✅ | ✅¹ |
111
 
112
- ¹ The harness greps for the literal token `tools`; the model writes "**Tool** preservation", so it reports a false negative — the 3-bullet answer is correct and coherent.
 
 
 
 
 
 
 
 
113
 
114
- | Metric | Value |
115
- |--------|-------|
116
- | MTP decode (`draft-mtp`, `--spec-draft-n-max 2`) | ✅ active — ~18.6 tok/s decode on gfx1151 |
117
- | PPL — code corpus (~110 chunks @ ctx 512) | flagship **2.922** vs stock `Q6_K` **2.917** → **Δ +0.17%** (both ±0.04 — within error) |
118
 
119
- > Flagship and stock `Q6_K` are within error of each other — the recipe preserves Q6_K-level fidelity at the ROCmFPX format. (Source BF16 PPL over the full 675-chunk corpus was 3.2374; the slice above is easier, hence the lower absolute numbers — only the flagship-vs-Q6_K delta is meaningful.)
120
- > All numbers are hardware/driver/prompt-sensitive — reproduce before relying.
121
 
122
- ## Performance — Strix Halo (gfx1151)
 
 
 
123
 
124
- `llama-bench`, fork build `553504b`, `-fa 1`. Prompt-processing throughput (t/s) vs stock `Q6_K` — same engine, so the gap is the quant **format**:
125
 
126
- | Context | stock `Q6_K` | flagship | Δ |
127
  |---|---|---|---|
128
  | pp512 | 204 | 269 | **+32%** |
129
- | pp2048 | 193 | 254 | **+31%** |
130
- | pp10k | 181 | 236 | **+30%** |
131
- | pp16k | 174 | 225 | **+29%** |
132
- | pp32k | 158 | 199 | **+26%** |
133
  | pp64k | 134 | 163 | **+22%** |
134
 
135
- - **Prompt processing ~30% faster than `Q6_K`** — the gfx1151-tuned ROCmFPX kernels win the compute-bound prefill. The edge is largest at short context and **narrows toward +22% at 64k**, as O(n²) attention starts to dominate the prefill.
136
- - **Decode is bandwidth-bound** — flagship ≈ `Q6_K` raw (~9 tok/s), format-independent; **MTP (`--spec-type draft-mtp`) ~doubles it** in serving (~18 tok/s). `Q4_0_ROCMFP4` is the decode king (~13 tok/s) if you'll trade quality.
137
- - Measured on this build. Single-rep, so treat the absolutes as ±a few %.
138
 
139
  ## Credits & license
140
 
141
- - **License:** Apache-2.0 (inherited).
142
- - **Chain:** Jackrong + Kyle Hessling (fine-tune) → Qwen3.6-27B (base) → [charlie12345 / ROCmFPX](https://github.com/charlie12345/ROCmFPX) (quant fork). ROCmFPX quantization by this repo's author.
143
- - Experimental community release; no full safety eval. The base card's 67% SWE-bench figure is a no-thinking Q5_K_M number — treat these FPX numbers as independent.
 
22
  - vision
23
  ---
24
 
25
+ # Qwopus3.6-27B-Coder · ROCmFPX
26
 
27
+ ### Stock `Q6_K` quality, ~30% faster prompt-processing on AMD Strix Halo (`gfx1151`)
28
 
29
+ ROCmFPX 4→8-bit quants of [`Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF`](https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF) — 27B, MTP speculative decoding + Qwen3-VL vision, agent/tool-use tuned.
 
30
 
31
+ | | |
32
+ |---|---|
33
+ | **Quality** | ≈ stock `Q6_K` — PPL **+0.17%** (within error) |
34
+ | **Prompt processing** | **+32%** vs Q6_K (short ctx) → +22% at 64k |
35
+ | **Decode** | ~18 tok/s with MTP (~9 raw) |
36
+ | **Vision** | Qwen3-VL — bundled `mmproj/` |
37
+ | **Context** | up to 128k |
38
 
39
+ > ⚠️ **Requires the [ROCmFPX fork](https://github.com/charlie12345/ROCmFPX) @ [`553504b`](https://github.com/charlie12345/ROCmFPX/commit/553504bd5d16fe1374e9b1508a1f395c9a35d746)** — custom AMD quant types (enum IDs 110–115), not upstream-stable. **Won't load** in stock llama.cpp / LM Studio / Ollama. HF's precision badge is wrong — **pick the file by name**.
40
 
41
+ ## Pick a tier
42
 
43
+ | File suffix | Size | Best for |
44
+ |---|---|---|
45
+ | `…embF16-headQ6-Q6_0_ROCMFPX_AGENT.gguf` ★ | 26 GB | **best overall** — the flagship |
46
+ | `…embF16-Q8_0_ROCMFPX.gguf` | 29 GB | maximum fidelity |
47
+ | `…embF16-Q4_0_ROCMFP4.gguf` | 19 GB | fastest decode (4-bit) |
48
+ | `…embF16-Q3_0_ROCMFPX.gguf` | 17 GB | smallest |
49
 
50
+ Agent-routed `_AGENT` tiers + the full enum/bpw table are in the details below and the **Files** tab. All filenames prefixed `Qwopus3.6-27B-Coder-MTP-STRIX-`.
51
 
52
+ ## Quick start
 
 
 
 
 
 
 
53
 
 
54
  ```bash
55
+ # build the fork once (pinned commit so the enum IDs match)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
  git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
57
+ git checkout 553504b && JOBS=16 scripts/build-strix-rocmfp4-mtp.sh
 
58
 
59
+ # serve the flagship — MTP + vision
60
+ HSA_OVERRIDE_GFX_VERSION=11.5.1 build-strix-rocmfp4/bin/llama-server \
 
61
  -m Qwopus3.6-27B-Coder-MTP-STRIX-embF16-headQ6-Q6_0_ROCMFPX_AGENT.gguf \
 
62
  -dev ROCm0 -ngl 999 -fa on -c 32768 \
63
  --spec-type draft-mtp --spec-draft-ngl all --spec-draft-n-max 2 \
64
+ --jinja --mmproj mmproj/mmproj-F32.gguf --host 0.0.0.0 --port 8080
 
65
  ```
66
 
67
+ Tool calls: point your client at the **`qwen3_coder`** parser, or the model narrates code instead of emitting structured calls.
 
 
 
 
 
68
 
69
+ <details>
70
+ <summary><b>All tiers · recipe · benchmarks</b></summary>
71
 
72
+ ### All tiers
 
 
73
 
74
+ | File suffix | Preset | Enum | Size | Role |
75
+ |---|---|---|---|---|
76
+ | `embF16-headQ6-Q6_0_ROCMFPX_AGENT.gguf` | `Q6_0_ROCMFPX_AGENT` | 114 | 26 GB | flagship — f16 emb + Q6_K head + imatrix |
77
+ | `embF16-Q8_0_ROCMFPX_AGENT.gguf` | `Q8_0_ROCMFPX_AGENT` | 115 | 30 GB | highest-fidelity agent |
78
+ | `embF16-Q8_0_ROCMFPX.gguf` | `Q8_0_ROCMFPX` | 111 | 29 GB | highest fidelity |
79
+ | `embF16-Q6_0_ROCMFPX.gguf` | `Q6_0_ROCMFPX` | 110 | 24 GB | balanced |
80
+ | `embF16-Q3_0_ROCMFPX_AGENT.gguf` | `Q3_0_ROCMFPX_AGENT` | 113 | 21 GB | smallest agent |
81
+ | `embF16-Q3_0_ROCMFPX.gguf` | `Q3_0_ROCMFPX` | 112 | 17 GB | smallest |
82
+ | `embF16-Q4_0_ROCMFP4.gguf` | `Q4_0_ROCMFP4` | 100 | 19 GB | fastest decode (4-bit body) |
83
 
84
+ f16 token embeddings throughout; `_AGENT` presets keep attention/FFN routing at higher precision for tool-call/JSON coherence. (HF labels `Q4`/`Q8` but not `Q6`/`Q3` — the latter aren't standard llama.cpp quant names.)
 
 
 
85
 
86
+ ### Verification (Strix Halo gfx1151)
 
87
 
88
+ | Metric | Value |
89
+ |---|---|
90
+ | Functional smoke (`check-rocmfpx-agentic-smoke.sh`) | chat / coding / JSON / tool-call ✅ |
91
+ | PPL vs `Q6_K` (code corpus) | flagship **2.922** vs Q6_K **2.917** → **+0.17%** (within ±0.04) |
92
 
93
+ ### Performance — prompt-processing throughput (t/s) vs `Q6_K`
94
 
95
+ | Context | `Q6_K` | flagship | Δ |
96
  |---|---|---|---|
97
  | pp512 | 204 | 269 | **+32%** |
98
+ | pp2048 | 193 | 254 | +31% |
99
+ | pp10k | 181 | 236 | +30% |
100
+ | pp16k | 174 | 225 | +29% |
101
+ | pp32k | 158 | 199 | +26% |
102
  | pp64k | 134 | 163 | **+22%** |
103
 
104
+ The gfx1151-tuned kernels win the compute-bound prefill; the edge is largest at short context and narrows toward +22% at 64k as O(n²) attention takes over. Decode is bandwidth-bound (≈ Q6_K raw), and **MTP (`--spec-type draft-mtp`) ~doubles it** in serving. `Q4_0_ROCMFP4` is the decode king (~13 tok/s raw). *Single-rep `llama-bench`; treat absolutes as ±a few %.*
105
+
106
+ </details>
107
 
108
  ## Credits & license
109
 
110
+ Apache-2.0 (inherited). Jackrong + Kyle Hessling (fine-tune) → Qwen3.6-27B (base) → [charlie12345 / ROCmFPX](https://github.com/charlie12345/ROCmFPX) (quant fork). ROCmFPX quantization by this repo's author.