philtheriver commited on
Commit
d5f0355
·
verified ·
1 Parent(s): de02985

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +113 -0
README.md ADDED
@@ -0,0 +1,113 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model:
4
+ - Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF
5
+ base_model_relation: quantized
6
+ pipeline_tag: text-generation
7
+ library_name: llama.cpp
8
+ language:
9
+ - en
10
+ tags:
11
+ - gguf
12
+ - rocmfpx
13
+ - rocm
14
+ - amd
15
+ - strix-halo
16
+ - gfx1151
17
+ - mtp
18
+ - speculative-decoding
19
+ - agent
20
+ - tool-use
21
+ - code
22
+ - qwen3
23
+ - vision
24
+ ---
25
+
26
+ # Qwopus3.6-27B-Coder-MTP — ROCmFPX (`Q6_0_ROCMFPX_AGENT`)
27
+
28
+ A **ROCmFPX** quantization of [`Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF`](https://huggingface.co/Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF) built for **AMD Strix Halo** (Ryzen AI MAX, `gfx1151`, 128 GB unified). 27B dense, Apache-2.0, MTP head intact (`draft=2`), agent/tool-use tuned, multimodal (Qwen3-VL).
29
+
30
+ > ⚠️ **Requires the [ROCmFPX llama.cpp fork](https://github.com/charlie12345/ROCmFPX).**
31
+ > `ROCmFPX` is a family of **custom AMD quant types** (enum IDs 110–115). This file will **NOT load** in stock `llama.cpp`, LM Studio, Ollama, or any other runtime — only the fork's `llama-quantize`/`llama-server` understand it. Build the fork first (see *Running* below).
32
+
33
+ > ℹ️ **The Hugging Face precision badge is wrong.** HF's GGUF parser can't read ROCmFPX block layouts and will mislabel this (e.g. "F16"). Pick the file by **filename**; the real footprint is **~6.5 bpw** (+ f16 token embeddings + Q6_K output head).
34
+
35
+ ---
36
+
37
+ ## What this is
38
+
39
+ `Q6_0_ROCMFPX_AGENT` — the **agent-routed** ROCmFPX Q6 recipe: ~6.5 bpw weights with attention / FFN routing kept at higher precision for tool-call and JSON coherence, **f16 token embeddings** and a **Q6_K output head**, quantized with a **code-calibrated importance matrix**. More faithful than the published FP4 builds, while staying small enough to keep MTP speculative decoding fast on a 128 GB box.
40
+
41
+ ## Recipe
42
+
43
+ | Lever | Value |
44
+ |-------|-------|
45
+ | Base preset | `Q6_0_ROCMFPX_AGENT` (ROCmFPX enum 114 — *agent/tool-call coherent ROCmFPx Q6 routing*) |
46
+ | Token embeddings | `f16` (`--token-embedding-type f16`) |
47
+ | Output head | `Q6_K` (`--output-tensor-type q6_K`) |
48
+ | Importance matrix | code-calibrated (~1.3 MB corpus: C#/.NET, Python, JavaScript, SQL) |
49
+ | Source | `Qwopus3.6-27B-Coder-MTP-BF16.gguf` (54.66 GB BF16, MTP head spliced in) |
50
+ | Output size | ~25 GiB |
51
+
52
+ Exact build:
53
+ ```bash
54
+ llama-quantize \
55
+ --imatrix qwopus-coder.imatrix \
56
+ --token-embedding-type f16 --output-tensor-type q6_K \
57
+ Qwopus3.6-27B-Coder-MTP-BF16.gguf \
58
+ Qwopus3.6-27B-Coder-MTP-STRIX-embF16-headQ6-Q6_0_ROCMFPX_AGENT.gguf \
59
+ Q6_0_ROCMFPX_AGENT
60
+ ```
61
+
62
+ ## ROCmFPX tier reference
63
+
64
+ | Preset | Enum | ~bpw | Notes |
65
+ |--------|------|------|-------|
66
+ | `Q3_0_ROCMFPX` | 112 | 3.50 | experimental, ROCm/Vulkan staging |
67
+ | `Q3_0_ROCMFPX_AGENT` | 113 | ~3.5 | agent/tool-call routing |
68
+ | `Q6_0_ROCMFPX` | 110 | 6.50 | experimental |
69
+ | **`Q6_0_ROCMFPX_AGENT`** | **114** | **~6.5** | **this build** — agent/tool-call routing |
70
+ | `Q8_0_ROCMFPX` | 111 | 8.25 | experimental |
71
+ | `Q8_0_ROCMFPX_AGENT` | 115 | ~8.25 | agent/tool-call routing |
72
+
73
+ ## Running (ROCmFPX fork required)
74
+
75
+ ```bash
76
+ # 1) build the fork once (ROCm/HIP toolchain, gfx1151)
77
+ git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
78
+ JOBS=16 scripts/build-strix-rocmfp4-mtp.sh # -> build-strix-rocmfp4/bin/
79
+
80
+ # 2) serve with MTP speculative decoding + vision
81
+ HSA_OVERRIDE_GFX_VERSION=11.5.1 \
82
+ build-strix-rocmfp4/bin/llama-server \
83
+ -m Qwopus3.6-27B-Coder-MTP-STRIX-embF16-headQ6-Q6_0_ROCMFPX_AGENT.gguf \
84
+ --alias qwopus-coder \
85
+ -dev ROCm0 -ngl 999 -fa on -c 32768 \
86
+ --spec-type draft-mtp --spec-draft-ngl all --spec-draft-n-max 2 \
87
+ --jinja --mmproj mmproj-F32.gguf \
88
+ --host 0.0.0.0 --port 8080
89
+ ```
90
+
91
+ - **MTP:** the head is built in (`draft=2`). Run with `--spec-type draft-mtp`; `--spec-draft-n-max 2` was optimal in testing.
92
+ - **Vision (Qwen3-VL):** bundled `mmproj-F32.gguf`; pass `--mmproj` (add `--image-min-tokens 1024` for high-detail images).
93
+ - **Tool calls:** point your client at the **`qwen3_coder`** tool-call parser, or the model narrates code instead of emitting structured calls.
94
+ - The chat template is embedded — `--jinja` reads it.
95
+
96
+ ## Verification
97
+
98
+ <!-- TODO: fill from scripts/sweep-rocmfpx-perplexity.sh + check-rocmfpx-mtp-smoke.sh -->
99
+ | Metric | Value |
100
+ |--------|-------|
101
+ | PPL (code corpus) vs `Q6_K` | _TODO_ |
102
+ | KL-divergence vs `Q6_K` | _TODO_ |
103
+ | MTP draft acceptance (`n-max 2`) | _TODO_ |
104
+ | Tool-call JSON gate | _TODO (pass/fail)_ |
105
+
106
+ > Reference: source BF16 perplexity on the calibration corpus measured during imatrix = **3.2374 ± 0.018**.
107
+ > All numbers are hardware/driver/prompt-sensitive — reproduce before relying.
108
+
109
+ ## Credits & license
110
+
111
+ - **License:** Apache-2.0 (inherited).
112
+ - **Chain:** Jackrong + Kyle Hessling (fine-tune) → Qwen3.6-27B (base) → [charlie12345 / ROCmFPX](https://github.com/charlie12345/ROCmFPX) (quant fork). ROCmFPX quantization by this repo's author.
113
+ - Experimental community release; no full safety eval. The base card's 67% SWE-bench figure is a no-thinking Q5_K_M number — treat these FPX numbers as independent.