kingjones777/Ternary-Bonsai-2-27B-ROCm-gfx1151

Ternary-Bonsai-2 27B on ROCm (gfx1151) — measurements and the MTP patch

This repository contains no weights. It documents running prism-ml/Ternary-Bonsai-2-27B-gguf on ROCm, which its model card does not cover — the backends listed there are CUDA, Metal, CPU.

It does run on ROCm — well — but which file you pick decides whether it is fast or three times slower, and enabling MTP needs an 18-line source fix. Both are documented here with the numbers.

---

⛔ Use PQ2_0, not the smaller PTQ1_0 — on ROCm the bigger file is faster

ggml/src/ggml-cuda/mmq.cuh wraps the ternary type in a HIP exclusion at every switch (~lines 66, 404, 573, 749):

case GGML_TYPE_PQ2_0:          // always compiled
#if !defined(GGML_USE_HIP)
case GGML_TYPE_PTQ1_0:         // excluded on ROCm
#endif

So on ROCm only PQ2_0 gets the MMQ kernel; PTQ1_0 falls back to dequantise→F16.

packing	size	backend	pp512	tg128
PQ2_0	6.70 GiB	ROCm	216.78	23.78
PTQ1_0	5.53 GiB	ROCm	277.97	20.12
PTQ1_0	5.53 GiB	Vulkan	210.52	3.83
PQ2_0	—	Vulkan	no PQ2_0 shader exists	—

llama-bench, quiet box, 3 reps. Both packings hold the same ternary weights — 1.75 vs 2.13 bpw is storage layout only, so quality is identical and the choice is purely speed.

• ⛔ Vulkan is not viable for these kernels: 3.83 t/s on AMD RADV and 1.03 t/s on an Intel Arc 140T, against 23.78 on ROCm. Only ptq1_0.glsl / dequant_ptq1_0.comp exist.
• ⛔ The upstream card states PQ2_0 is "faster at prompt processing everywhere". On gfx1151 it is inverted: −22% prefill, +18% decode.

MTP: +66% decode, and the 18-line fix it needs

Using the grafted head from decent-jawfish/bonsai-2-27b-mtp (PQ2_0 + the Qwen3.8-27B MTP head, 866 tensors / blocks 0–64 / nextn_predict_layers = 1).

Unpatched, --spec-type draft-mtp fails at context creation:

Hadamard-latent table 'token_embd.weight' is read without the inverse transform

The MTP draft graph builds its own token-embedding lookup and skips the inverse that llm_graph_context::build_inp_embd() applies on the main path. Add, immediately after tok_embd = ggml_get_rows(ctx0, tok_embd_w, inp->tokens); in src/models/qwen35.cpp:

if (hadamard_inverses) {
    const auto it = hadamard_inverses->find(tok_embd_w);
    if (it != hadamard_inverses->end()) {
        tok_embd = llama_mul_mat_hadamard(ctx0, tok_embd, it->second.rot);
        if (it->second.signs) {
            tok_embd = ggml_mul(ctx0, tok_embd, it->second.signs);
        }
    }
}

plus #include "llama-impl.h". Order matters: rotation first, sign flip second — the reverse of the folded forward path. Mirror the in-tree reference rather than guessing. The full diff is in this repo as qwen35-mtp-hadamard-inverse.patch.

Results (3 distinct generation tasks + 3 distinct rewrite sources, fresh server per arm)

arm	generation	verbatim rewrite
PQ2_0, no speculation	20.49	23.52
PQ2_0 + ngram-mod	20.25	45.06
MTP file, no speculation	19.51	20.56
MTP + draft-mtp --spec-draft-n-max 1	31.07	37.57
MTP + draft-mtp --spec-draft-n-max 2	33.93	38.55
MTP + draft-mtp --spec-draft-n-max 3	30.75	41.37

All arms produced byte-exact rewrites and passed tool calling 3/3.

• MTP is the only mode that accelerates free generation. ngram-mod is excellent on quoting/rewrite traffic (+92%) but −1% on generation, because it can only draft from text already in the context. This model defaults to xhigh reasoning, so generation dominates.
• ✅ MTP fits at the full -c 131072 — no context reduction (this model is ~75% linear attention, so its KV is small).
• ✅ MTP and --mmproj coexist: tools 3/3, vision 3/3, zero GGML_ABORT.
• ⚠️ Honest caveat: generation measurement on this model carries ±10% run-to-run noise (a trailing baseline re-measure read 23.62 / 19.82 / 22.79 on identical work). n-max 1 and 2 are therefore not statistically separated; the low-noise rewrite column favours n3. All three are a large win over no speculation.

DFlash2 also runs on ROCm — but MTP is the better pick here

ProCreations/Ternary-Bonsai-2-27B-DFlash2 ships a 2.06 GB Q8_0 draft head (arch dflash, 5 blocks, target_layers [6, 20, 34, 48, 62]) that runs against the unchanged PQ2_0 target. Its release runtime targets CUDA 13.3 / SM120 Blackwell; its source tarball builds clean for gfx1151 with the same HIP flags used above — no changes needed.

⛔ The stock PrismML prism branch cannot load this draft:

done_getting_tensors: wrong number of tensors; expected 81, got 58

The r2-dflash head's layout differs from the older in-branch dflash implementation, so you must build runtime/prism-dflash2-source.tar.gz from that repo (swap -DGGML_CUDA=ON for the HIP flags above). That tree also already carries the qwen35 Hadamard-inverse fix described earlier.

Head-to-head, same binary, same harness, same session

Every arm ran on the DFlash2 runtime so the comparison isolates the speculation method, not the build. Control is the base PQ2_0 with no speculation, measured in the same session.

arm	generation	vs control	verbatim rewrite	vs control
control, no speculation	23.76	—	23.54	—
draft-mtp --spec-draft-n-max 2	39.04	1.64x	46.12	1.96x
draft-dflash --spec-draft-n-max 2	39.87	1.68x	42.97	1.83x
draft-dflash --spec-draft-n-max 3 (their default)	35.61	1.50x	43.38	1.84x
draft-dflash --spec-draft-n-max 5	33.83	1.42x	44.48	1.89x

All arms produced byte-exact rewrites, 3/3.

Read it this way: DFlash2's best arm is +2.1% on generation over MTP, which is inside the ±10% generation noise on this model and therefore not a real difference. On rewrite — the low-noise measurement (±0.3%) — MTP is 7.3% ahead. MTP also costs 0.35 GB inside a single file against a 2.06 GB sidecar. On this hardware MTP is the better choice; DFlash2 is a genuine alternative, not an upgrade.

⛔ Their default n-max 3 is not optimal on gfx1151. The ordering here is n2 (1.68x) > n3 (1.50x) > n5 (1.42x); the published default was tuned on Blackwell. Sweep n-max on your own hardware rather than inheriting it.

⭐ Compare ratios, not absolute tok/s. MTP n2 measured 33.93 on one build and 39.04 on another, but the ratios against each session's own control were 1.656x and 1.643x — 0.8% apart. The runtimes are equivalent; only the baseline moved. Always carry a control arm in the same session.

⛔ What does not work

• You cannot requantize this model. The packs use PrismML-private ggml types GGML_TYPE_PQ2_0 = 142 / GGML_TYPE_PTQ1_0 = 143. A stock llama.cpp cannot read them.
• The repo's F16 is not a quantization source. general.basename = "folded" and it carries prism.hadamard.* — the rotation is folded into the weights and the runtime must apply the matching activation transform. Quantizing it without that yields wrong output, not a worse model.
• A 4-bit rebuild is pointless anyway: the weights are already ~1.72 bits/weight. Going to 4 bpw cannot restore information and roughly doubles the file.
• Do not deploy on an Intel iGPU. On an Arc 140T (Core Ultra 9 285H, ~90 GB/s): Vulkan tg128 1.03, CPU pp512 3.50 with tg128 not completing in 24 minutes.

Files

file	what it is
README.md	this document
qwen35-mtp-hadamard-inverse.patch	the 18-line MTP fix for src/models/qwen35.cpp

Reproduction

# 1. runtime — PrismML's fork, built for gfx1151
git clone -b prism https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp   # tested at 9a9394a
git apply /path/to/qwen35-mtp-hadamard-inverse.patch      # required only for --spec-type draft-mtp
export ROCM_PATH=/opt/rocm HIP_PATH=/opt/rocm
cmake -S . -B build-rocm -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DAMDGPU_TARGETS=gfx1151 \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
  -DCMAKE_HIP_COMPILER_ROCM_ROOT=/opt/rocm \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build-rocm -j8 --target llama-server llama-bench

# 2. weights (upstream, unmodified)
hf download decent-jawfish/bonsai-2-27b-mtp      Bonsai-2-27B-PQ2_0-MTP.gguf        --local-dir .
hf download prism-ml/Ternary-Bonsai-2-27B-gguf   Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf --local-dir .

# 3. serve
./build-rocm/bin/llama-server \
  -m Bonsai-2-27B-PQ2_0-MTP.gguf --mmproj Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf \
  -dev ROCm0 -ngl 999 -fa on --no-mmap -fit off \
  -np 1 -b 2048 -ub 1024 -t 16 -c 131072 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 2

⛔ Plain -DGGML_HIP=ON fails with "Failed to find ROCm root directory" — pass CMAKE_HIP_COMPILER explicitly. ⭐ This fork splits tools into shared libraries: llama-server is a ~17 KB launcher over libllama-server-impl.so. A small binary is not a failed link.

Credits and licence

All weights belong to their authors: prism-ml for Bonsai and the mmproj, decent-jawfish for the MTP graft and for identifying the Hadamard-inverse requirement. The runtime is PrismML-Eng/llama.cpp. This repository adds only measurements and the ROCm build path. Apache-2.0, matching upstream.