christopher-kapic/MiMo-V2.5-ROCmFP4-GGUF

MiMo-V2.5 — ROCmFP4 / ROCmFPX GGUF (Strix Halo / gfx1151)

ROCmFP4 and ROCmFPX quantizations of MiMo-V2.5 (310B total / 15B active MoE), built for AMD Strix Halo (Ryzen AI Max+ 395, gfx1151) with the ROCmFPX fork of llama.cpp.

There are two ways to run this model on Strix Halo, and they want different files:

you have	use	why
two 128 GB boxes + USB4/Thunderbolt	ROCmFP4-FAST/	best quality; 153 GiB does not fit one node
one 128 GB box	Q2_0_ROCMFPX-COHERENT/	90.9 GiB fits a single node — and is +37% faster at decode than the two-node setup

The single-node option being faster is not a mistake. llama.cpp's RPC layer split runs the nodes sequentially, so a second box buys capacity, not speed. If the model fits on one node, one node wins.

Note on quality. The ROCmFP4 and ROCmFP4_FAST variants are not imatrix-calibrated; Unsloth's UD-Q4_K_XL is, and is tensor-aware on top of that. They win on speed and size — benchmark both if marginal output quality matters more to you than tok/s.

Q2_0_ROCMFPX-COHERENT is imatrix-calibrated (it has to be — see below). But it is 2.5 bpw against FAST's 4.26, and that reduction is real. It passes coherence checks on code, factual recall, arithmetic and JSON, but no pass@1 evaluation has been run, so treat the quality gap as unquantified rather than absent.

⚠️ Requirements — stock llama.cpp will not load these

Q4_0_ROCMFP4 and Q4_0_ROCMFP4_FAST are quantization types defined by the ROCmFPX fork. Upstream llama.cpp, Ollama, LM Studio and every downstream that vendors mainline ggml cannot read these files — you will get an unknown-ggml-type error, not a slow model. You must build the fork.

They also target gfx1151 specifically (Ryzen AI Max+ 395 / Strix Halo). The formats are built around that hardware's dequant path; on other GPUs, expect either a build failure or no benefit.

Build

git clone https://github.com/charlie12345/ROCmFPX.git
cd ROCmFPX
git checkout 3edc3d31ee5ebcea47fd7e0f42c89767bb4245db   # the commit these were built and tested with

cmake -S . -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_HIP=ON \
    -DGGML_RPC=ON \
    -DGGML_HIP_FORCE_MMQ=ON \
    -DGGML_HIP_ROCWMMA_FATTN=OFF \
    -DGGML_VULKAN=OFF -DGGML_CUDA=OFF \
    -DCMAKE_HIP_ARCHITECTURES=gfx1151 \
    -DGPU_TARGETS=gfx1151 \
    -DLLAMA_BUILD_SERVER=ON \
    -DLLAMA_BUILD_WEBUI=OFF -DLLAMA_USE_PREBUILT_WEBUI=OFF \
    -DLLAMA_BUILD_TESTS=OFF -DGGML_BUILD_TESTS=OFF

cmake --build build -j "$(nproc)" --target \
    llama-cli llama-server llama-bench llama-quantize rpc-server

-DGGML_RPC=ON is what gives you rpc-server and the RPC0 device — required for the two-node split. Built against ROCm 6.4.

Use the ROCm backend, not Vulkan. RADV imposes a per-buffer allocation ceiling that a model this size runs straight into; -dev Vulkan0 will OOM where ROCm0 works fine.

Making the memory available

A single Strix Halo box defaults to a GPU carve-out far below what these need. Either set GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 in the environment, or raise the GTT limit on the kernel command line (what we run):

amdgpu.gttsize=126976 ttm.pages_limit=32505856 ttm.page_pool_size=32505856

With the cmdline set, UNIFIED_MEMORY is no longer needed. We measured no throughput difference between the two approaches — the cmdline is just less fragile.

⚡ Force the GPU power level — worth 14% decode

This is the single highest-value host setting, and it is easy to miss.

echo high | sudo tee /sys/class/drm/card*/device/power_dpm_force_performance_level

power_dpm_force_performance_level	pp512	tg128
auto (default)	322.87	20.94
high	329.43	23.87

+14% decode, +2% prefill, from one sysfs write. The DPM governor never ramps to peak on MoE decode — the workload is bursty and low-occupancy, so auto reads it as near-idle and leaves the GPU and fabric below spec. Effective memory bandwidth goes from 167 to 190.7 GB/s (65% -> 75% of the 256 GB/s theoretical). DRAM is at its rated 8000 MT/s either way; this is purely a clock-governor effect.

The gain is specific to single-stream decode. Under concurrency the GPU is already loaded enough that auto ramps by itself — aggregate throughput at C8 is unchanged.

Transparent hugepages were tested alongside this and make no difference (21.24 with THP alone vs 20.94 baseline), despite the 153 GiB working set. Not worth the system-wide side effects.

It resets on reboot. To persist:

# /etc/systemd/system/amdgpu-perf-high.service
[Unit]
Description=Force amdgpu DPM to high
After=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'for c in /sys/class/drm/card*/device/power_dpm_force_performance_level; do echo high > "$c"; done'
ExecStop=/bin/sh -c 'for c in /sys/class/drm/card*/device/power_dpm_force_performance_level; do echo auto > "$c"; done'

[Install]
WantedBy=multi-user.target

Apply it on every node — the RPC peer's clocks matter just as much as the head node's.

Variants

variant	effective bpw	size	shards	fits 1 node?	folder
Q4_0_ROCMFP4_FAST ← best quality	4.26	153.47 GiB (164,795,562,720 B)	4	no	ROCmFP4-FAST/
Q2_0_ROCMFPX ← fastest, single-node	2.52	90.92 GiB (97,626,242,048 B)	3	yes	Q2_0_ROCMFPX-COHERENT/
Q4_0_ROCMFP4	5.24	189.02 GiB (202,969,561,824 B)	5	no	ROCmFP4/

For reference, unsloth/MiMo-V2.5-GGUF UD-Q4_K_XL is 178.44 GiB at 4.95 bpw — ROCmFP4_FAST is 14% smaller.

ROCmFP4_FAST dominates ROCmFP4 on every axis we measured — smaller, faster prefill, faster decode. Unless you specifically want the higher-precision tensors, take FAST. ROCmFP4 is published for completeness.

A note on the bpw labels. llama-quantize advertises Q4_0_ROCMFP4 as "4.50 bpw", but measured against the 309.77 B parameter count the real figure is 5.24 bpw — the recipe promotes several tensor classes (e.g. ffn_gate → q5_K) rather than quantizing everything to ROCmFP4. Q4_0_ROCMFP4_FAST measures 4.26 bpw against its 4.25 label, so that one is honest. This is why the "non-fast" build ends up larger than UD-Q4_K_XL rather than smaller.

All were quantized from the BF16 GGUF (unsloth/MiMo-V2.5-GGUF, 14 shards, 619,638,702,336 bytes, verified byte-exact).

The 4-bit variants need no special handling:

llama-quantize MiMo-V2.5-BF16-00001-of-00014.gguf \
    MiMo-V2.5-ROCmFP4-FAST.gguf Q4_0_ROCMFP4_FAST 16

Q2_0_ROCMFPX needs two extra flags or it produces garbage

This is the important part of this section. A plain Q2_0_ROCMFPX run — no imatrix, default tensor types — yields a model that loads, benchmarks at a healthy 34.97 t/s, and emits this:

，问题，1.问题，1.问题，10000000000000000000000000000000000000...

Fluent-looking throughput, zero usable output. Both of the following are required:

llama-quantize \
    --imatrix MiMo-V2.5-imatrix.gguf \
    --token-embedding-type q6_k \
    --output-tensor-type q6_k \
    MiMo-V2.5-BF16-00001-of-00014.gguf \
    MiMo-V2.5-Q2_0_ROCMFPX-COHERENT.gguf Q2_0_ROCMFPX 16

• --imatrix — sub-3-bit quantization needs importance-weighted rounding. The proof is that mainline IQ2_XXS stays coherent at 2.06 bpw, lower than our 2.5, because it is imatrix-aware. Uniform rounding survives 4 bpw; it does not survive 2.5. The imatrix used here is bartowski's (MiMo-V2.5-imatrix.gguf, 287 entries over 818 chunks).
• --token-embedding-type q6_k --output-tensor-type q6_k — the embedding table (152,576 tokens) and LM head are the most precision-sensitive tensors in the model, and the imatrix does not cover them (did not find weights for token_embd.weight). Crush them to 2.5 bpw and nearby token vectors collapse into one another, which is exactly the wrong-language-token + repetition signature above. Cost: +0.3 GB on a 97 GB file.

ROCmFPX's own naming corroborates this: Q4_0_ROCMFP4_COHERENT is defined as "ROCmFP4 + Q6_K token embeddings". Nobody names a variant coherent unless the alternative was incoherent. Q2_0_ROCMFPX ships with no such provision, hence the -COHERENT suffix on this folder.

Measured performance

Test setup

Two Beelink GTR 9 Pro (Ryzen AI Max+ 395, gfx1151, 128 GB unified) linked by a single USB4 cable. The 4-bit variants are layer-split across both with llama.cpp RPC; Q2_0_ROCMFPX runs on one box with no RPC at all (-ngl 999, no -dev), which is the whole reason it decodes faster. All numbers are single-stream unless the concurrency section says otherwise.

engine	ROCmFPX (llama.cpp fork), ROCm 6.4, HIP backend
OS / kernel	Ubuntu 24.04, mainline 6.18.6
topology	node2 = head, node1 = rpc-server over Thunderbolt (192.168.2.1:50052)
GPU carve-out	126976 MiB via amdgpu.gttsize / ttm.pages_limit kernel cmdline
transport	TCP over thunderbolt0 (RDMA measured, no difference)

Benchmark command:

llama-bench -m <model.gguf> \
    -rpc 192.168.2.1:50052 -dev ROCm0/RPC0 \
    -ngl 999 -fa 1 -mmp 0 -r 1 -p 512 -n 128

-mmp 0 (no mmap) is required — with mmap the working set thrashes against the 128 GB of RAM and never converges. Note llama-bench wants -dev entries separated by /, while llama-cli wants ,.

Single-stream results

All rows below are measured with power_dpm_force_performance_level=high on both nodes (see Requirements) — without it every number drops 9–14%.

variant	nodes	size	prefill (pp512)	decode (tg128)
Q2_0_ROCMFPX	1	90.92 GiB	241.05 t/s	32.53 t/s
Q4_0_ROCMFP4_FAST	2	153.47 GiB	328.73 t/s	23.80 t/s
(reference) UD-Q4_K_XL	2	178.44 GiB	339.31 t/s	17.16 t/s
Q4_0_ROCMFP4 (at dpm=auto)	2	189.02 GiB	248.10 t/s	16.80 t/s

Q2_0_ROCMFPX on one node decodes 36.7% faster than ROCmFP4_FAST on two, for 26.7% less prefill. Two effects compound: fewer bytes to stream per token (2.52 vs 4.26 bpw), and no RPC layer split — which runs the nodes sequentially, so it costs decode latency rather than saving it. Prefill is the opposite case: it is compute-bound and genuinely benefits from a second GPU, which is why the two-node rows win there.

Q2 error bars over -r 2: pp512 ±0.75, tg128 ±0.08.

ROCmFP4_FAST is +38.7% decode over UD-Q4_K_XL for −3.1% prefill, while being 14% smaller. The decode gain far exceeds what the size reduction alone predicts (4.26 vs 4.95 bpw) — the single-scale layout also dequantizes more cheaply on gfx1151.

Both were re-measured at dpm=high so the comparison is like-for-like. Worth noting UD-Q4_K_XL gains only ~9% from that tuning where ROCmFP4_FAST gains 14%, consistent with the FAST dequant path being more clock-sensitive. At the old dpm=auto default the gap read as +33%.

ROCmFP4 (non-fast) has not been re-measured at dpm=high; its row is from dpm=auto and is not comparable to the two above. It was the weakest of the three on throughput at equal settings and is published only for completeness.

Reproducibility: ROCmFP4_FAST at dpm=high was measured three times — 329.43/23.87, 329.14/23.89, 328.73/23.80 — a 0.2% spread on pp512 and 0.4% on tg128.

Decode vs. context depth

Q4_0_ROCMFP4_FAST, with power_dpm_force_performance_level=high:

depth	pp512	tg128
0	328.73	23.80
8192	286.70	23.00
32768	214.53	21.72

Decode is remarkably flat with context — a consequence of the 9-full/39-SWA attention split, where only 9 layers grow with depth. Prefill decays normally.

For reference, the same curve at the default dpm=auto was 315.5/21.07, 267.1/20.68 and 197.7/19.53 — the tuning is worth ~11–13% of decode at every depth.

Q2_0_ROCMFPX on a single node, same benchmark, dpm=high, -r 2:

depth	pp512	tg128	vs FAST 2-node decode
0	241.05 ± 0.75	32.53 ± 0.08	+36.7%
8192	220.14 ± 1.73	29.19 ± 0.67	+26.9%
32768	177.29 ± 1.66	28.64 ± 0.14	+31.9%

Q2 on one node beats FAST on two at decode at every depth, and loses at prefill at every depth. Its decode curve is also flatter past 8k (29.19 → 28.64, −1.9% over a 4× context increase) — same 9-full/39-SWA attention structure as the 4-bit variants, with less weight traffic per token on top.

Pick by workload: long prompts / short answers → two-node FAST (prefill wins). Short prompts / long answers, or interactive chat → single-node Q2 (decode wins, and on one machine).

Aggregate throughput under concurrency

Total tokens/s across all streams, 16 distinct prompts so that slots cannot share a prefix-cache hit and inflate the result (--parallel 8, -c 32768).

variant	C4	C6	C8
Q4_0_ROCMFP4_FAST (dpm=high)	33.20 t/s	34.51 t/s	33.57 t/s
Q4_0_ROCMFP4_FAST (dpm=auto)	31.93 t/s	32.61 t/s	33.92 t/s
Q4_0_ROCMFP4 (dpm=auto)	24.96 t/s	26.37 t/s	28.32 t/s
(reference) UD-Q4_K_XL (dpm=auto)	25.81 t/s	29.19 t/s	31.72 t/s

Aggregate throughput plateaus around 33–35 t/s and the dpm=high tuning barely helps here — under concurrency the GPU is already busy enough that the governor ramps on its own. That is the mirror image of the single-stream case, where forcing high is worth 14%.

ROCmFP4_FAST is fastest at every concurrency level, but its margin over UD-Q4_K_XL shrinks as concurrency rises — the FAST layout's advantage is in memory bandwidth and dequant cost, which dominate single-stream decode; as batch size grows the workload shifts toward expert scatter and compute, where the quants converge.

MiMo-V2.5 is a 256-expert top-8 MoE, the least favourable case for batch amortization: the number of distinct experts touched at batch B grows as 256·(1−(1−8/256)^B), so more expert weights must be read as concurrency rises instead of being amortized across the batch. This is why aggregate throughput scales so weakly — ROCmFP4_FAST gains only 6% going from 4 streams to 8, and per-stream latency roughly halves over that range (7.98 → 4.24 t/s).

Usage

Single node — Q2_0_ROCMFPX (fastest decode)

GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 \
llama-server -m Q2_0_ROCMFPX-COHERENT/MiMo-V2.5-Q2_0_ROCMFPX-COHERENT-00001-of-00003.gguf \
    -ngl 999 -fa 1 --no-mmap -ctk f16 -ctv f16 \
    -c 32768 --parallel 8 --jinja --host 0.0.0.0 \
    --temp 1.0 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.05

No --rpc, no -dev. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 is required — 90.92 GiB exceeds the hard VRAM cap even though it fits the 128 GB unified pool.

Two nodes — Q4_0_ROCMFP4_FAST (best quality)

# node1
rpc-server -H 0.0.0.0 -p 50052

# node2
llama-server -m ROCmFP4-FAST/MiMo-V2.5-ROCmFP4-FAST-00001-of-00004.gguf \
    --rpc 192.168.2.1:50052 -dev ROCm0,RPC0 \
    -ngl 999 -fa 1 --no-mmap -ctk f16 -ctv f16 \
    -c 32768 --parallel 8 --jinja --host 0.0.0.0 \
    --temp 1.0 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.05

Point llama.cpp at the first shard; it pulls in the rest automatically.

Sampling — use the model's own recommendation

Xiaomi's model card specifies temperature=1.0, top_p=0.95 for local deployment. --repeat-penalty 1.05 and --min-p 0.0 come from the community configuration in llama.cpp #23074.

Use --jinja with the model's built-in chat template. No corrected template exists or is needed — neither Unsloth nor bartowski overrides it. You may see Expected iterable or object type in for loop: got None in the log during reasoning-format detection; it is harmless and also appears on runs that work.

Note a MiMo-level quirk, not a quantization artifact: MiMo-V2.5 can enter runaway chain-of-thought — "extremely long CoT (sometimes for hundreds of thousands of tokens) with no progress or tool calls" (#23074), reproduced on Q8_0, so it is not caused by low-bit quantization. Cap it with --reasoning-budget, and pass "chat_template_kwargs": {"enable_thinking": false} per-request to disable thinking entirely. Thinking costs ~0.5% of decode rate — its real cost is the extra tokens.

Mind the flag spelling — llama-server takes --rpc and comma-separated -dev, while llama-bench takes -rpc and slash-separated -dev (ROCm0/RPC0). Passing the wrong one makes the server exit immediately with error: invalid argument.

Things worth knowing

• Keep KV cache at f16 for speed. Measured on this exact model and split:

  -ctk / -ctv	tg128 @d0	tg128 @32k
  f16 / f16	20.71	19.44
  q8_0 / q8_0	20.26	17.13
  q8_0 / q4_0	20.17	17.24
  q4_0 / q4_0	20.04	17.22

  The penalty is ~2% at zero depth but 13% at 32k — ROCm dequant costs more than the bandwidth it saves, and the gap widens as the cache fills. Note the three quantized configs are indistinguishable: the cost comes from quantizing at all, not from how aggressively.

  But quantizing buys context. f16 KV is 22.5 KiB/token here; q4_0/q4_0 is ~6.6 KiB — about 3.4× the KV pool for that 13%. If maximum context matters more to you than decode speed, -ctk q4_0 -ctv q4_0 is the trade.

• Prefer the USB4/Thunderbolt link over ethernet for the RPC hop. Against a switched 1 GbE path we measured −5% decode and −9.5% prefill. The causes differ: decode is latency-bound (llama.cpp RPC does ~3–4 round trips per token, so hop latency multiplies), while prefill is bandwidth-bound (4 MB of activations per 512-token chunk). A switched 10 GbE link fixes the prefill half but not the decode half; a direct point-to-point cable fixes both.

• Speculative decoding does not pay off on this model. Measured on this exact build and split, with --temp 0 --repeat-penalty 1.0:

  workload	no speculation	DFlash n=1	DFlash n=2	DFlash n=4
  structured JSON	21.00	14.87	13.80	10.02
  Rust code	21.18	13.00	11.96	8.94
  narrative prose	21.21	12.19	10.72	8.16

  DFlash costs 30–60%, and gets worse the deeper you draft. The cause is acceptance. Mean accept length is 1.52 / 1.32 / 1.23 (JSON / code / prose), and per-position acceptance on JSON runs 0.500, 0.150, 0.011, 0.000 — the drafter is right about half the time on token 1 and essentially never by token 3.

  A DFlash step costs 2.15× a normal decode step here, so break-even needs accept length ≥ 2.15. Reference implementations on other engines report 3.78 on this same model and drafter, which would be ~1.76× — so the headroom is real, it is just not reachable from this engine. An independent DFlash draft GGUF for MiMo-V2.5-Pro on ik_llama.cpp reports the same shape (54.6–60.4% acceptance, 55.6–59.4 t/s drafted vs 59.9–60.8 undrafted — also a net loss).

  Measured with thinking off; enabling it changes JSON and prose by ~0 and costs code about 16%.

  MTP is unavailable (llama.cpp issue #23924 closed not_planned, though these GGUFs do carry the blk.48-50.nextn.* tensors), and no EAGLE3 drafter has been published for MiMo-V2.5.

• If you do experiment with DFlash over an RPC split, the target's LM head and token embeddings must be pinned to the local device or it aborts at load in ggml_backend_sched_backend_id_from_cur: -ot "output\.weight=ROCm0" -ot "token_embd\.weight=ROCm0" -devd ROCm0

• KV geometry: MiMo-V2.5 has 9 full-attention + 39 sliding-window layers, so only the 9 full layers scale with context — about 22.5 KiB/token.

Speculative decoding (MTP) — present, but do not expect a win

MiMo-V2.5 ships its own MTP / NextN head (model_mtp.safetensors upstream → blk.48/49/50.nextn.* in GGUF, 3 NextN layers on top of 48 trunk layers). All files here preserve those tensors, so they are available if your build can use them.

Using them requires engine support that is not yet in any release. llama.cpp PR #26228 adds mimo2 MTP draft support and is still open; the ROCmFPX fork does not carry it either. Without it the nextn tensors are simply ignored (harmless — the files still load and run normally on a stock build; verified).

Two warnings, both measured on this hardware, because the intuitive expectations are wrong in both directions:

1. Across a two-node RPC split, MTP is a 33% loss — even at 100% acceptance.

2-node ROCmFP4_FAST, -dev RPC0,ROCm0	decode	acceptance
no MTP (control)	23.55 t/s	—
MTP, n-max 1	15.87 t/s	1.000 (27/27)

Acceptance was perfect and it still lost a third of throughput. The deficit is per-draft-step cross-device synchronisation, not draft quality — which means no better drafter can fix it. EAGLE3 would be worse still: it needs three hidden-state taps from layers spread across the trunk, so on a layer split some taps are always remote, whereas MTP needs only the final pre-norm hidden state.

If you do try it, note the load-time crash and its fix: with -dev ROCm0,RPC0 the trailing nextn blocks land on the remote node, and reading the hidden state back aborts inside ggml_backend_rpc_buffer_get_tensor during startup. Flip the device order to -dev RPC0,ROCm0 — layers are assigned to -dev entries in order, so the last device named gets the trailing blocks. -ot moves weights but not the KV cache; pinning weights alone does not fix it.

2. On a single node, whether MTP helps depends on the quant.

single node	base	best MTP	verdict
IQ2_XXS (third-party, 2.06 bpw)	20.75 t/s	25.81 t/s (+24%)	worth it
Q2_0_ROCMFPX (this repo)	35.57 t/s	31.50 t/s (−11%)	not worth it

Two compounding reasons MTP loses on Q2_0_ROCMFPX: the base is already fast, so there is less fixed overhead to hide, and the nextn head is itself quantized to 2.5 bpw — mean acceptance length falls to 1.39–1.75 here versus 1.95 on IQ2_XXS. Lifting token_embd/output to Q6_K does not help the drafter; the nextn tensors stay at 2.5 bpw.

Two counter-intuitive tuning notes if you experiment: higher --spec-draft-p-min is better (0.75 → 31.50 vs 0.10 → 30.26; fewer, more confident drafts waste less verify work), and --spec-draft-n-max 1 is optimal — depth ≥2 loses badly (22.19 t/s at n=2). Deeper drafts widen the verify batch, and on a 256-expert top-8 MoE a wider batch activates more experts, so weight traffic grows faster than accepted tokens. Mean acceptance length keeps rising with depth while throughput falls, so acceptance alone is a misleading metric here.

License

Inherits the license of the base model, XiaomiMiMo/MiMo-V2.5. Quantization adds no additional restrictions.

Credits

• XiaomiMiMo — MiMo-V2.5
• unsloth — BF16 GGUF conversion used as the source
• ROCmFPX — the ROCmFP4 quant formats