Qwen2.5-7B-Instruct (Q4_K_M) for AMD Loom

A Loom package (.loom programs and a model.json card) for the unmodified GGUF checkpoint bartowski/Qwen2.5-7B-Instruct-GGUF (Qwen2.5-7B-Instruct-Q4_K_M.gguf), for the AMD Ryzen AI Max+ 395 "Strix Halo", AMD Radeon AI PRO R9700 and AMD Radeon gfx1201 16 GB class (RX 9070 XT, RX 9070) GPUs, generated with the loom-skills workflow. Learn more on the Loom ecosystem website.

Quick start

Open your GPU and copy its commands.

AMD Ryzen AI Max+ 395 "Strix Halo" (gfx1151)
pipx install "hrx-loom[hf]==0.2.0"
loom-fetch huggingface:loom-fun-staging/bartowski-Qwen2.5-7B-Instruct-loom:chat --materialize=copy --output=Qwen2.5-7B-Instruct
loom-serve --model=Qwen2.5-7B-Instruct '--device=amdgpu://?completion_spin_ns=4294967295' --pool_backing=fixed --parameter_mode=preload
  • --materialize=copy puts copies of the fetched package files in the output directory.
  • completion_spin_ns=4294967295 enables active waiting for GPU completions (uses CPU while GPU work is in flight); the card's performance measurements use this setting.
  • --pool_backing=fixed commits the preset's memory pool up front instead of growing elastic backing.
  • --parameter_mode=preload only changes how the weights are loaded on the host, so the performance rows were measured without it unless their job served with it.

In another terminal, send a request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen2.5-7B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
AMD Radeon AI PRO R9700 (gfx1201, 32 GB)
pipx install "hrx-loom[hf]==0.2.0"
loom-fetch huggingface:loom-fun-staging/bartowski-Qwen2.5-7B-Instruct-loom:chat --materialize=copy --output=Qwen2.5-7B-Instruct
loom-serve --model=Qwen2.5-7B-Instruct '--device=amdgpu://?completion_spin_ns=4294967295' --pool_backing=fixed --parameter_mode=preload
  • --materialize=copy puts copies of the fetched package files in the output directory.
  • completion_spin_ns=4294967295 enables active waiting for GPU completions (uses CPU while GPU work is in flight); the card's performance measurements use this setting.
  • --pool_backing=fixed commits the preset's memory pool up front instead of growing elastic backing.
  • --parameter_mode=preload only changes how the weights are loaded on the host, so the performance rows were measured without it unless their job served with it.

In another terminal, send a request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen2.5-7B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
Radeon gfx1201, 16 GB (RX 9070 XT, RX 9070)
pipx install "hrx-loom[hf]==0.2.0"
loom-fetch huggingface:loom-fun-staging/bartowski-Qwen2.5-7B-Instruct-loom:chat-16gb --materialize=copy --output=Qwen2.5-7B-Instruct
loom-serve --model=Qwen2.5-7B-Instruct '--device=amdgpu://?completion_spin_ns=4294967295' --memory_bytes=15000000000 --state_slab_bytes=4mib --weight_slab_bytes=64mib --parameter_mode=preload
  • --materialize=copy puts copies of the fetched package files in the output directory.
  • completion_spin_ns=4294967295 enables active waiting for GPU completions (uses CPU while GPU work is in flight); the card's performance measurements use this setting.
  • --memory_bytes sets the serving memory budget for this GPU.
  • --state_slab_bytes sets the allocation slab size for state memory.
  • --weight_slab_bytes sets the allocation slab size for weight memory.
  • --parameter_mode=preload only changes how the weights are loaded on the host, so the performance rows were measured without it unless their job served with it.

In another terminal, send a request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Qwen/Qwen2.5-7B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'

The server listens on 127.0.0.1:8080 and answers OpenAI-style /v1/chat/completions requests.

Performance vs llama.cpp

Strix Halo (gfx1151)

metric llama.cpp Loom margin vs best llama.cpp roofline Loom % of roofline llama.cpp Vulkan
decode tok/s, short prompt 46.4 46.8 +1 % 49.4 95 % 45.5
decode tok/s, 2k context 43.6 44.3 +2 % 48.2 92 % 43.6
decode tok/s, 32k context 21.9 33.4 +4 % 34.8 96 % 32.2
prefill tok/s, 1k document 1428.2 1968.4 +38 % 4452.4 44 % 1136.4
prefill tok/s, 2k document 1434.2 2053.7 +43 % 4386.0 47 % 1161.1
prefill tok/s, 8k document 1281.1 1864.2 +46 % 4025.6 46 % 1014.7
prefill tok/s, 16k document 1085.3 1648.1 +52 % 3628.1 45 % 861.7
prefill tok/s, 32k document 825.6 1204.9 +46 % 3029.8 40 % 661.8
4 streams, aggregate decode tok/s 144.8 175.4 +21 % 191.7 91 % 143.6
speculative decoding, drafted tokens accepted n/a n/a n/a n/a n/a n/a

Methodology

  • Compared: Loom against llama.cpp ROCm llamacpp-rocm/b1374, llama.cpp Vulkan llamacpp-vulkan/b12211; the same GGUF, the same DevLab runner type, different jobs, the same prompts; Loom served with the active wait. Values are the median of 3 sessions (4 streams: median of 6); Loom is timed by the client, llama.cpp by its server.
  • llama.cpp measured on 2026-11-13 (DevLab Dispatch run 43392598780), Loom on 2026-11-17 (DevLab Dispatch run 44088665443): same runner type, different jobs (llama.cpp was not re-measured for this package).
  • Out of the box on both engines: Loom serves the preset shipped in this package; llama.cpp runs its build defaults plus only the server arguments listed below (offload, context and slots). No tuning sweep on either engine.
  • Drafting, 1-stream rows (decode, decode_2k, decode_32k, prefill): no drafting on either engine (no MTP head in the GGUF and no drafter published for this exact model). Each engine is loaded once for all these rows.
  • 4 streams: both engines loaded with drafting off, 4 parallel requests, aggregate tokens per second.
  • Rows: decode = short-context decode; decode_2k / decode_32k = decode after a ~2k / ~32k-token context; prefill_Nk = prompt processing of an N k-token prompt; streams4 = 4-stream aggregate decode. A margin within +-3 % (the measured same-runner noise) is a tie.
  • llama.cpp ROCm server arguments: 1-stream rows: -ngl 99 -cram 0 -c 65536 -np 1; 4 streams: -ngl 99 -cram 0 -c 65536 -kvu -np 4.
  • llama.cpp Vulkan server arguments: 1-stream rows: -ngl 99 -cram 0 -c 36864 -np 1; 4 streams: -ngl 99 -cram 0 -c 147456 -np 4.
  • Measured on AMD DevLab Dispatch Strix Halo runners (devlab-perf jobs); runners differ between sessions.
  • Margin vs best llama.cpp: (Loom / max(ROCm, Vulkan) - 1) Γ— 100 %, using the available measured tokens/s for each row on this GPU. With only one valid backend, that backend is used; with neither, the margin is not measured. Raw backend values and their measurement sources are retained below and in the table.
  • Measured 2026-11-17; device AMD Strix Halo (DevLab Dispatch runner X2-LA03-S01-D08), Linux; hrx-loom 0.2.0 (hrx-system 0cd370f, recovered); source: DevLab Dispatch run 44088665443.
  • llama.cpp Vulkan: measured 2026-11-13; source: DevLab Dispatch run 43392598780. Archive sha256 9759acd.
  • Roofline: the hardware ceiling for this GGUF on AMD Ryzen AI Max+ 395 "Strix Halo" (gfx1151) at 216 GB/s and 59 TFLOP/s, from loom-roofline roofline.py (loom-skills 1868f0b).
  • β€” not measured; n/a not applicable.

Radeon AI PRO R9700 (gfx1201, 32 GB)

metric llama.cpp Loom margin vs best llama.cpp roofline Loom % of roofline llama.cpp Vulkan
decode tok/s, short prompt 108.2 117.4 +2 % 140.3 84 % 115.6
decode tok/s, 2k context 104.5 108.7 -3 % 136.7 80 % 112.5
decode tok/s, 32k context 68.9 69.3 -18 % 98.8 70 % 84.9
prefill tok/s, 1k document 4435.0 6040.3 +36 % 16149.4 37 % 3191.0
prefill tok/s, 2k document 4449.4 6358.5 +43 % 15908.4 40 % 3286.2
prefill tok/s, 8k document 3976.6 5869.9 +48 % 14601.3 40 % 3009.3
prefill tok/s, 16k document 3422.0 5195.1 +52 % 13159.6 39 % 2642.9
prefill tok/s, 32k document 2678.9 3824.5 +43 % 10989.5 35 % 2113.5
4 streams, aggregate decode tok/s 295.6 423.1 +17 % 543.9 78 % 362.8
speculative decoding, drafted tokens accepted n/a n/a n/a n/a n/a n/a

Methodology

  • Compared: Loom against llama.cpp ROCm llamacpp-rocm/b1375, llama.cpp Vulkan llamacpp-vulkan/b12211; the same GGUF, the same DevLab runner type, different jobs, the same prompts; Loom served with the active wait. Values are the median of 3 sessions (4 streams: median of 6); Loom is timed by the client, llama.cpp by its server.
  • llama.cpp measured on 2026-11-14 (DevLab Dispatch run 43633449157), Loom on 2026-11-17 (DevLab Dispatch run 43993223590): same runner type, different jobs (llama.cpp was not re-measured for this package).
  • Out of the box on both engines: Loom serves the preset shipped in this package; llama.cpp runs its build defaults plus only the server arguments listed below (offload, context and slots). No tuning sweep on either engine.
  • Drafting, 1-stream rows (decode, decode_2k, decode_32k, prefill): no drafting on either engine (no MTP head in the GGUF and no drafter published for this exact model). Each engine is loaded once for all these rows.
  • 4 streams: both engines loaded with drafting off, 4 parallel requests, aggregate tokens per second.
  • Rows: decode = short-context decode; decode_2k / decode_32k = decode after a ~2k / ~32k-token context; prefill_Nk = prompt processing of an N k-token prompt; streams4 = 4-stream aggregate decode. A margin within +-3 % (the measured same-runner noise) is a tie.
  • llama.cpp ROCm server arguments: -ngl 99 -cram 0 -c 65536 -np 1.
  • llama.cpp Vulkan server arguments: -ngl 99 -cram 0 -c 36864 -np 1.
  • Measured on AMD DevLab Dispatch R9700 runners (devlab-perf jobs); runners differ between sessions.
  • Margin vs best llama.cpp: (Loom / max(ROCm, Vulkan) - 1) Γ— 100 %, using the available measured tokens/s for each row on this GPU. With only one valid backend, that backend is used; with neither, the margin is not measured. Raw backend values and their measurement sources are retained below and in the table.
  • Measured 2026-11-17; device DevLab Dispatch runner ephemeral-devlab-sm-01-02-r9700s-3-133738799824, AMD Radeon AI PRO R9700S (gfx1201), Linux; hrx-loom 0.2.0 (hrx-system commit unknown, recovered); source: DevLab Dispatch run 43993223590.
  • llama.cpp Vulkan: measured 2026-11-14; source: DevLab Dispatch run 43633449157. Archive sha256 9759acd.
  • Roofline: the hardware ceiling for this GGUF on AMD Radeon AI PRO R9700S (gfx1201, 32 GB, 2.35 GHz) at 613 GB/s and 214 TFLOP/s, from loom-roofline roofline.py (loom-skills 203cf33).
  • β€” not measured; n/a not applicable.

Radeon RX 9070 XT

metric llama.cpp Loom margin vs best llama.cpp roofline Loom % of roofline llama.cpp Vulkan
decode tok/s, short prompt 109.2 117.7 +0 % 146.4 80 % 117.5
decode tok/s, 2k context 105.1 107.8 -6 % 142.7 76 % 114.1
decode tok/s, 32k context 70.8 69.3 -19 % 103.1 67 % 85.9
prefill tok/s, 1k document 4346.5 6025.6 +39 % 14685.4 41 % 3789.5
prefill tok/s, 2k document 4363.8 6388.3 +46 % 14466.3 44 % 3868.3
prefill tok/s, 8k document 3898.2 5986.7 +54 % 13277.6 45 % 3543.5
prefill tok/s, 16k document 3331.9 5294.5 +59 % 11966.7 44 % 3114.8
prefill tok/s, 32k document 2620.7 3816.5 +46 % 9993.3 38 % 2504.9
4 streams, aggregate decode tok/s 301.1 429.8 +13 % 567.9 76 % 379.1
speculative decoding, drafted tokens accepted n/a n/a n/a n/a n/a n/a

Methodology

  • Compared: Loom against llama.cpp ROCm llamacpp-rocm/b1375, llama.cpp Vulkan llamacpp-vulkan/b12211; the same GGUF, the same machine and session, the same prompts; Loom served with the active wait. Values are the median of 3 sessions (4 streams: median of 3); Loom is timed by the client, llama.cpp by its server.
  • Out of the box on both engines: Loom serves the preset shipped in this package; llama.cpp runs its build defaults plus only the server arguments listed below (offload, context and slots). No tuning sweep on either engine.
  • Drafting, 1-stream rows (decode, decode_2k, decode_32k, prefill): no drafting on either engine (no MTP head in the GGUF and no drafter published for this exact model). Each engine is loaded once for all these rows.
  • 4 streams: both engines loaded with drafting off, 4 parallel requests, aggregate tokens per second.
  • Rows: decode = short-context decode; decode_2k / decode_32k = decode after a ~2k / ~32k-token context; prefill_Nk = prompt processing of an N k-token prompt; streams4 = 4-stream aggregate decode. A margin within +-3 % (the measured same-runner noise) is a tie.
  • llama.cpp ROCm server arguments: -ngl 99 -cram 0 -c 65536 -np 1.
  • llama.cpp Vulkan server arguments: -ngl 99 -cram 0 -c 36864 -np 1.
  • Measured on AMD Ryzen 7 9800X3D 8-Core Processor, AMD Radeon RX 9070 XT, Ubuntu 26.04, a named machine (this GPU has no DevLab runner); both engines ran on it.
  • Margin vs best llama.cpp: (Loom / max(ROCm, Vulkan) - 1) Γ— 100 %, using the available measured tokens/s for each row on this GPU. With only one valid backend, that backend is used; with neither, the margin is not measured. Raw backend values and their measurement sources are retained below and in the table.
  • Measured 2026-11-17; device AMD Ryzen 7 9800X3D 8-Core Processor, AMD Radeon RX 9070 XT, Ubuntu 26.04; hrx-loom 0.2.0 (hrx-system 0cd370f); source: named-machine run.
  • llama.cpp Vulkan: measured 2026-11-17; source: named-machine run. Archive sha256 9759acd.
  • Roofline: the hardware ceiling for this GGUF on AMD Radeon RX 9070 XT (gfx1201, 16 GB) at 640 GB/s and 195 TFLOP/s, from loom-roofline roofline.py (loom-skills 963ead9).
  • β€” not measured; n/a not applicable.

Features

feature status detail
API βœ… OpenAI-style /v1/chat/completions
tool calling βœ… OpenAI-style tools and tool_calls
reasoning / thinking β€” not applicable
reasoning effort β€” not applicable
speculative decoding β€” not applicable
image input β€” not applicable
audio input / output β€” not applicable
sampling βœ… temperature, top_p, top_k, seed
context size βœ… 32,768 tokens per request
concurrency βœ… 4 concurrent requests
max output tokens βœ… 4,096 (preset max_tokens)

Correctness

gate description Strix Halo Radeon AI PRO R9700 Radeon RX 9070 XT
kernel_tests Unit tests of the generated kernels against reference implementations βœ… pass (59/59) ΒΉ βœ… pass (59/59) Β² βœ… pass (59/59) Β³
golden llama.cpp reference recorded on the same GGUF: greedy tokens and top-5 log-probs, short prompts and 1k-32k documents βœ… pass ⁴ βœ… pass ⁡ βœ… pass ⁢
compare Greedy output vs the llama.cpp reference, token by token; every divergence is a near-tie βœ… pass ⁴ βœ… pass ⁡ βœ… pass ⁢
vision Image input vs llama.cpp with the same projector: identical prompt tokens, replies equal up to near-ties β€” not applicable β€” not applicable β€” not applicable
consistency Concurrent requests give the same output as the same requests run alone βœ… pass ⁴ βœ… pass ⁡ βœ… pass ⁢
spec Speculative decoding on gives exactly the output of speculative decoding off β€” not applicable β€” not applicable β€” not applicable
agent Multi-turn tool-calling loop: exact prompt rendering, history reuse, valid tool calls, output vs llama.cpp βœ… pass ⁴ βœ… pass ⁡ βœ… pass ⁢
agent_think_off The tool-calling loop with thinking off β€” not applicable β€” not applicable β€” not applicable
sampling temperature, top_k and seed behave as specified; first-token distribution matches llama.cpp βœ… pass ⁴ βœ… pass ⁡ βœ… pass ⁢
logits Host logits readback (llm.logits on): at T=0 every returned row's argmax is the device's token, the top-k matches llama.cpp, greedy text is unchanged ? not re-run on this package (the last pass was on a previous build, sha256 b989b3f; the nightly gates re-run it) ? not re-run on this package (the last pass was on a previous build, sha256 b989b3f; the nightly gates re-run it) ❌ no evidence
decision Decision answers (choice, score, noul probabilities) vs llama.cpp on the same prompts β€” not applicable β€” not applicable β€” not applicable
  • ΒΉ DevLab Dispatch run 44088665443, 2026-11-17; device AMD Strix Halo (DevLab Dispatch runner X2-LA03-S01-D08), Linux; hrx-loom 0.2.0 (hrx-system 0cd370f, recovered).
  • Β² DevLab Dispatch run 43993223590, 2026-11-17; device DevLab Dispatch runner ephemeral-devlab-sm-01-02-r9700s-3-133738799824, AMD Radeon AI PRO R9700S (gfx1201), Linux; hrx-loom 0.2.0 (hrx-system commit unknown, recovered).
  • Β³ named-machine run, 2026-11-17; device AMD Ryzen 7 9800X3D 8-Core Processor, AMD Radeon RX 9070 XT, Ubuntu 26.04; hrx-loom 0.2.0 (hrx-system 0cd370f).
  • ⁴ DevLab Dispatch run 44088665443, LEVEL=dev, 2026-11-17; device AMD Strix Halo (DevLab Dispatch runner X2-LA03-S01-D08), Linux; hrx-loom 0.2.0 (hrx-system 0cd370f, recovered).
  • ⁡ DevLab Dispatch run 43993223590, LEVEL=dev, 2026-11-17; device DevLab Dispatch runner ephemeral-devlab-sm-01-02-r9700s-3-133738799824, AMD Radeon AI PRO R9700S (gfx1201), Linux; hrx-loom 0.2.0 (hrx-system commit unknown, recovered).
  • ⁢ named-machine run, LEVEL=dev, 2026-11-17; device AMD Ryzen 7 9800X3D 8-Core Processor, AMD Radeon RX 9070 XT, Ubuntu 26.04; hrx-loom 0.2.0 (hrx-system 0cd370f).
  • Gates are the loom-benchmark correctness gates (gates.sh); a gate without a recorded run shows no evidence.

Files

  • model.json / model.json.lock: the card and the pinned source files (sha256 checked by loom-fetch).
  • program/*.loom, weights.loom: the generated Loom programs.
  • tokenizer.json: the tokenizer.
  • llm_config.json: the configuration the programs were generated from.
  • info.json: the measurements, gate evidence and versions this card shows; everything else comes from the files above.
  • REPORT.md: how the values above were measured: setup, commands and the value of every run.

Versions: card schema loom-card/3; loom-skills unknown; hrx-loom unknown; card generator b931191.

Source files, fetched by loom-fetch from their own repositories:

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for loom-fun-staging/bartowski-Qwen2.5-7B-Instruct-loom

Base model

Qwen/Qwen2.5-7B
Finetuned
(2)
this model