Qwen2.5-0.5B-Instruct (Q4_K_M) for AMD Loom

A Loom package (.loom programs and a model.json card) for the unmodified GGUF checkpoint bartowski/Qwen2.5-0.5B-Instruct-GGUF (Qwen2.5-0.5B-Instruct-Q4_K_M.gguf), for the AMD Ryzen AI Max+ 395 "Strix Halo" and AMD Radeon AI PRO R9700 GPUs, generated with the loom-skills workflow. Learn more on the Loom ecosystem website.

Quick start

pipx install "hrx-loom[hf]==0.1.0"

Fetch and serve on the AMD Ryzen AI Max+ 395 "Strix Halo" (gfx1151):

loom-fetch https://huggingface.co/loom-fun/bartowski-Qwen2.5-0.5B-Instruct-loom/resolve/main/model.json --preset=chat --materialize=copy --output=Qwen2.5-0.5B-Instruct
loom-serve --model=Qwen2.5-0.5B-Instruct/model.json '--device=amdgpu://?completion_spin_ns=4294967295' --memory_bytes=5000000000

Fetch and serve on the AMD Radeon AI PRO R9700 (gfx1201, 32 GB):

loom-fetch https://huggingface.co/loom-fun/bartowski-Qwen2.5-0.5B-Instruct-loom/resolve/main/model.json --preset=chat-r9700 --materialize=copy --output=Qwen2.5-0.5B-Instruct
loom-serve --model=Qwen2.5-0.5B-Instruct/model.json '--device=amdgpu://?completion_spin_ns=4294967295' --pool_backing=fixed

The server listens on 127.0.0.1:8080 and answers OpenAI-style /v1/chat/completions requests.

On the AMD Radeon AI PRO R9700, the Quick start fetches the chat-r9700 preset of model.json: the default preset with prefill_capacity 4,096 for its 32 GB. The programs, weights and tokenizer are the same.

Performance vs llama.cpp

Strix Halo (gfx1151)

metric llama.cpp Loom margin roofline Loom % of roofline llama.cpp Vulkan
decode tok/s, short prompt 293.4 311.5 +6 % 551.2 57 % 324.1
decode tok/s, 2k context 279.1 273.6 -2 % 518.7 53 % 299.8
decode tok/s, 32k context 167.1 170.1 +2 % 275.1 62 % 181.0
prefill tok/s, 1k document 15164.4 18074.2 +19 % 77768.6 23 % 12499.0
prefill tok/s, 2k document 15666.3 19036.7 +22 % 73596.4 26 % 12411.4
prefill tok/s, 8k document 13328.6 15245.4 +14 % 55675.2 27 % 8856.4
prefill tok/s, 16k document 9894.8 12420.9 +26 % 42029.3 30 % 6004.3
prefill tok/s, 32k document 6501.2 8667.6 +33 % 28203.9 31 % 3610.7
4 streams, aggregate decode tok/s 872.1 1232.3 +41 % 2050.5 60 % 943.1
speculative decoding, drafted tokens accepted n/a n/a n/a n/a n/a n/a

Methodology

  • Compared: Loom against llama.cpp ROCm llamacpp-rocm/b1374, llama.cpp Vulkan llamacpp-vulkan/b12211; the same GGUF, the same DevLab runner and job, the same prompts; Loom served with the active wait. Values are the mean of 3 rounds (4 streams: median of 2); Loom is timed by the client, llama.cpp by its server.
  • Out of the box on both engines: Loom serves the preset shipped in this package; llama.cpp runs its build defaults plus only the server arguments listed below (offload, context and slots). No tuning sweep on either engine.
  • Drafting, 1-stream rows (decode, decode_2k, decode_32k, prefill): no drafting on either engine (no MTP head in the GGUF and no drafter published for this exact model). Each engine is loaded once for all these rows.
  • 4 streams: both engines loaded with drafting off, 4 parallel requests, aggregate tokens per second.
  • Rows: decode = short-context decode; decode_2k / decode_32k = decode after a ~2k / ~32k-token context; prefill_Nk = prompt processing of an N k-token prompt; streams4 = 4-stream aggregate decode. A margin within +-3 % (the measured same-runner noise) is a tie.
  • llama.cpp ROCm server arguments: 1-stream rows: -ngl 99 -cram 0 -c 65536 -np 1; 4 streams: -ngl 99 -cram 0 -c 65536 -kvu -np 4.
  • llama.cpp Vulkan server arguments: 1-stream rows: -ngl 99 -cram 0 -c 36864 -np 1; 4 streams: -ngl 99 -cram 0 -c 147456 -np 4.
  • Measured on AMD DevLab Dispatch Strix Halo runners (devlab-perf jobs); runners differ between sessions.
  • Measured 2026-11-13; device AMD Strix Halo (DevLab Dispatch runner X2-LA03-S03-D03), Linux; hrx-loom 0.1.0 (hrx-system d01d1c1, recovered); source: DevLab Dispatch run 43392598780.
  • Strix Halo performance measured on programs 73d0a49; the default path of this package (sha256 7042cd0) is identical after canonicalization (fleet/stage_equiv.py at d2be0384d7 against the measured revision 21391f4839; storage, prepare and control differ only inside the logits blocks; checked 2026-11-15).
  • Strix Halo re-measured on programs 73d0a49 on 2026-11-15: within the per-row bands (methodology, 2-4 %) of the published values except streams4 (faster, confirmed in 2 runs; new values shown) (DevLab Dispatch run 43649821117, DevLab Dispatch run 43659277905).
  • streams4: the card value is the mean of the two re-measure runs' medians (each the median of 3 sessions; streams4 1202.8 and 1261.8); the published value was streams4 1134.6; llama.cpp values are as published (not re-measured).
  • llama.cpp Vulkan: measured 2026-11-13; source: DevLab Dispatch run 43392598780. Archive sha256 9759acd.
  • Roofline: the hardware ceiling for this GGUF on AMD Ryzen AI Max+ 395 "Strix Halo" (gfx1151) at 216 GB/s and 59 TFLOP/s, from loom-roofline roofline.py (loom-skills acd9ac7).
  • β€” not measured; n/a not applicable.

Radeon AI PRO R9700 (gfx1201, 32 GB)

metric llama.cpp Loom margin roofline Loom % of roofline llama.cpp Vulkan
decode tok/s, short prompt 390.2 646.4 +66 % 1564.3 41 % 529.4
decode tok/s, 2k context 364.5 516.2 +42 % 1472.0 35 % 492.6
decode tok/s, 32k context 257.2 333.5 +30 % 780.8 43 % 351.2
prefill tok/s, 1k document 32118.6 44215.7 +38 % 282075.9 16 % 26574.5
prefill tok/s, 2k document 34742.8 51005.1 +47 % 266943.1 19 % 28701.3
prefill tok/s, 8k document 31706.8 44704.4 +41 % 201940.7 22 % 25259.2
prefill tok/s, 16k document 25408.3 36158.4 +42 % 152445.4 24 % 19075.0
prefill tok/s, 32k document 17812.2 26021.5 +46 % 102298.9 25 % 12534.5
4 streams, aggregate decode tok/s 1005.4 2273.9 +126 % 5819.4 39 % 1040.6
speculative decoding, drafted tokens accepted n/a n/a n/a n/a n/a n/a

Methodology

  • Compared: Loom against llama.cpp ROCm llamacpp-rocm/b1375, llama.cpp Vulkan llamacpp-vulkan/b12211; the same GGUF, the same DevLab runner and job, the same prompts; Loom served with the active wait. Values are the median of 6 sessions (a session's value is the median of its samples: 3 samples for decode_2k, decode_32k, prefill_8k; one sample for the others) (4 streams: median of 12); Loom is timed by the client, llama.cpp by its server.
  • Out of the box on both engines: Loom serves the preset shipped in this package; llama.cpp runs its build defaults plus only the server arguments listed below (offload, context and slots). No tuning sweep on either engine.
  • Drafting, 1-stream rows (decode, decode_2k, decode_32k, prefill): no drafting on either engine (no MTP head in the GGUF and no drafter published for this exact model). Each engine is loaded once for all these rows.
  • 4 streams: both engines loaded with drafting off, 4 parallel requests, aggregate tokens per second.
  • Rows: decode = short-context decode; decode_2k / decode_32k = decode after a ~2k / ~32k-token context; prefill_Nk = prompt processing of an N k-token prompt; streams4 = 4-stream aggregate decode. A margin within +-3 % (the measured same-runner noise) is a tie.
  • llama.cpp ROCm server arguments: -ngl 99 -cram 0 -c 65536 -np 1.
  • llama.cpp Vulkan server arguments: -ngl 99 -cram 0 -c 36864 -np 1.
  • Measured on AMD DevLab Dispatch R9700 runners (devlab-perf jobs); runners differ between sessions.
  • Measured 2026-11-14; device DevLab Dispatch runner ephemeral-devlab-sm-01-02-r9700s-6-132564945487, AMD Radeon AI PRO R9700S (gfx1201), Linux; hrx-loom 0.1.0 (hrx-system commit unknown, recovered); source: DevLab Dispatch run 43633449157.
  • Radeon AI PRO R9700 performance measured on programs 73d0a49; the default path of this package (sha256 7042cd0) is identical after canonicalization (fleet/stage_equiv.py at d2be0384d7 against the measured revision 21391f4839; storage, prepare and control differ only inside the logits blocks; checked 2026-11-15).
  • llama.cpp Vulkan: measured 2026-11-14; source: DevLab Dispatch run 43633449157. Archive sha256 9759acd.
  • Roofline: the hardware ceiling for this GGUF on AMD Radeon AI PRO R9700S (gfx1201, 32 GB, 2.35 GHz) at 613 GB/s and 214 TFLOP/s, from loom-roofline roofline.py (loom-skills ba84a7a).
  • β€” not measured; n/a not applicable.

Features

feature status detail
API βœ… OpenAI-style /v1/chat/completions
tool calling βœ… OpenAI-style tools and tool_calls
reasoning / thinking β€” not applicable
reasoning effort β€” not applicable
speculative decoding β€” not applicable
image input β€” not applicable
audio input / output β€” not applicable
sampling βœ… temperature, top_p, top_k, seed
context size βœ… 32,768 tokens per request
concurrency βœ… 4 concurrent requests
max output tokens βœ… 4,096 (preset max_tokens)

Correctness

gate description Strix Halo Radeon AI PRO R9700
kernel_tests Unit tests of the generated kernels against reference implementations βœ… pass (56/56) ΒΉ βœ… pass (56/56) Β²
golden llama.cpp reference recorded on the same GGUF: greedy tokens and top-5 log-probs, short prompts and 1k-32k documents βœ… pass Β³ βœ… pass ⁴
compare Greedy output vs the llama.cpp reference, token by token; every divergence is a near-tie βœ… pass Β³ βœ… pass ⁴
vision Image input vs llama.cpp with the same projector: identical prompt tokens, replies equal up to near-ties β€” not applicable β€” not applicable
consistency Concurrent requests give the same output as the same requests run alone βœ… pass Β³ βœ… pass ⁴
spec Speculative decoding on gives exactly the output of speculative decoding off β€” not applicable β€” not applicable
agent Multi-turn tool-calling loop: exact prompt rendering, history reuse, valid tool calls, output vs llama.cpp βœ… pass Β³ βœ… pass ⁴
agent_think_off The tool-calling loop with thinking off β€” not applicable β€” not applicable
sampling temperature, top_k and seed behave as specified; first-token distribution matches llama.cpp βœ… pass Β³ βœ… pass ⁴
logits Host logits readback (llm.logits on): at T=0 every returned row's argmax is the device's token, the top-k matches llama.cpp, greedy text is unchanged βœ… pass (argmax = device token (3 EOG steps); top-5 = llama.cpp's) ⁡ βœ… pass (argmax = device token (3 EOG steps); top-5 = llama.cpp's) ⁢
decision Decision answers (choice, score, noul probabilities) vs llama.cpp on the same prompts β€” not applicable β€” not applicable
  • ΒΉ DevLab Dispatch run 43659277905, 2026-11-15; device AMD Strix Halo (DevLab Dispatch runner X2-LA03-S01-D01), Linux; hrx-loom 0.1.0 (hrx-system d01d1c1, recovered).
  • Β² DevLab Dispatch run 43638050231, 2026-11-14; device DevLab Dispatch runner ephemeral-devlab-sm-01-02-r9700s-0-132564945445, AMD Radeon AI PRO R9700S (gfx1201), Linux; hrx-loom 0.1.0 (hrx-system commit unknown, recovered).
  • Β³ DevLab Dispatch run 43659277905, LEVEL=dev, 2026-11-15; device AMD Strix Halo (DevLab Dispatch runner X2-LA03-S01-D01), Linux; hrx-loom 0.1.0 (hrx-system d01d1c1, recovered).
  • ⁴ DevLab Dispatch run 43638050231, LEVEL=dev, 2026-11-14; device DevLab Dispatch runner ephemeral-devlab-sm-01-02-r9700s-0-132564945445, AMD Radeon AI PRO R9700S (gfx1201), Linux; hrx-loom 0.1.0 (hrx-system commit unknown, recovered).
  • ⁡ DevLab Dispatch run 43721362070, LEVEL=dev, 2026-11-15; device AMD Strix Halo (DevLab Dispatch runner X2-LA03-S01-D05), Linux; hrx-loom 0.1.0 (hrx-system d01d1c1, recovered).
  • ⁢ DevLab Dispatch run 43721368522, LEVEL=dev, 2026-11-15; device DevLab Dispatch runner ephemeral-devlab-sm-01-02-r9700s-0-132835931069, AMD Radeon AI PRO R9700S (gfx1201), Linux; hrx-loom 0.1.0 (hrx-system commit unknown, recovered).
  • Gates are the loom-benchmark correctness gates (gates.sh); a gate without a recorded run shows no evidence.

Files

  • model.json / model.json.lock: the card and the pinned source files (sha256 checked by loom-fetch).
  • program/*.loom, weights.loom: the generated Loom programs.
  • tokenizer.json: the tokenizer.
  • llm_config.json: the configuration the programs were generated from.
  • info.json: the measurements, gate evidence and versions this card shows; everything else comes from the files above.
  • REPORT.md: how the values above were measured: setup, commands and the value of every run.

Versions: card schema loom-card/3; loom-skills d2be038; hrx-loom 0.1.0 (install fingerprint 4358bf6e19ba3594); card generator d2be038.

Source files, fetched by loom-fetch from their own repositories:

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for loom-fun/bartowski-Qwen2.5-0.5B-Instruct-loom

Finetuned
(2)
this model