loom-fun/ggml-org-Flint-loom

Flint (Q4_K_M) for AMD Loom

A Loom package (.loom programs and a model.json card) for the unmodified GGUF checkpoint ggml-org/Flint-GGUF (Flint-Q4_K_M.gguf), generated on an AMD Ryzen AI Max+ 395 "Strix Halo" with the loom-skills workflow.

Quick start

pip install "hrx-loom[hf]"
loom-fetch https://huggingface.co/loom-fun/ggml-org-Flint-loom/resolve/main/model.json --materialize=copy --output=flint
loom-serve --model=flint/model.json --device=amdgpu --memory_bytes=69000000000

The server listens on 127.0.0.1:8080 and answers OpenAI-style /v1/chat/completions requests.

Performance vs llama.cpp

metric	llama.cpp	Loom	margin	roofline	Loom % of roofline	DevLab llama.cpp	DevLab Loom	local ÷ DevLab (llama.cpp, Loom)
decode tok/s, short prompt	60.7	99.0	+63 %	145.7	68 %	78.3	98.6 ¹	0.78, 1.00
decode tok/s, 2k context	58.7	71.1	+21 %	122.6	58 %	60.9	70.2 ¹	0.96, 1.01
decode tok/s, 32k context	52.7	45.9	-13 %	101.6	45 %	49.0	50.6 ¹	1.08, 0.91
prefill tok/s, 1k document	1027.5	1877.9	+83 %	12248.4	15 %	1171.9	2204.7 ¹	0.88, 0.85
prefill tok/s, 2k document	1029.5	2562.6	+149 %	11945.5	21 %	1142.8	2612.4 ¹	0.90, 0.98
prefill tok/s, 8k document	1057.9	2613.2	+147 %	10484.7	25 %	882.9	2607.2 ¹	1.20, 1.00
prefill tok/s, 16k document	903.0	2347.9	+160 %	9014.9	26 %	752.5	2312.6 ¹	1.20, 1.02
prefill tok/s, 32k document	639.7	1872.9	+193 %	7040.8	27 %	603.8	1878.7 ¹	1.06, 1.00
4 streams, aggregate decode tok/s	125.5	133.2	+6 %	223.7	60 %	137.1	117.8 ¹	0.92, 1.13
speculative decoding, drafted tokens accepted	—	64 % ²	—	n/a	n/a	n/a	n/a	n/a

• llama.cpp: build llamacpp-rocm/b1355, server arguments -ngl 99 -cram 0 --spec-type draft-dflash --spec-draft-model ~/.cache/huggingface/hub/models--incoai--Flint-DFlash2-GGUF/snapshots/74e6ac0c538cfd393d97bd93694e5c4c6b6aa8c3/Flint-DFlash2-Q4_K_M.gguf --spec-draft-n-max 3 -ngld 99 -c 65536 -kvu -np 4.
• Loom: speculative decoding on (DFlash, depth 3).
• Same GGUF on both engines, tokens per second. Loom: client-side timing; llama.cpp: server-side timings. Runs per value: decode and prefill 2 (mean), 4 streams 3 (median).
• Measured 2026-11-08; device AMD RYZEN AI MAX+ 395 w/ Radeon 8060S, AMD Radeon 8060S Graphics, Ubuntu 26.04; hrx-loom 0.1.0 (hrx-system d01d1c1); source: local run.
• ¹ Measured 2026-11-08; device not recorded; hrx-loom unknown; source: DevLab Dispatch run 42503493195.
• ² Measured 2026-11-08; device AMD RYZEN AI MAX+ 395 w/ Radeon 8060S, AMD Radeon 8060S Graphics, Ubuntu 24.04; hrx-loom 0.1.0 (hrx-system d01d1c1); source: DevLab Dispatch run 42507398484.
• DevLab: the same measurement on an AMD DevLab Dispatch Strix Halo runner; local ÷ DevLab is the local value over the DevLab value of each engine, recorded for comparison, not a verdict.
• Drafted tokens accepted: the share of the drafter's proposed tokens the model accepted during the spec gate's requests.
• Roofline: the hardware ceiling for this GGUF on AMD Ryzen AI Max+ 395 "Strix Halo" (gfx1151) at 216 GB/s and 59 TFLOP/s, from loom-roofline roofline.py (loom-skills ecb661b); decode and 4 streams with speculative decoding: verify passes × (1 + acceptance × depth 3) tokens per pass, the drafter's cost not counted; acceptance measured in the row's own session: decode 50 %, decode at 2k 38 %, decode at 32k 40 %, 4 streams 42 %.
• — not measured; n/a not applicable.

Features

feature	status	detail
API	✅	OpenAI-style /v1/chat/completions
tool calling	✅	OpenAI-style tools and tool_calls
reasoning / thinking	✅	always on
reasoning effort	❌	not supported
speculative decoding	✅	DFlash, depth 3, on by default; drafter incoai/Flint-DFlash2-GGUF
image input	❌	not supported
audio input / output	❌	not supported
sampling	✅	temperature, top_p, top_k, seed
context size	✅	262,144 tokens per request
concurrency	✅	4 concurrent requests
max output tokens	✅	4,096 (preset max_tokens)

Correctness

gate	description	result
kernel_tests	Unit tests of the generated kernels against reference implementations	✅ pass (51/51) ¹
golden	llama.cpp reference recorded on the same GGUF: greedy tokens and top-5 log-probs, short prompts and 1k-32k documents	✅ pass ²
compare	Greedy output vs the llama.cpp reference, token by token; every divergence is a near-tie	✅ pass ²
vision	Image input vs llama.cpp with the same projector: identical prompt tokens, replies equal up to near-ties	— not applicable
consistency	Concurrent requests give the same output as the same requests run alone	✅ pass ²
spec	Speculative decoding on gives exactly the output of speculative decoding off	✅ pass ²
agent	Multi-turn tool-calling loop: exact prompt rendering, history reuse, valid tool calls, output vs llama.cpp	✅ pass ²
agent_think_off	The tool-calling loop with thinking off	— not applicable
sampling	temperature, top_k and seed behave as specified; first-token distribution matches llama.cpp	✅ pass ²
needle	Needle-in-a-haystack prompt at up to half the context, compared with llama.cpp; over-capacity prompt refused	✅ pass ³
needle_full	Needle-in-a-haystack prompt at the full context is served	✅ pass ²
decision	Decision answers (choice, score, noul probabilities) vs llama.cpp on the same prompts	— not applicable

• ¹ local run, 2026-11-08; device AMD RYZEN AI MAX+ 395 w/ Radeon 8060S, AMD Radeon 8060S Graphics, Ubuntu 26.04; hrx-loom 0.1.0 (hrx-system d01d1c1).
• ² DevLab Dispatch run 42507398484, LEVEL=full, 2026-11-08; device AMD RYZEN AI MAX+ 395 w/ Radeon 8060S, AMD Radeon 8060S Graphics, Ubuntu 24.04; hrx-loom 0.1.0 (hrx-system d01d1c1).
• ³ DevLab Dispatch run 42342529061, LEVEL=quick, 2026-11-07; device AMD RYZEN AI MAX+ 395 w/ Radeon 8060S, AMD Radeon 8060S Graphics, Ubuntu 24.04; hrx-loom 0.1.0 (hrx-system d01d1c1).
• Gates are the loom-benchmark correctness gates (gates.sh); a gate without a recorded run shows no evidence.

Files

• model.json / model.json.lock: the card and the pinned source files (sha256 checked by loom-fetch).
• program/*.loom, weights.loom: the generated Loom programs.
• tokenizer.json: the tokenizer.
• llm_config.json: the configuration the programs were generated from.
• info.json: the measurements, gate evidence and versions this card shows; everything else comes from the files above.
• REPORT.md: how the values above were measured: setup, commands and the value of every run.

Versions: card schema loom-card/2; loom-skills ca8e16b; hrx-loom 0.1.0 (hrx-system d01d1c1); card generator f113cd3.

Source files, fetched by loom-fetch from their own repositories:

• checkpoint: ggml-org/Flint-GGUF @ 468307a, Flint-Q4_K_M.gguf
• drafter: incoai/Flint-DFlash2-GGUF @ 74e6ac0, Flint-DFlash2-Q4_K_M.gguf

Next steps

• prefill_1k sits at about 15% of roofline locally (the guardrail is 16.5%), while running 1.9x faster than llama.cpp on DevLab (2205 vs 1172 tok/s).
• Measured breakdown of a 1k prefill (DevLab, doubling each kernel group): dense Q8 GEMMs about 24%, expert gate/up 22%, expert down 17%, attention 11%, router, bucket and combine 8%.
• Tried and ruled out: a 32-row expert tile (prefill_1k +1.4%, 2k -6%, 8k-32k -10%); prefill chunk size and padding (--prefill 1024 or 2048: prefill_1k unchanged, longer prompts 1-18% slower); int8 WMMA directly on the Q4_K codes (works and passes the gates, but 1.6x slower: iu8 has the same rate as f16 on RDNA3 and the per-32-group epilogue costs more than the saved dequant).
• Remaining ideas: a cheaper dense Q8 GEMM path, or fused expert dequant.