Qwen2.5-Coder-7B-Instruct โ€” Edge-Calibrated imatrix GGUF Suite

Importance-matrix (llama-imatrix) GGUF quantizations of Qwen/Qwen2.5-Coder-7B-Instruct, calibrated specifically for agentic tool-calling, code review, and declarative infrastructure reasoning and benchmarked across 8 GiB NVIDIA Jetson Orin Nano (CUDA 12.6 sm_87) and x86_64 AVX2.

Quality (FP16 Logit KL-Divergence) & Hardware Telemetry

Unlike perplexity-only tables, every quantization below was scored directly against the exact FP16 base model probability distribution (llama-perplexity --kl-divergence), alongside real hardware throughput and INA3221/RAPL power telemetry:

Quant imatrix BPW Size (GiB) Mean KLD (nats) Top-1 Match PPL Orin CUDA tg128 Orin Power / Energy x86 AVX2 tg128
Q8_0 Yes 8.50 7.54 0.00026 99.61% 8.276 OOM (>8 GiB) โ€” 2.2 tok/s
Q6_K Yes 6.57 5.83 0.00121 98.69% 8.278 OOM (>8 GiB) โ€” 2.6 tok/s
Q5_K_M Yes 5.72 5.07 0.00286 97.97% 8.268 OOM (>8 GiB) โ€” 2.0 tok/s
Q4_K_M Yes 4.92 4.36 0.01005 96.08% 8.356 13.0 tok/s 20.4 W (1.570 J/tok) 2.8 tok/s
Q4_K_M-std No (control) 4.92 4.36 0.01321 95.49% 8.281 13.0 tok/s 20.3 W (1.565 J/tok) 3.0 tok/s
IQ4_XS Yes 4.43 3.93 0.01287 95.36% 8.307 16.3 tok/s 20.5 W (1.256 J/tok) 2.4 tok/s
IQ3_M Yes 3.75 3.33 0.04187 90.52% 8.374 11.9 tok/s 20.9 W (1.756 J/tok) 1.1 tok/s

imatrix Ablation (Q4_K_M vs Q4_K_M-std): Calibrating second-moment channel activations on agentic tool-calling + systems code changed Mean KL-Divergence from 0.01321 to 0.01005 (+23.9% KLD reduction at identical byte size).

GQA KV-Cache Sizing Formula (Qwen2.5-Coder-7B-Instruct)

Qwen2.5-Coder-7B-Instruct uses Grouped-Query Attention with 28 layers, 4 KV heads, and head_dim = 128 (56.0 KiB/token in FP16 KV cache, or 448 MiB at 8,192 context). Use the interactive Edge Unified-Memory Sizer to verify exact headroom for your context length and KV quantization (FP16 / Q8_0 / Q4_0).

Reproducing with Nix

nix run github:kleinbem/hf-bench#hf-bench -- run-local --model qwen2.5-coder-7b
nix run github:kleinbem/hf-bench#hf-bench -- bench-orin --model qwen2.5-coder-7b --host orin-nano
Downloads last month
600
GGUF
Model size
8B params
Architecture
qwen2
Hardware compatibility
Log In to add your hardware

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for kleinbem/Qwen2.5-Coder-7B-Instruct-Edge-GGUF

Base model

Qwen/Qwen2.5-7B
Quantized
(287)
this model