Phi-4 GGUF references

gguf · llama-cpp · quantized · phi4

Four preserved GGUF alternatives for Microsoft Phi-4, prepared by Xylolabs with llama.cpp. Choose the existing Q4 baseline, plain or imatrix IQ4, or Q5 reference.

This export needs about 10.50 GB of device memory for its largest GGUF weight choice alone, plus room for the KV cache and buffers.

Formats

Each choice contains all 243 tensors of the 40-layer fused Phi3-format decoder, embedding and output head, plus the tokenizer and original chat template.

File	Download GB	Average bits/weight
`phi-4-Q4_K_M.gguf`	9.059	4.942
`phi-4-IQ4_XS.gguf`	8.259	4.505
`phi-4-IQ4_XS-imat.gguf`	8.187	4.466
`phi-4-Q5_K_M.gguf`	10.500	5.728

Actual mixed tensor types:

Choice	Tensor types and counts
`Q4_K_M`	F32: 81, Q4_K: 121, Q6_K: 41
`IQ4_XS`	F32: 81, IQ4_XS: 116, Q5_K: 45, Q6_K: 1
`IQ4_XS-imat`	F32: 81, IQ4_XS: 121, Q5_K: 40, Q6_K: 1
`Q5_K_M`	F32: 81, Q5_K: 121, Q6_K: 41

Average bits count all stored parameters, including higher-precision tables and float32 operands. Existing calibration results and original relative metadata labels remain; no dataset is included. No new training, calibration or quantization was performed for publication.

Quickstart

With a llama.cpp build, download one file and run:

llama-cli -m phi-4-Q5_K_M.gguf -c 2048 -n 32 --temp 0 --jinja \
  --single-turn -p "Name the capital of France. Reply with only the city."

This command is checked with llama.cpp v0.9.0. Choose one file at a time. The example uses 2,048 context tokens; original context metadata remains unchanged. KV and working buffers need additional memory.

Evaluation

All four choices pass the relative bar with zero counted losses or broken answers. Raw main-answer totals differ. Stored BF16 and current Q4 answer 29/33, plain IQ4 28/33, imatrix IQ4 27/33 and Q5 30/33.

Choice	Mean KL	Top-1 agreement with BF16	KL / Q4	Right
`Q4_K_M`	0.017991	93.65%	1.000×	29/33
`IQ4_XS`	0.020568	92.62%	1.143×	28/33
`IQ4_XS-imat`	0.019728	93.03%	1.097×	27/33
`Q5_K_M`	0.005521	95.96%	0.307×	30/33

Plain IQ4 loses en-translate-ko in raw correctness; imatrix IQ4 loses that item and en-reason-weekday. They are excluded from counted loss because the original BF16 top-two gap is below 1.0 at the first divergent token. Q5 gains en-reason-ball. These raw differences remain visible above. No reasoning exemption pardons a loss.

The seven thinking-format rechecks score 0/7 for BF16 and every choice. They are inapplicable for this model and grant no recovery exemptions.

The original 16 matched 1,024-token English/Korean plain/chat sequences cover 16,280 positions. All original sequence IDs, tokens, first scored positions, 33+7 prompts, stop IDs and generation budgets match, as does the complete calibration/evaluation fingerprint. The existing raw teacher equals every original stored PT slice without a new BF16 forward. Current llama.cpp GPU checks use context 8,192. Each candidate KL ratio is below 1.5×, and the answer rule permits at most one more counted loss than the Q4 reference.

The exact public Q5 CLI completes one query. These checks cover retained texts and prompts; an integration needs its own task evaluation. Speed is not reported here.

License

MIT, as declared by the pinned Microsoft release. Its full original license is retained in LICENSE, and NOTICE identifies the source and modifications. Provided as is, without warranty or upstream endorsement.