ik_llama.cpp experimental GGUFs

Validation files for open and recent ik_llama.cpp pull requests. They load only with ik_llama.cpp, some only with an open PR. The formats may change, and the files will be replaced if they do. This is not a general-purpose model repository.

File Source Types Needs
Qwen3.8-27B-NVFP4_KS.gguf nvidia/Qwen3.8-27B-NVFP4 NVFP4_KS, Q8_0 ik_llama.cpp main
fp8/Qwen3.8-Flash-Next-NVFP4_KS-F8.gguf nvidia/Qwen3.8-Flash-Next-NVFP4 NVFP4_KS, MTP head F8_E4M3_B128 #2918 (converted with #2919)
fp8/Qwen3.8-27B-NVFP4_KS-F8.gguf nvidia/Qwen3.8-27B-NVFP4 NVFP4_KS, F8_E4M3_KS #2918 (converted with #2919)
fp8/nvidia-Qwen3-8B-FP8-KS.gguf nvidia/Qwen3-8B-FP8 F8_E4M3_KS #2918
fp8/Qwen-Qwen3-8B-FP8-B128.gguf Qwen/Qwen3-8B-FP8 F8_E4M3_B128 #2918
fp8/RedHatAI-Qwen3-30B-A3B-FP8-dynamic-KS.gguf RedHatAI/Qwen3-30B-A3B-FP8-dynamic F8_E4M3_KS #2918
fp8/Qwen-Qwen3-30B-A3B-FP8-B128.gguf Qwen/Qwen3-30B-A3B-FP8 F8_E4M3_B128 #2918
fp8/liodon-ai-LFM2.5-8B-A1B-FP8-KS.gguf liodon-ai/LFM2.5-8B-A1B-FP8 F8_E4M3_KS #2918

FP8 (E4M3) files

sha256:

  • fp8/Qwen3.8-Flash-Next-NVFP4_KS-F8.gguf: dac63a9214b757c2379687fefc172420651607f272ba9ec56e2fbe4ee0b70e8d
  • fp8/nvidia-Qwen3-8B-FP8-KS.gguf: c9c21094725ba18f0649f4025c7349955989129411d9ade2ad2561bb7347c1fb
  • fp8/Qwen3.8-27B-NVFP4_KS-F8.gguf: f6f7761b98ce372989bee940ccfc412b02538b738da1b9df3faa30cd207fd5ea
  • fp8/Qwen-Qwen3-8B-FP8-B128.gguf: 1e86aafc158d419786cc9dbad54dfb4dc0a5026020bd3b0a22c3f53960749179
  • fp8/RedHatAI-Qwen3-30B-A3B-FP8-dynamic-KS.gguf: 1c1db89c2e094fed83d8a6d0174bddd63f2578b5d469ee31c349dd67033859ce
  • fp8/Qwen-Qwen3-30B-A3B-FP8-B128.gguf: 138cb2e9dc36be6c0c8a6a2d3b8ca3bde6554883aef2ba4372c815df72120fc6
  • fp8/liodon-ai-LFM2.5-8B-A1B-FP8-KS.gguf: 6fc683d27e63a376d7f1b90a742939839901dc2485fbc83eabd214a0721e0625

Qwen3.8-27B NVFP4_KS

nvidia/Qwen3.8-27B-NVFP4 at revision 482ca0f, in the NVFP4_KS layout of ik_llama.cpp. The NVFP4 weights are the checkpoint's own, nothing is requantized, and each NVFP4 weight keeps its activation input scale. llama.cpp has no NVFP4_KS type.

How it was made

  1. convert_hf_to_gguf.py with conversion/ from llama.cpp f7591a6, run with --fp8-as-q8. The checkpoint is ModelOpt MIXED_PRECISION, and ik_llama.cpp had no FP8 type, so the FP8 layers are stored as Q8_0. These tensors are Q8_0: attn_qkv, attn_gate and ssm_out in 46 layers, and attn_q, attn_k, attn_v and attn_output in 15 layers.
  2. llama-quantize <mainline file> Qwen3.8-27B-NVFP4_KS.gguf COPY with ik_llama.cpp at f415f91. COPY writes each NVFP4 weight as NVFP4_KS with its rows unchanged and keeps its .input_scale.

The file has 193 NVFP4_KS, 208 Q8_0, 105 BF16 and 553 F32 tensors, and 193 input scales. The MTP layers are not converted.

Use

ik_llama.cpp loads the file with mmap. On RTX 50 and GB10, --nvfp4-precision auto (the default) uses the input scales for 4-bit activations; --nvfp4-precision int8 uses 8-bit activations.

Check

Perplexity on wikitext-2 wiki.test.raw, -c 512 --chunks 8, GB10. With both settings, the file saved by --kl-divergence-base (encoded log-probabilities) is identical to the one from the mainline file in ik_llama.cpp.

--nvfp4-precision PPL
auto 6.7052 ยฑ 0.36538
int8 6.6061 ยฑ 0.35988

License

Each file has the license of its source checkpoint. The Qwen3.8 and Qwen3 files are Apache-2.0 (LICENSE). The LFM2.5 file is under the LFM Open License v1.0 (fp8/LICENSE-LFM).

Downloads last month
977
GGUF
Model size
27B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ji-farthing/ik_llama-experimental

Quantized
(3)
this model