Instructions to use ji-farthing/ik_llama-experimental with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use ji-farthing/ik_llama-experimental with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf ji-farthing/ik_llama-experimental:NVFP4_KS # Run inference directly in the terminal: llama cli -hf ji-farthing/ik_llama-experimental:NVFP4_KS
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf ji-farthing/ik_llama-experimental:NVFP4_KS # Run inference directly in the terminal: llama cli -hf ji-farthing/ik_llama-experimental:NVFP4_KS
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf ji-farthing/ik_llama-experimental:NVFP4_KS # Run inference directly in the terminal: ./build/bin/llama cli -hf ji-farthing/ik_llama-experimental:NVFP4_KS
Use Docker
docker model run hf.co/ji-farthing/ik_llama-experimental:NVFP4_KS
- Llama new
- LM Studio
- Jan
- Ollama
How to use ji-farthing/ik_llama-experimental with Ollama:
ollama run hf.co/ji-farthing/ik_llama-experimental:NVFP4_KS
- Unsloth Desktop
- Pi
How to use ji-farthing/ik_llama-experimental with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ji-farthing/ik_llama-experimental:NVFP4_KS
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "ji-farthing/ik_llama-experimental:NVFP4_KS" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use ji-farthing/ik_llama-experimental with Docker Model Runner:
docker model run hf.co/ji-farthing/ik_llama-experimental:NVFP4_KS
- Lemonade
How to use ji-farthing/ik_llama-experimental with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull ji-farthing/ik_llama-experimental:NVFP4_KS
Run and chat with the model
lemonade run user.ik_llama-experimental-NVFP4_KS
List all available models
lemonade list
- Hermes Agent
How to use ji-farthing/ik_llama-experimental with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ji-farthing/ik_llama-experimental:NVFP4_KS
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default ji-farthing/ik_llama-experimental:NVFP4_KS
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use ji-farthing/ik_llama-experimental with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf ji-farthing/ik_llama-experimental:NVFP4_KS
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "ji-farthing/ik_llama-experimental:NVFP4_KS" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ji-farthing/ik_llama-experimental:NVFP4_KS# Run inference directly in the terminal:
llama cli -hf ji-farthing/ik_llama-experimental:NVFP4_KSBuild from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf ji-farthing/ik_llama-experimental:NVFP4_KS# Run inference directly in the terminal:
./build/bin/llama cli -hf ji-farthing/ik_llama-experimental:NVFP4_KSUse Docker
docker model run hf.co/ji-farthing/ik_llama-experimental:NVFP4_KSik_llama.cpp experimental GGUFs
Validation files for open and recent ik_llama.cpp pull requests. They load only with ik_llama.cpp, some only with an open PR. The formats may change, and the files will be replaced if they do. This is not a general-purpose model repository.
| File | Source | Types | Needs |
|---|---|---|---|
Qwen3.8-27B-NVFP4_KS.gguf |
nvidia/Qwen3.8-27B-NVFP4 | NVFP4_KS, Q8_0 |
ik_llama.cpp main |
fp8/Qwen3.8-Flash-Next-NVFP4_KS-F8.gguf |
nvidia/Qwen3.8-Flash-Next-NVFP4 | NVFP4_KS, MTP head F8_E4M3_B128 |
#2918 (converted with #2919) |
fp8/Qwen3.8-27B-NVFP4_KS-F8.gguf |
nvidia/Qwen3.8-27B-NVFP4 | NVFP4_KS, F8_E4M3_KS |
#2918 (converted with #2919) |
fp8/nvidia-Qwen3-8B-FP8-KS.gguf |
nvidia/Qwen3-8B-FP8 | F8_E4M3_KS |
#2918 |
fp8/Qwen-Qwen3-8B-FP8-B128.gguf |
Qwen/Qwen3-8B-FP8 | F8_E4M3_B128 |
#2918 |
fp8/RedHatAI-Qwen3-30B-A3B-FP8-dynamic-KS.gguf |
RedHatAI/Qwen3-30B-A3B-FP8-dynamic | F8_E4M3_KS |
#2918 |
fp8/Qwen-Qwen3-30B-A3B-FP8-B128.gguf |
Qwen/Qwen3-30B-A3B-FP8 | F8_E4M3_B128 |
#2918 |
fp8/liodon-ai-LFM2.5-8B-A1B-FP8-KS.gguf |
liodon-ai/LFM2.5-8B-A1B-FP8 | F8_E4M3_KS |
#2918 |
FP8 (E4M3) files
sha256:
fp8/Qwen3.8-Flash-Next-NVFP4_KS-F8.gguf:dac63a9214b757c2379687fefc172420651607f272ba9ec56e2fbe4ee0b70e8dfp8/nvidia-Qwen3-8B-FP8-KS.gguf:c9c21094725ba18f0649f4025c7349955989129411d9ade2ad2561bb7347c1fbfp8/Qwen3.8-27B-NVFP4_KS-F8.gguf:f6f7761b98ce372989bee940ccfc412b02538b738da1b9df3faa30cd207fd5eafp8/Qwen-Qwen3-8B-FP8-B128.gguf:1e86aafc158d419786cc9dbad54dfb4dc0a5026020bd3b0a22c3f53960749179fp8/RedHatAI-Qwen3-30B-A3B-FP8-dynamic-KS.gguf:1c1db89c2e094fed83d8a6d0174bddd63f2578b5d469ee31c349dd67033859cefp8/Qwen-Qwen3-30B-A3B-FP8-B128.gguf:138cb2e9dc36be6c0c8a6a2d3b8ca3bde6554883aef2ba4372c815df72120fc6fp8/liodon-ai-LFM2.5-8B-A1B-FP8-KS.gguf:6fc683d27e63a376d7f1b90a742939839901dc2485fbc83eabd214a0721e0625
Qwen3.8-27B NVFP4_KS
nvidia/Qwen3.8-27B-NVFP4 at revision 482ca0f, in the NVFP4_KS layout of ik_llama.cpp. The NVFP4 weights are the checkpoint's own, nothing is requantized, and each NVFP4 weight keeps its activation input scale. llama.cpp has no NVFP4_KS type.
How it was made
convert_hf_to_gguf.pywithconversion/from llama.cppf7591a6, run with--fp8-as-q8. The checkpoint is ModelOptMIXED_PRECISION, and ik_llama.cpp had no FP8 type, so the FP8 layers are stored as Q8_0. These tensors are Q8_0:attn_qkv,attn_gateandssm_outin 46 layers, andattn_q,attn_k,attn_vandattn_outputin 15 layers.llama-quantize <mainline file> Qwen3.8-27B-NVFP4_KS.gguf COPYwith ik_llama.cpp atf415f91.COPYwrites each NVFP4 weight asNVFP4_KSwith its rows unchanged and keeps its.input_scale.
The file has 193 NVFP4_KS, 208 Q8_0, 105 BF16 and 553 F32 tensors, and 193 input scales. The MTP layers are not converted.
Use
ik_llama.cpp loads the file with mmap. On RTX 50 and GB10, --nvfp4-precision auto (the default) uses the input scales for 4-bit activations; --nvfp4-precision int8 uses 8-bit activations.
Check
Perplexity on wikitext-2 wiki.test.raw, -c 512 --chunks 8, GB10. With both settings, the file saved by --kl-divergence-base (encoded log-probabilities) is identical to the one from the mainline file in ik_llama.cpp.
--nvfp4-precision |
PPL |
|---|---|
auto |
6.7052 ยฑ 0.36538 |
int8 |
6.6061 ยฑ 0.35988 |
License
Each file has the license of its source checkpoint. The Qwen3.8 and Qwen3 files are Apache-2.0 (LICENSE). The LFM2.5 file is under the LFM Open License v1.0 (fp8/LICENSE-LFM).
- Downloads last month
- 1,192
4-bit
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf ji-farthing/ik_llama-experimental:NVFP4_KS# Run inference directly in the terminal: llama cli -hf ji-farthing/ik_llama-experimental:NVFP4_KS