Instructions to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF
# Run inference directly in the terminal:
llama cli -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF
# Run inference directly in the terminal:
llama cli -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF
# Run inference directly in the terminal:
./build/bin/llama cli -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Use Docker

docker model run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Llama new
LM Studio
Jan

vLLM

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Ollama
How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with Ollama:
```
ollama run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF
```
Unsloth Desktop

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with Docker Model Runner:
```
docker model run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF
```

Lemonade

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Run and chat with the model

lemonade run user.KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

IsValorum commited on Sep 20

Commit

bb19ce8

verified ·

1 Parent(s): ef45431

Add third-party benchmark (zephel01), sampling parameters, and VRAM scaling

Browse files

Files changed (1) hide show

README.md +5 -5

README.md CHANGED Viewed

@@ -61,11 +61,9 @@ quantized_by: IsValorum
 - [Comparative Quantization Analysis (vs. Flat Quants & Generic APEX)](#comparative-analysis)
 - [Everyday Laptop Benchmarks (DDR4 / DDR5 RAM)](#laptop-benchmarks)
 - [The 24GB Miracle: Full 256K Context Runs In VRAM!](#context-scaling)
-- [Hardware Throughput Projections (RTX 30 / 40 / 50)](#throughput-projections)
-- [The Speed vs. Precision Trade-off](#speed-vs-precision)
-- [Surgical Tensor Quantization Map](#tensor-map)
-- [Recommended Configuration & Setup](#recommended-setup)
 ---
 <a id="model-specifications"></a>
@@ -152,6 +150,7 @@ When running with full GPU offload (`-ngl 99`), KAT-Coder's fine-grained MoE arc
 | **NVIDIA RTX 4080 / 5070 (16GB)** | Partial offload (approx. 30 layers) | **35 – 45+ tok/s** | **800 – 1,200+ tok/s** | High-efficiency local coding assistant |
 | **Consumer Laptop (4GB GPU + 32GB RAM)**| Hybrid Offload | **20 – 24+ tok/s** | **300 – 450+ tok/s** | Smooth streaming from system DDR4/DDR5 RAM |
 ### ⚙️ Recommended Generation Parameters (Kwaipilot Official)
 Official sampling hyperparameters specified by [Kwaipilot](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) across benchmark evaluation tracks:
@@ -163,6 +162,7 @@ Official sampling hyperparameters specified by [Kwaipilot](https://huggingface.c
 | **PinchBench & SciCode Scientific Logic** | `0.60 – 0.70` | `1.00` | `20` | `1.50` | `32,768` | `preserve_thinking: true` |
 > [!IMPORTANT]
 > ### 🔍 Model Inherent Behavior vs. Quantization Fidelity Notice
 > Any behavioral limitations, stylistic habits, or syntactic oversights (such as occasional omitted standard library imports in automated zero-shot scripts, e.g., missing `from decimal import ROUND_HALF_UP` or `@dataclass`) **stem entirely from the original unquantized base model weights and pre-training distribution, NOT from the APEX-I quantization process.**
 > Handcrafted APEX-I-MiniPlus strictly preserves mathematical tensor fidelity—keeping 100% of expert routing matrices (`gate_inp`) in uncompressed `F32` (zero router drift), armoring the token output head in `Q6_K`, and safeguarding attention gates in `Q8_0`. Empirical verification confirms near-zero perplexity loss (ΔPPL ≈ +0.06), ensuring that token logits, routing decisions, and reasoning trajectories are mathematically faithful to the original base model.

 - [Comparative Quantization Analysis (vs. Flat Quants & Generic APEX)](#comparative-analysis)
 - [Everyday Laptop Benchmarks (DDR4 / DDR5 RAM)](#laptop-benchmarks)
 - [The 24GB Miracle: Full 256K Context Runs In VRAM!](#context-scaling)
+- [Hardware Throughput & Offload Benchmarks (RTX 30 / 40 / 50)](#throughput-projections)
+- [Recommended Generation Parameters (Creator Official)](#generation-parameters)
+- [Model Inherent Behavior vs. Quantization Fidelity Notice](#quantization-fidelity)
 ---
 <a id="model-specifications"></a>
 | **NVIDIA RTX 4080 / 5070 (16GB)** | Partial offload (approx. 30 layers) | **35 – 45+ tok/s** | **800 – 1,200+ tok/s** | High-efficiency local coding assistant |
 | **Consumer Laptop (4GB GPU + 32GB RAM)**| Hybrid Offload | **20 – 24+ tok/s** | **300 – 450+ tok/s** | Smooth streaming from system DDR4/DDR5 RAM |
+<a id="generation-parameters"></a>
 ### ⚙️ Recommended Generation Parameters (Kwaipilot Official)
 Official sampling hyperparameters specified by [Kwaipilot](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) across benchmark evaluation tracks:
 | **PinchBench & SciCode Scientific Logic** | `0.60 – 0.70` | `1.00` | `20` | `1.50` | `32,768` | `preserve_thinking: true` |
 > [!IMPORTANT]
+> <a id="quantization-fidelity"></a>
 > ### 🔍 Model Inherent Behavior vs. Quantization Fidelity Notice
 > Any behavioral limitations, stylistic habits, or syntactic oversights (such as occasional omitted standard library imports in automated zero-shot scripts, e.g., missing `from decimal import ROUND_HALF_UP` or `@dataclass`) **stem entirely from the original unquantized base model weights and pre-training distribution, NOT from the APEX-I quantization process.**
 > Handcrafted APEX-I-MiniPlus strictly preserves mathematical tensor fidelity—keeping 100% of expert routing matrices (`gate_inp`) in uncompressed `F32` (zero router drift), armoring the token output head in `Q6_K`, and safeguarding attention gates in `Q8_0`. Empirical verification confirms near-zero perplexity loss (ΔPPL ≈ +0.06), ensuring that token logits, routing decisions, and reasoning trajectories are mathematically faithful to the original base model.