Instructions to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF
# Run inference directly in the terminal:
llama cli -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF
# Run inference directly in the terminal:
llama cli -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF
# Run inference directly in the terminal:
./build/bin/llama cli -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Use Docker

docker model run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Llama new
LM Studio
Jan

vLLM

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Ollama
How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with Ollama:
```
ollama run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF
```
Unsloth Desktop

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with Docker Model Runner:
```
docker model run hf.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF
```

Lemonade

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Run and chat with the model

lemonade run user.KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

IsValorum commited on Sep 18

Commit

edfbfe8

verified ·

1 Parent(s): 086644f

docs: add warning clarifying APEX-I-MiniPlus (all versions) vs generic community APEX-I-Mini

Browse files

Files changed (1) hide show

README.md +6 -54

README.md CHANGED Viewed

@@ -48,6 +48,12 @@ quantized_by: IsValorum
 > 👉 To explore or download the **V2.1** edition of KAT-Coder-V2.5-Dev optimized for system RAM streaming, visit:
 > **[IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF](https://huggingface.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF)**
 ---
 ## <a id="quick-navigation"></a>⚡ Quick Navigation Index
@@ -146,57 +152,3 @@ When running with full GPU offload (`-ngl 99`), KAT-Coder's fine-grained MoE arc
 | **NVIDIA RTX 4080 / 5070 (16GB)** | Partial offload (approx. 30 layers) | **35 – 45+ tok/s** | **800 – 1,200+ tok/s** | High-efficiency local coding assistant |
 | **Consumer Laptop (4GB GPU + 32GB RAM)**| Hybrid Offload | **20 – 24+ tok/s** | **300 – 450+ tok/s** | Smooth streaming from system DDR4/DDR5 RAM |
-> [!NOTE]
-> ### 🔬 Empirical Testbed Architecture & Desktop/Server Scaling
-> - **Empirical Benchmark Hardware:** The hybrid offload and system RAM streaming figures documented above were measured on a consumer laptop powered by an **Intel 12th Gen Alder Lake architecture featuring a hybrid design of Performance Cores (P-Cores) and Efficient Cores (E-Cores)** paired with dual-channel system RAM and constrained laptop power/thermal envelopes.
-> - **Thread Scheduling & E-Core Contention:** In hybrid architectures like Alder Lake, OS thread scheduling across background E-Cores and lower single-core mobile power limits introduce memory bandwidth and thread synchronization overhead during CPU dequantization.
-> - **Dramatic Scaling on Higher-End Processors:** When running on desktop or server processors (such as modern AMD Ryzen 7000 / 9000 Zen 4/5 series or high-TDP Intel desktop platforms with dedicated performance cores, large L3 caches, and high-bandwidth dual- or quad-channel DDR5 running at 6000+ MT/s), **streaming generation speeds and prefill throughput will scale dramatically higher**, substantially exceeding these measured mobile numbers.
----
-<a id="speed-vs-precision"></a>
-## ⚖️ The Speed vs. Precision Trade-off
-Why does KAT-Coder settle at 20–24 tok/s on DDR4 laptops while crude flat MoE quants reach 23–26 tok/s?
-- **The Dequantization Cost:** Non-linear `IQ` codebooks require table lookups during CPU dequantization, compared to single-cycle AVX2 instructions used by linear `Q_K` blocks.
-- **Why It Is Essential for KAT-Coder:** In a 256 micro-expert model, linear blocks cause router drift and degrade narrow experts. By trading approx. 2 tok/s on CPU memory, KAT-Coder retains **flawless syntax, intact indentation logic, and zero code hallucination**.
----
-<a id="tensor-map"></a>
-## 🛠️ Surgical Tensor Quantization Map
-| Tensor Pattern | Layer Scope | Quant Type | BPW | Engineering Rationale |
-| :--- | :--- | :--- | :--- | :--- |
-| **`output.weight`** | Vocabulary Head | **`Q6_K`** | 6.56 | Uncompromised 6-bit precision across 248k tokens for exact syntax & operators |
-| **`token_embd.weight`** | Embedding | **`High-Prec`** | High | Preserves semantic token embeddings without imatrix distortion |
-| **`ffn_gate_inp.weight`** | Expert Routers | **`F32`** | 32.0 | Uncompressed full-precision routers preventing micro-expert misrouting |
-| **`attn_gate.weight`** | Attention Gates | **`Q8_0`** | 8.50 | High-precision 8-bit gating for attention routing dynamics |
-| **`ffn_*_shexp`** | Shared Experts | **`IQ4_NL`** | 4.50 | 4-bit non-linear codebook for the 100% active shared foundational expert |
-| **`ffn_down/up/gate`** | Edges (0–9, 30–39) | **`IQ3_S`** | 3.44 | Armored boundary layers protecting prompt ingest and final code generation |
-| **`ffn_down/up/gate`** | Core (10–29) | **`IQ3_XXS`** | 3.06 | Deep compression (272 MiB → 98 MiB per block) calibrated via code imatrix |
-| **`attn_qkv`, `attn_output`** | Attention Projections| **`IQ3_S` / `Q6_K`** | Multi | High-fidelity attention heads preventing contextual cross-talk |
-| **Norms & Biases** | All Layers | **`F32`** | 32.0 | Absolute numerical stability across deep 40-layer computation |
----
-<a id="recommended-setup"></a>
-## 📖 Recommended Configuration & Setup
-### Unsloth Studio:
-1. Load **`KAT-Coder-V2.5-Dev.APEX-I-MiniPlus-V2.gguf`**.
-2. Configure KV Cache Dtype to **`q8_0`** and Context Checkpoints to **`1`**.
-3. Set GPU Offload to **100%** (`-ngl 99`) on 24GB GPUs, or allocate 30–34 layers on 16GB GPUs.
-4. Expand context up to **128k – 256k** for full repository inspection.
-### llama.cpp CLI:
-```bash
-llama-cli -m KAT-Coder-V2.5-Dev.APEX-I-MiniPlus-V2.gguf \
-  -ngl 99 \
-  -c 32768
-```
-### LM Studio / Ollama:
-1. Import the `.gguf` file into your local library.
-2. Maximize GPU acceleration to offload all layers.
-3. Configure template with Qwen 2.5 / 3.5 chat format.

 > 👉 To explore or download the **V2.1** edition of KAT-Coder-V2.5-Dev optimized for system RAM streaming, visit:
 > **[IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF](https://huggingface.co/IsValorum/KAT-Coder-V2.5-Dev-APEX-I-MiniPlus-V2.1-GGUF)**
+> [!WARNING]
+> ### ⚠️ DO NOT CONFUSE APEX-I-MINIPLUS WITH GENERIC COMMUNITY APEX-I-MINI!
+> **Regardless of release version (whether V1, V2, or V2.1), NEVER confuse handcrafted APEX-I-MiniPlus builds with generic community APEX-I-Mini releases:**
+> - **Generic Community APEX-I-Mini:** Uniformly compresses all core MoE experts down to aggressive 2-bit `IQ2_S` (dropping below the critical quality floor), leaves the sensitive token output head unarmored at 3-bit `Q3_K_M`, and compresses attention projections down to `Q3_K`. In deep reasoning models, this triggers severe perplexity spikes, syntax errors, and broken code brackets.
+> - **Handcrafted APEX-I-MiniPlus (All Editions by IsValorum):** Every single MiniPlus release—from V1 and V2 to V2.1—is a custom tensor-by-tensor architecture that preserves uncompressed `F32` router gates, armors the token output head in high-precision `Q6_K`, safeguards attention gates in `Q8_0`, and keeps core reasoning experts at or above calibrated 3-bit (`IQ3_XXS`/`IQ3_S`). Even our earlier builds vastly outperform generic community APEX recipes and flat 3-bit quants.
 ---
 ## <a id="quick-navigation"></a>⚡ Quick Navigation Index
 | **NVIDIA RTX 4080 / 5070 (16GB)** | Partial offload (approx. 30 layers) | **35 – 45+ tok/s** | **800 – 1,200+ tok/s** | High-efficiency local coding assistant |
 | **Consumer Laptop (4GB GPU + 32GB RAM)**| Hybrid Offload | **20 – 24+ tok/s** | **300 – 450+ tok/s** | Smooth streaming from system DDR4/DDR5 RAM |