Instructions to use KorshLAB/KorshAI-4B-v5 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use KorshLAB/KorshAI-4B-v5 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf KorshLAB/KorshAI-4B-v5:Q4_K_M # Run inference directly in the terminal: llama cli -hf KorshLAB/KorshAI-4B-v5:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf KorshLAB/KorshAI-4B-v5:Q4_K_M # Run inference directly in the terminal: llama cli -hf KorshLAB/KorshAI-4B-v5:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf KorshLAB/KorshAI-4B-v5:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf KorshLAB/KorshAI-4B-v5:Q4_K_M
Use Docker
docker model run hf.co/KorshLAB/KorshAI-4B-v5:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use KorshLAB/KorshAI-4B-v5 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KorshLAB/KorshAI-4B-v5" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KorshLAB/KorshAI-4B-v5", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/KorshLAB/KorshAI-4B-v5:Q4_K_M
- Ollama
How to use KorshLAB/KorshAI-4B-v5 with Ollama:
ollama run hf.co/KorshLAB/KorshAI-4B-v5:Q4_K_M
- Unsloth Desktop
- Pi
How to use KorshLAB/KorshAI-4B-v5 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KorshLAB/KorshAI-4B-v5:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "KorshLAB/KorshAI-4B-v5:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use KorshLAB/KorshAI-4B-v5 with Docker Model Runner:
docker model run hf.co/KorshLAB/KorshAI-4B-v5:Q4_K_M
- Lemonade
How to use KorshLAB/KorshAI-4B-v5 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull KorshLAB/KorshAI-4B-v5:Q4_K_M
Run and chat with the model
lemonade run user.KorshAI-4B-v5-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use KorshLAB/KorshAI-4B-v5 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KorshLAB/KorshAI-4B-v5:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default KorshLAB/KorshAI-4B-v5:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use KorshLAB/KorshAI-4B-v5 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KorshLAB/KorshAI-4B-v5:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "KorshLAB/KorshAI-4B-v5:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
KorshAI-4B-v5 (GGUF)
KorshAI-4B-v5 is a proprietary, state-of-the-art 4-billion parameter compact foundation coding and systems reasoning model architected, trained, and verified by KorshLab and the Korsh Foundation.
Engineered specifically for low-level systems programming, high-reliability web architectures, and formal multi-language verification, KorshAI-4B-v5 delivers enterprise-grade code generation speed and compiler-grade precision on edge workstations, consumer GPUs, and local CPU servers.
๐ Key Highlights
- Proprietary Compact Architecture: Native 4B parameter dense architecture tailored for algorithmic reasoning, zero-simulation code synthesis, and deep compiler syntax fidelity.
- Polyglot Systems Mastery: Specialized in low-level systems engineering (C11, C++20, Rust), type-safe application logic (TypeScript, Python 3.12+), backend services (PHP 8.2+), and modern web platforms.
- Strict Multi-Tier Verification: Every synthetic training unit was vetted using the KorshLab multi-tier sandbox compiler gate with zero-tolerance flags (
-Wall -Wextra -Werror -pedantic). - Optimal imatrix Quantization: Quantized with custom importance matrices (imatrix) computed against polyglot systems corpora, preserving fp16 perplexity with a lean memory footprint.
- High-Throughput Edge Inference: Fully optimized for
llama.cppAVX2/AVX-512 CPU execution and CUDA/Metal acceleration, sustaining high generation velocity even on entry-level edge servers.
๐ Benchmark & Empirical Evaluation (v5)
KorshAI-4B-v5 was evaluated under the rigorous KorshLab Evaluation Suite (20 challenging multi-domain tasks evaluated via automated deterministic test runners and compiler validation):
| Category | Evaluation Score | Verification Standard |
|---|---|---|
| JSON Schema & Structured Output | 75.0% | Strict RFC 7159 JSONSchema validation & constraint adherence |
| Core Coding & Algorithms | 68.0% | Multi-language algorithmic problem solving & edge cases |
| Reasoning & Formal Logic | 42.4% | Chain-of-thought deductive steps & invariant analysis |
| C++ Systems & Memory Safety | 38.0% | G++ 12+ C++20 strict compilation (-Wall -Wextra -Werror) |
| Overall Composite Score | 55.85 | Champion Model in KorshAI 4B Lineage |
๐ป Hardware Compatibility Matrix
1. Requirements
| Platform / Tier | Hardware Target | RAM / VRAM Footprint | Expected Speed |
|---|---|---|---|
| CPU Only (Workstation / Edge) | 4-Core x86_64 AVX2 (e.g. Dell PowerEdge T40, Intel Core i5/i7, AMD Ryzen) | 6 GB RAM (active footprint ~2.2 GB) | 5.0 โ 15.0 tokens/s (TG) ~30-50 tok/s (PP) |
| GPU Accelerated (Consumer Desktop) | NVIDIA RTX 3060, 3070, 4060, 4070 (6-8 GB VRAM) | 6 GB โ 8 GB VRAM (active ~2.5 GB) | 60.0 โ 90.0 tokens/s (TG) ~900+ tok/s (PP) |
| Apple Silicon (macOS Metal) | Apple M1 / M2 / M3 / M4 | 8 GB Unified Memory | 40.0 โ 60.0 tokens/s (TG) |
| Low-Power Edge (SBC) | Raspberry Pi 5 / RK3588 (ARM64) | 8 GB LPDDR4X | 6.0 โ 9.0 tokens/s (TG) |
2. Supported Runtimes
llama.cpp(b3000+)ollama(v0.1.30+)LM Studio/Jan.ai/LocalAI/Text Generation WebUI- Python via
llama-cpp-python
๐ฆ Provided Model Artifacts
| File | Format | Quantization | Size | SHA-256 / MD5 | Description |
|---|---|---|---|---|---|
KorshAI-4B-v5-Q4_K_M.gguf |
GGUF | Q4_K_M (imatrix) | 2.69 GB (2,886,966,752 B) | MD5: 2434b5a7afb933372d7bccf1466942d1 |
Primary recommended quantization for production inference. |
๐ Quickstart & Usage
1. CLI Execution with llama.cpp
llama-cli -m KorshAI-4B-v5-Q4_K_M.gguf \
-p "<|im_start|>system\nYou are KorshAI, an expert systems and software engineer.<|im_end|>\n<|im_start|>user\nWrite an idiomatic, zero-allocation circular ring buffer in C++20.<|im_end|>\n<|im_start|>assistant\n" \
-c 4096 \
-ngl 99 \
--temp 0.2
2. Production OpenAI-Compatible HTTP Server
llama-server -m KorshAI-4B-v5-Q4_K_M.gguf \
--host 0.0.0.0 \
--port 8080 \
-c 8192 \
-t 4 -tb 4 \
--flash-attn on \
-ctk q4_0 -ctv q4_0
Query the server:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "KorshAI-4B-v5",
"messages": [
{"role": "system", "content": "You are KorshAI, an expert polyglot engineer."},
{"role": "user", "content": "Implement a thread-safe singleton pattern in modern C++20."}
],
"temperature": 0.2
}'
3. Ollama Integration
Create a file named Modelfile:
FROM ./KorshAI-4B-v5-Q4_K_M.gguf
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
SYSTEM """You are KorshAI, a world-class autonomous software engineering model created by KorshLab and korsh.org."""
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
PARAMETER temperature 0.2
Create and run:
ollama create korshai:4b-v5 -f Modelfile
ollama run korshai:4b-v5
๐๏ธ About Korsh & KorshLab
KorshAI-4B-v5 is developed, benchmarked, and maintained by KorshLab in partnership with the Korsh Foundation.
- ๐ Official Website: https://korsh.org
- ๐ฌ Laboratory: https://korshlab.space
- ๐ฌ Discord Community: Join our official Discord
๐ License & Attribution
This model is licensed under the Apache License, Version 2.0. Copyright ยฉ 2026 KorshLab & Korsh Foundation. All rights reserved.
- Downloads last month
- 56
4-bit