Instructions to use KorshLAB/KorshAI-4B-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use KorshLAB/KorshAI-4B-v2 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf KorshLAB/KorshAI-4B-v2 # Run inference directly in the terminal: llama cli -hf KorshLAB/KorshAI-4B-v2
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf KorshLAB/KorshAI-4B-v2 # Run inference directly in the terminal: llama cli -hf KorshLAB/KorshAI-4B-v2
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf KorshLAB/KorshAI-4B-v2 # Run inference directly in the terminal: ./build/bin/llama cli -hf KorshLAB/KorshAI-4B-v2
Use Docker
docker model run hf.co/KorshLAB/KorshAI-4B-v2
- Llama new
- LM Studio
- Jan
- vLLM
How to use KorshLAB/KorshAI-4B-v2 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KorshLAB/KorshAI-4B-v2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KorshLAB/KorshAI-4B-v2", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/KorshLAB/KorshAI-4B-v2
- Ollama
How to use KorshLAB/KorshAI-4B-v2 with Ollama:
ollama run hf.co/KorshLAB/KorshAI-4B-v2
- Unsloth Desktop
- Pi
How to use KorshLAB/KorshAI-4B-v2 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KorshLAB/KorshAI-4B-v2
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "KorshLAB/KorshAI-4B-v2" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use KorshLAB/KorshAI-4B-v2 with Docker Model Runner:
docker model run hf.co/KorshLAB/KorshAI-4B-v2
- Lemonade
How to use KorshLAB/KorshAI-4B-v2 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull KorshLAB/KorshAI-4B-v2
Run and chat with the model
lemonade run user.KorshAI-4B-v2-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use KorshLAB/KorshAI-4B-v2 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KorshLAB/KorshAI-4B-v2
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default KorshLAB/KorshAI-4B-v2
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use KorshLAB/KorshAI-4B-v2 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KorshLAB/KorshAI-4B-v2
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "KorshLAB/KorshAI-4B-v2" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf KorshLAB/KorshAI-4B-v2# Run inference directly in the terminal:
llama cli -hf KorshLAB/KorshAI-4B-v2Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf KorshLAB/KorshAI-4B-v2# Run inference directly in the terminal:
./build/bin/llama cli -hf KorshLAB/KorshAI-4B-v2Use Docker
docker model run hf.co/KorshLAB/KorshAI-4B-v2
๐ KorshAI-4B-v5 is now available! We strongly recommend the new champion release: KorshLAB/KorshAI-4B-v5 with improved polyglot reasoning and higher benchmark accuracy (Score: 55.85).
KorshAI-4B-v2 (GGUF)
KorshAI-4B-v2 is a proprietary, high-performance 4-billion parameter foundation coding model architected and trained by KorshLab and Korsh.
Engineered specifically for polyglot systems programming, modern web architectures, and rigorous formal code verification, KorshAI-4B-v2 delivers enterprise-grade speed and precision on consumer GPUs, workstations, and edge servers.
๐ Key Highlights
- Proprietary Architecture: Native 4B parameter dense architecture tailored for deep polyglot comprehension and algorithmic reasoning.
- Polyglot Mastery: Specialized in low-level systems programming (C11, C++20), modern web stacks (PHP 8.2+, HTML5, Bootstrap 5), Python 3, and structured JSON Schema generation.
- Strict Verification Protocol: Trained and verified using the KorshLab multi-tier sandbox suite with zero-tolerance compiler flags (
-Wall -Wextra -Werror -pedantic). - Optimal Quantization: Quantized with custom importance matrices (imatrix) computed against multi-language source corpora, preserving fp16 perplexity with minimal memory overhead.
- High-Throughput Inference: Optimized for
llama.cppCPU/GPU acceleration, delivering upwards of 14-85 tokens/s depending on acceleration tier.
๐ป Hardware Compatibility
1. Requirements Matrix
| Tier / Platform | Target Setup | Minimum Specs | Memory (RAM / VRAM) | Expected Inference Speed |
|---|---|---|---|---|
| CPU Only (Edge / PC) | Workstations, Laptops, Mini-PC | 4-Core x86_64 CPU (AVX2 support) or AMD Ryzen | 6 GB RAM (active footprint ~3.4 GB) | 5.0 โ 15.0 tokens/s (TG) ~45 tok/s (PP) |
| GPU Accelerated | Consumer / Gaming Desktop | NVIDIA GPU (RTX 3060, 3070, 4060, 4070 or better) | 6 GB โ 8 GB VRAM (active footprint ~3.6 GB) | 60.0 โ 85.0 tokens/s (TG) ~950 tok/s (PP) |
| Apple Silicon | MacBooks, Mac Mini, Mac Studio | Apple M1 / M2 / M3 / M4 (Metal) | 8 GB Unified Memory | 40.0 โ 55.0 tokens/s (TG) ~480 tok/s (PP) |
| Low-Power Edge | Single-Board Computers (SBC) | Raspberry Pi 5 / Rockchip RK3588 (ARM64) | 8 GB LPDDR4X | 6.0 โ 8.5 tokens/s (TG) |
2. Memory Footprint by Context Window
Memory usage scales predictably with the KV cache allocation according to the target context length:
| Context Window (Tokens) | RAM / VRAM Footprint (Q4_K_M) | Recommended System Memory | Recommended GPU |
|---|---|---|---|
| 2,048 tokens | ~3.1 GB | 6 GB RAM | 4 GB VRAM GPU |
| 4,096 tokens (Standard) | ~3.4 GB | 8 GB RAM | 6 GB VRAM GPU (e.g. RTX 2060, 3060) |
| 8,192 tokens | ~4.1 GB | 8 GB โ 12 GB RAM | 6 GB โ 8 GB VRAM GPU |
| 16,384 tokens | ~5.5 GB | 12 GB โ 16 GB RAM | 8 GB VRAM GPU (e.g. RTX 3070, 4060 Ti) |
| 32,768 tokens | ~8.3 GB | 16 GB+ RAM | 12 GB+ VRAM GPU (e.g. RTX 3060 12GB, 4070) |
3. Supported Compute Backends
- CUDA / cuBLAS: Native acceleration on NVIDIA GPUs (compute capability 6.0+).
- Metal: Native hardware acceleration for Apple Silicon macOS devices.
- ROCm / HIP: Native acceleration for supported AMD Radeon GPUs on Linux.
- Vulkan: Universal GPU acceleration across Intel, AMD, and mobile GPUs.
- AVX2 / AVX-512: CPU SIMD vector extensions for high-speed CPU-only inference.
- ARM NEON: Native acceleration for 64-bit ARM architectures (Apple, Raspberry Pi 5, Rockchip).
4. Supported Operating Systems & Runtimes
- Operating Systems: Linux (Ubuntu, Debian, Fedora, Arch, RHEL), Windows 10/11 (MSVC native & WSL2), macOS 12+ (Apple Silicon & Intel).
- Compatible Runtimes:
llama.cpp(b3000+)ollama(v0.1.30+)LM Studio/Jan.ai/LocalAI/Text Generation WebUI- Python bindings via
llama-cpp-python
๐ Benchmark & Verification Results
All outputs from KorshAI-4B-v2 have undergone automated compilation, execution, and validation against the KorshLab Evaluation Suite:
| Domain / Stack | Test Scope | Verification Standard | Pass Rate |
|---|---|---|---|
| C11 Systems | FP32 / SIMD Dot Product, Bit Manipulation | GCC 12+ (-Wall -Wextra -Werror -pedantic -std=c11) |
100% (PASSED) |
| C++20 Modern | Custom Arena Allocator, RAII Safe Handles | G++ 12+ (-Wall -Wextra -Werror -std=c++20) |
100% (PASSED) |
| PHP 8.2+ Backend | Strict Types (declare(strict_types=1)), OOP, Assertions |
PHP 8.2 CLI Engine | 100% (PASSED) |
| HTML5 & Web | Semantic UI, Accessible Modals, Bootstrap 5 | W3C / WCAG 2.1 AA Compliance Check | 100% (PASSED) |
| Structured Output | Strict JSON Schema Compliance | Python jsonschema RFC 7159 |
100% (PASSED) |
๐ฆ Provided Files
| File | Format | Quantization | Size | Description |
|---|---|---|---|---|
KorshAI-4B-v2.gguf |
GGUF | Q4_K_M (imatrix) | 2.74 GB | Recommended general-purpose quantization with high fidelity and low VRAM footprint (~3.5 GB VRAM recommended). |
๐ Quickstart & Usage
1. Running with llama.cpp
You can run KorshAI-4B-v2 directly using llama-cli:
llama-cli -m KorshAI-4B-v2.gguf \
-p "<|im_start|>system\nYou are KorshAI, an expert systems and software engineer.<|im_end|>\n<|im_start|>user\nWrite a thread-safe memory pool in C11.<|im_end|>\n<|im_start|>assistant\n" \
-c 4096 \
-ngl 99 \
--temp 0.2
2. OpenAI-Compatible API Server (llama-server)
Start an OpenAI-compatible endpoint:
llama-server -m KorshAI-4B-v2.gguf \
--host 0.0.0.0 \
--port 8080 \
-c 8192 \
-ngl 99
Query the model via curl:
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "KorshAI-4B-v2",
"messages": [
{"role": "system", "content": "You are KorshAI, an expert software engineer."},
{"role": "user", "content": "Explain RAII in C++20 with a practical code example."}
],
"temperature": 0.2
}'
3. Ollama Integration
Create a file named Modelfile:
FROM ./KorshAI-4B-v2.gguf
TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
SYSTEM """You are KorshAI, a world-class autonomous software engineering model created by KorshLab and korsh.org."""
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
PARAMETER temperature 0.2
Build and run with Ollama:
ollama create korshai:4b-v2 -f Modelfile
ollama run korshai:4b-v2
4. Python (llama-cpp-python)
from llama_cpp import Llama
llm = Llama(
model_path="KorshAI-4B-v2.gguf",
n_ctx=4096,
n_gpu_layers=-1
)
response = llm.create_chat_completion(
messages=[
{"role": "system", "content": "You are KorshAI, an expert polyglot programmer."},
{"role": "user", "content": "Implement a binary search tree in C11."}
],
temperature=0.2
)
print(response["choices"][0]["message"]["content"])
๐๏ธ About Korsh & KorshLab
KorshAI-4B-v2 is developed and maintained by KorshLab in partnership with the Korsh Foundation.
- ๐ Official Website: https://korsh.org
- ๐ฌ Laboratory: https://korshlab.space
- ๐ฌ Discord Community: Join our official Discord
๐ License & Attribution
This model is licensed under the Apache License, Version 2.0. Copyright ยฉ 2026 KorshLab & Korsh Foundation. All rights reserved.
- Downloads last month
- 445
We're not able to determine the quantization variants.
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf KorshLAB/KorshAI-4B-v2# Run inference directly in the terminal: llama cli -hf KorshLAB/KorshAI-4B-v2