How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf KorshLAB/KorshAI-4B-v2
# Run inference directly in the terminal:
llama cli -hf KorshLAB/KorshAI-4B-v2
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf KorshLAB/KorshAI-4B-v2
# Run inference directly in the terminal:
llama cli -hf KorshLAB/KorshAI-4B-v2
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf KorshLAB/KorshAI-4B-v2
# Run inference directly in the terminal:
./build/bin/llama cli -hf KorshLAB/KorshAI-4B-v2
Use Docker
docker model run hf.co/KorshLAB/KorshAI-4B-v2
Quick Links

KorshAI Banner

๐Ÿš€ KorshAI-4B-v5 is now available! We strongly recommend the new champion release: KorshLAB/KorshAI-4B-v5 with improved polyglot reasoning and higher benchmark accuracy (Score: 55.85).

KorshAI-4B-v2 (GGUF)

KorshAI-4B-v2 is a proprietary, high-performance 4-billion parameter foundation coding model architected and trained by KorshLab and Korsh.

Engineered specifically for polyglot systems programming, modern web architectures, and rigorous formal code verification, KorshAI-4B-v2 delivers enterprise-grade speed and precision on consumer GPUs, workstations, and edge servers.


๐ŸŒŸ Key Highlights

  • Proprietary Architecture: Native 4B parameter dense architecture tailored for deep polyglot comprehension and algorithmic reasoning.
  • Polyglot Mastery: Specialized in low-level systems programming (C11, C++20), modern web stacks (PHP 8.2+, HTML5, Bootstrap 5), Python 3, and structured JSON Schema generation.
  • Strict Verification Protocol: Trained and verified using the KorshLab multi-tier sandbox suite with zero-tolerance compiler flags (-Wall -Wextra -Werror -pedantic).
  • Optimal Quantization: Quantized with custom importance matrices (imatrix) computed against multi-language source corpora, preserving fp16 perplexity with minimal memory overhead.
  • High-Throughput Inference: Optimized for llama.cpp CPU/GPU acceleration, delivering upwards of 14-85 tokens/s depending on acceleration tier.

๐Ÿ’ป Hardware Compatibility

1. Requirements Matrix

Tier / Platform Target Setup Minimum Specs Memory (RAM / VRAM) Expected Inference Speed
CPU Only (Edge / PC) Workstations, Laptops, Mini-PC 4-Core x86_64 CPU (AVX2 support) or AMD Ryzen 6 GB RAM (active footprint ~3.4 GB) 5.0 โ€“ 15.0 tokens/s (TG)
~45 tok/s (PP)
GPU Accelerated Consumer / Gaming Desktop NVIDIA GPU (RTX 3060, 3070, 4060, 4070 or better) 6 GB โ€“ 8 GB VRAM (active footprint ~3.6 GB) 60.0 โ€“ 85.0 tokens/s (TG)
~950 tok/s (PP)
Apple Silicon MacBooks, Mac Mini, Mac Studio Apple M1 / M2 / M3 / M4 (Metal) 8 GB Unified Memory 40.0 โ€“ 55.0 tokens/s (TG)
~480 tok/s (PP)
Low-Power Edge Single-Board Computers (SBC) Raspberry Pi 5 / Rockchip RK3588 (ARM64) 8 GB LPDDR4X 6.0 โ€“ 8.5 tokens/s (TG)

2. Memory Footprint by Context Window

Memory usage scales predictably with the KV cache allocation according to the target context length:

Context Window (Tokens) RAM / VRAM Footprint (Q4_K_M) Recommended System Memory Recommended GPU
2,048 tokens ~3.1 GB 6 GB RAM 4 GB VRAM GPU
4,096 tokens (Standard) ~3.4 GB 8 GB RAM 6 GB VRAM GPU (e.g. RTX 2060, 3060)
8,192 tokens ~4.1 GB 8 GB โ€“ 12 GB RAM 6 GB โ€“ 8 GB VRAM GPU
16,384 tokens ~5.5 GB 12 GB โ€“ 16 GB RAM 8 GB VRAM GPU (e.g. RTX 3070, 4060 Ti)
32,768 tokens ~8.3 GB 16 GB+ RAM 12 GB+ VRAM GPU (e.g. RTX 3060 12GB, 4070)

3. Supported Compute Backends

  • CUDA / cuBLAS: Native acceleration on NVIDIA GPUs (compute capability 6.0+).
  • Metal: Native hardware acceleration for Apple Silicon macOS devices.
  • ROCm / HIP: Native acceleration for supported AMD Radeon GPUs on Linux.
  • Vulkan: Universal GPU acceleration across Intel, AMD, and mobile GPUs.
  • AVX2 / AVX-512: CPU SIMD vector extensions for high-speed CPU-only inference.
  • ARM NEON: Native acceleration for 64-bit ARM architectures (Apple, Raspberry Pi 5, Rockchip).

4. Supported Operating Systems & Runtimes

  • Operating Systems: Linux (Ubuntu, Debian, Fedora, Arch, RHEL), Windows 10/11 (MSVC native & WSL2), macOS 12+ (Apple Silicon & Intel).
  • Compatible Runtimes:
    • llama.cpp (b3000+)
    • ollama (v0.1.30+)
    • LM Studio / Jan.ai / LocalAI / Text Generation WebUI
    • Python bindings via llama-cpp-python

๐Ÿ“Š Benchmark & Verification Results

All outputs from KorshAI-4B-v2 have undergone automated compilation, execution, and validation against the KorshLab Evaluation Suite:

Domain / Stack Test Scope Verification Standard Pass Rate
C11 Systems FP32 / SIMD Dot Product, Bit Manipulation GCC 12+ (-Wall -Wextra -Werror -pedantic -std=c11) 100% (PASSED)
C++20 Modern Custom Arena Allocator, RAII Safe Handles G++ 12+ (-Wall -Wextra -Werror -std=c++20) 100% (PASSED)
PHP 8.2+ Backend Strict Types (declare(strict_types=1)), OOP, Assertions PHP 8.2 CLI Engine 100% (PASSED)
HTML5 & Web Semantic UI, Accessible Modals, Bootstrap 5 W3C / WCAG 2.1 AA Compliance Check 100% (PASSED)
Structured Output Strict JSON Schema Compliance Python jsonschema RFC 7159 100% (PASSED)

๐Ÿ“ฆ Provided Files

File Format Quantization Size Description
KorshAI-4B-v2.gguf GGUF Q4_K_M (imatrix) 2.74 GB Recommended general-purpose quantization with high fidelity and low VRAM footprint (~3.5 GB VRAM recommended).

๐Ÿš€ Quickstart & Usage

1. Running with llama.cpp

You can run KorshAI-4B-v2 directly using llama-cli:

llama-cli -m KorshAI-4B-v2.gguf \
  -p "<|im_start|>system\nYou are KorshAI, an expert systems and software engineer.<|im_end|>\n<|im_start|>user\nWrite a thread-safe memory pool in C11.<|im_end|>\n<|im_start|>assistant\n" \
  -c 4096 \
  -ngl 99 \
  --temp 0.2

2. OpenAI-Compatible API Server (llama-server)

Start an OpenAI-compatible endpoint:

llama-server -m KorshAI-4B-v2.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -c 8192 \
  -ngl 99

Query the model via curl:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "KorshAI-4B-v2",
    "messages": [
      {"role": "system", "content": "You are KorshAI, an expert software engineer."},
      {"role": "user", "content": "Explain RAII in C++20 with a practical code example."}
    ],
    "temperature": 0.2
  }'

3. Ollama Integration

Create a file named Modelfile:

FROM ./KorshAI-4B-v2.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

SYSTEM """You are KorshAI, a world-class autonomous software engineering model created by KorshLab and korsh.org."""

PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
PARAMETER temperature 0.2

Build and run with Ollama:

ollama create korshai:4b-v2 -f Modelfile
ollama run korshai:4b-v2

4. Python (llama-cpp-python)

from llama_cpp import Llama

llm = Llama(
    model_path="KorshAI-4B-v2.gguf",
    n_ctx=4096,
    n_gpu_layers=-1
)

response = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are KorshAI, an expert polyglot programmer."},
        {"role": "user", "content": "Implement a binary search tree in C11."}
    ],
    temperature=0.2
)

print(response["choices"][0]["message"]["content"])

๐Ÿ›๏ธ About Korsh & KorshLab

KorshAI-4B-v2 is developed and maintained by KorshLab in partnership with the Korsh Foundation.


๐Ÿ“„ License & Attribution

This model is licensed under the Apache License, Version 2.0. Copyright ยฉ 2026 KorshLab & Korsh Foundation. All rights reserved.

Downloads last month
445
GGUF
Model size
4B params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

We're not able to determine the quantization variants.

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support