Text Generation
GGUF
English
Spanish
korsh
korshlab
code
coding
c
cpp
python
php
html5
bootstrap5
imatrix
conversational
Instructions to use KorshLAB/KorshAI-4B-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use KorshLAB/KorshAI-4B-v2 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf KorshLAB/KorshAI-4B-v2 # Run inference directly in the terminal: llama cli -hf KorshLAB/KorshAI-4B-v2
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf KorshLAB/KorshAI-4B-v2 # Run inference directly in the terminal: llama cli -hf KorshLAB/KorshAI-4B-v2
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf KorshLAB/KorshAI-4B-v2 # Run inference directly in the terminal: ./build/bin/llama cli -hf KorshLAB/KorshAI-4B-v2
Use Docker
docker model run hf.co/KorshLAB/KorshAI-4B-v2
- Llama new
- LM Studio
- Jan
- vLLM
How to use KorshLAB/KorshAI-4B-v2 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KorshLAB/KorshAI-4B-v2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KorshLAB/KorshAI-4B-v2", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/KorshLAB/KorshAI-4B-v2
- Ollama
How to use KorshLAB/KorshAI-4B-v2 with Ollama:
ollama run hf.co/KorshLAB/KorshAI-4B-v2
- Unsloth Desktop
- Pi
How to use KorshLAB/KorshAI-4B-v2 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KorshLAB/KorshAI-4B-v2
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "KorshLAB/KorshAI-4B-v2" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use KorshLAB/KorshAI-4B-v2 with Docker Model Runner:
docker model run hf.co/KorshLAB/KorshAI-4B-v2
- Lemonade
How to use KorshLAB/KorshAI-4B-v2 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull KorshLAB/KorshAI-4B-v2
Run and chat with the model
lemonade run user.KorshAI-4B-v2-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use KorshLAB/KorshAI-4B-v2 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KorshLAB/KorshAI-4B-v2
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default KorshLAB/KorshAI-4B-v2
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use KorshLAB/KorshAI-4B-v2 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf KorshLAB/KorshAI-4B-v2
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "KorshLAB/KorshAI-4B-v2" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
docs: add empirical hardware compatibility matrix and benchmarks
Browse files
README.md
CHANGED
|
@@ -31,7 +31,7 @@ Engineered specifically for polyglot systems programming, modern web architectur
|
|
| 31 |
|
| 32 |
---
|
| 33 |
|
| 34 |
-
## 🌟 Key Highlights
|
| 35 |
|
| 36 |
- **Proprietary Architecture**: Native 4B parameter dense architecture tailored for deep polyglot comprehension and algorithmic reasoning.
|
| 37 |
- **Polyglot Mastery**: Specialized in low-level systems programming (C11, C++20), modern web stacks (PHP 8.2+, HTML5, Bootstrap 5), Python 3, and structured JSON Schema generation.
|
|
@@ -41,9 +41,9 @@ Engineered specifically for polyglot systems programming, modern web architectur
|
|
| 41 |
|
| 42 |
---
|
| 43 |
|
| 44 |
-
## 💻 Hardware Compatibility
|
| 45 |
|
| 46 |
-
### 1. Requirements Matrix
|
| 47 |
|
| 48 |
| Tier / Platform | Target Setup | Minimum Specs | Memory (RAM / VRAM) | Expected Inference Speed |
|
| 49 |
| :--- | :--- | :--- | :--- | :--- |
|
|
@@ -52,7 +52,7 @@ Engineered specifically for polyglot systems programming, modern web architectur
|
|
| 52 |
| **Apple Silicon** | MacBooks, Mac Mini, Mac Studio | Apple M1 / M2 / M3 / M4 (Metal) | **8 GB Unified Memory** | **40.0 – 55.0 tokens/s** (TG)<br>~480 tok/s (PP) |
|
| 53 |
| **Low-Power Edge** | Single-Board Computers (SBC) | Raspberry Pi 5 / Rockchip RK3588 (ARM64) | **8 GB LPDDR4X** | **6.0 – 8.5 tokens/s** (TG) |
|
| 54 |
|
| 55 |
-
### 2. Memory Footprint by Context Window
|
| 56 |
|
| 57 |
Memory usage scales predictably with the KV cache allocation according to the target context length:
|
| 58 |
|
|
@@ -64,7 +64,7 @@ Memory usage scales predictably with the KV cache allocation according to the ta
|
|
| 64 |
| **16,384 tokens** | ~5.5 GB | 12 GB – 16 GB RAM | 8 GB VRAM GPU (e.g. RTX 3070, 4060 Ti) |
|
| 65 |
| **32,768 tokens** | ~8.3 GB | 16 GB+ RAM | 12 GB+ VRAM GPU (e.g. RTX 3060 12GB, 4070) |
|
| 66 |
|
| 67 |
-
### 3. Supported Compute Backends
|
| 68 |
|
| 69 |
- **CUDA / cuBLAS**: Native acceleration on NVIDIA GPUs (compute capability 6.0+).
|
| 70 |
- **Metal**: Native hardware acceleration for Apple Silicon macOS devices.
|
|
@@ -73,7 +73,7 @@ Memory usage scales predictably with the KV cache allocation according to the ta
|
|
| 73 |
- **AVX2 / AVX-512**: CPU SIMD vector extensions for high-speed CPU-only inference.
|
| 74 |
- **ARM NEON**: Native acceleration for 64-bit ARM architectures (Apple, Raspberry Pi 5, Rockchip).
|
| 75 |
|
| 76 |
-
### 4. Supported Operating Systems & Runtimes
|
| 77 |
|
| 78 |
- **Operating Systems**: Linux (Ubuntu, Debian, Fedora, Arch, RHEL), Windows 10/11 (MSVC native & WSL2), macOS 12+ (Apple Silicon & Intel).
|
| 79 |
- **Compatible Runtimes**:
|
|
@@ -84,7 +84,7 @@ Memory usage scales predictably with the KV cache allocation according to the ta
|
|
| 84 |
|
| 85 |
---
|
| 86 |
|
| 87 |
-
## 📊 Benchmark & Verification Results
|
| 88 |
|
| 89 |
All outputs from KorshAI-4B-v2 have undergone automated compilation, execution, and validation against the KorshLab Evaluation Suite:
|
| 90 |
|
|
@@ -98,7 +98,7 @@ All outputs from KorshAI-4B-v2 have undergone automated compilation, execution,
|
|
| 98 |
|
| 99 |
---
|
| 100 |
|
| 101 |
-
## 📦 Provided Files
|
| 102 |
|
| 103 |
| File | Format | Quantization | Size | Description |
|
| 104 |
| :--- | :--- | :--- | :--- | :--- |
|
|
@@ -106,7 +106,7 @@ All outputs from KorshAI-4B-v2 have undergone automated compilation, execution,
|
|
| 106 |
|
| 107 |
---
|
| 108 |
|
| 109 |
-
## 🚀 Quickstart & Usage
|
| 110 |
|
| 111 |
### 1. Running with `llama.cpp`
|
| 112 |
|
|
|
|
| 31 |
|
| 32 |
---
|
| 33 |
|
| 34 |
+
## 🌟 Key Highlights
|
| 35 |
|
| 36 |
- **Proprietary Architecture**: Native 4B parameter dense architecture tailored for deep polyglot comprehension and algorithmic reasoning.
|
| 37 |
- **Polyglot Mastery**: Specialized in low-level systems programming (C11, C++20), modern web stacks (PHP 8.2+, HTML5, Bootstrap 5), Python 3, and structured JSON Schema generation.
|
|
|
|
| 41 |
|
| 42 |
---
|
| 43 |
|
| 44 |
+
## 💻 Hardware Compatibility
|
| 45 |
|
| 46 |
+
### 1. Requirements Matrix
|
| 47 |
|
| 48 |
| Tier / Platform | Target Setup | Minimum Specs | Memory (RAM / VRAM) | Expected Inference Speed |
|
| 49 |
| :--- | :--- | :--- | :--- | :--- |
|
|
|
|
| 52 |
| **Apple Silicon** | MacBooks, Mac Mini, Mac Studio | Apple M1 / M2 / M3 / M4 (Metal) | **8 GB Unified Memory** | **40.0 – 55.0 tokens/s** (TG)<br>~480 tok/s (PP) |
|
| 53 |
| **Low-Power Edge** | Single-Board Computers (SBC) | Raspberry Pi 5 / Rockchip RK3588 (ARM64) | **8 GB LPDDR4X** | **6.0 – 8.5 tokens/s** (TG) |
|
| 54 |
|
| 55 |
+
### 2. Memory Footprint by Context Window
|
| 56 |
|
| 57 |
Memory usage scales predictably with the KV cache allocation according to the target context length:
|
| 58 |
|
|
|
|
| 64 |
| **16,384 tokens** | ~5.5 GB | 12 GB – 16 GB RAM | 8 GB VRAM GPU (e.g. RTX 3070, 4060 Ti) |
|
| 65 |
| **32,768 tokens** | ~8.3 GB | 16 GB+ RAM | 12 GB+ VRAM GPU (e.g. RTX 3060 12GB, 4070) |
|
| 66 |
|
| 67 |
+
### 3. Supported Compute Backends
|
| 68 |
|
| 69 |
- **CUDA / cuBLAS**: Native acceleration on NVIDIA GPUs (compute capability 6.0+).
|
| 70 |
- **Metal**: Native hardware acceleration for Apple Silicon macOS devices.
|
|
|
|
| 73 |
- **AVX2 / AVX-512**: CPU SIMD vector extensions for high-speed CPU-only inference.
|
| 74 |
- **ARM NEON**: Native acceleration for 64-bit ARM architectures (Apple, Raspberry Pi 5, Rockchip).
|
| 75 |
|
| 76 |
+
### 4. Supported Operating Systems & Runtimes
|
| 77 |
|
| 78 |
- **Operating Systems**: Linux (Ubuntu, Debian, Fedora, Arch, RHEL), Windows 10/11 (MSVC native & WSL2), macOS 12+ (Apple Silicon & Intel).
|
| 79 |
- **Compatible Runtimes**:
|
|
|
|
| 84 |
|
| 85 |
---
|
| 86 |
|
| 87 |
+
## 📊 Benchmark & Verification Results
|
| 88 |
|
| 89 |
All outputs from KorshAI-4B-v2 have undergone automated compilation, execution, and validation against the KorshLab Evaluation Suite:
|
| 90 |
|
|
|
|
| 98 |
|
| 99 |
---
|
| 100 |
|
| 101 |
+
## 📦 Provided Files
|
| 102 |
|
| 103 |
| File | Format | Quantization | Size | Description |
|
| 104 |
| :--- | :--- | :--- | :--- | :--- |
|
|
|
|
| 106 |
|
| 107 |
---
|
| 108 |
|
| 109 |
+
## 🚀 Quickstart & Usage
|
| 110 |
|
| 111 |
### 1. Running with `llama.cpp`
|
| 112 |
|