Instructions to use PawanKrd/RastNus-Base-12B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use PawanKrd/RastNus-Base-12B-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M
Use Docker
docker model run hf.co/PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use PawanKrd/RastNus-Base-12B-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "PawanKrd/RastNus-Base-12B-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "PawanKrd/RastNus-Base-12B-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M
- Ollama
How to use PawanKrd/RastNus-Base-12B-GGUF with Ollama:
ollama run hf.co/PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use PawanKrd/RastNus-Base-12B-GGUF with Docker Model Runner:
docker model run hf.co/PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M
- Lemonade
How to use PawanKrd/RastNus-Base-12B-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.RastNus-Base-12B-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
RastNus-Base-12B-GGUF
ڕاستنووس — Kurdish Grammar & Spelling Correction
Built by PawanKrd · GGUF builds of RastNus-Base-12B · Apache 2.0
Overview
This repository contains GGUF builds of RastNus-Base-12B, a 12-billion-parameter model for grammar, spelling, and punctuation correction in Central Kurdish (Sorani / کوردیی ناوەندی), for use with llama.cpp and compatible runtimes (LM Studio, Ollama, llama-cpp-python, …).
All quantizations below 16-bit were made with an importance matrix computed on in-domain Kurdish correction data, which preserves quality at low bit widths. The chat template with the RastNus control switches is embedded in every file.
Model Family
| Model | Params | bf16 | FP8 | NVFP4 | GGUF |
|---|---|---|---|---|---|
| RastNus-Large | – | Not publicly available - Try it now | – | – | – |
| RastNus-Medium | – | Under development | – | – | – |
| RastNus-Base | 12B | bf16 | FP8 | NVFP4 | ✅ You are here |
| RastNus-Small | 5.1B | bf16 | FP8 | NVFP4 | GGUF |
| RastNus-Tiny | 1.03B | bf16 | FP8 | NVFP4 | GGUF |
Scores for every model and format are in the Benchmarks section below.
Files
| File | Bits | Size | Notes |
|---|---|---|---|
RastNus-Base-12B-BF16.gguf |
16-bit | 23.83 GB | Lossless reference |
RastNus-Base-12B-Q8_0.gguf |
8-bit | 12.67 GB | Near-lossless — recommended for quality |
RastNus-Base-12B-Q6_K.gguf |
6-bit | 9.79 GB | Near-lossless |
RastNus-Base-12B-Q4_K_M.gguf |
4-bit | 7.38 GB | Recommended for size/quality balance |
RastNus-Base-12B-Q2_K.gguf |
2-bit | 4.83 GB | ⚠️ Not recommended — no better than leaving text uncorrected |
All quantizations below 16-bit use an importance matrix computed on in-domain Kurdish correction data (imatrix.gguf). See Benchmarks for the scores of every file.
Benchmarks
Every RastNus model and format was evaluated on the same internal held-out test set of 329 Central Kurdish correction examples, none of which were seen during training by any RastNus model, using greedy decoding. CER / WER = character / word error rate against the reference correction (lower is better). For reference, returning the input unchanged scores 6.34% CER. Differences of about ±0.2% CER are within noise.
vLLM — bf16, FP8, NVFP4
| Model | Format | Size | CER ↓ | WER ↓ | Exact match ↑ | Throughput ↑ |
|---|---|---|---|---|---|---|
| RastNus-Base-12B | bf16 | 23.9 GB | 3.68% | 11.63% | 20.7% | 653 tok/s |
| RastNus-Base-12B-FP8 | FP8 | 13.1 GB | 3.75% | 11.64% | 21.3% | 1,009 tok/s |
| RastNus-Base-12B-NVFP4 | NVFP4 | 11.7 GB | 3.54% | 11.80% | 17.3% | 1,071 tok/s |
| RastNus-Small-5B | bf16 | 10.2 GB | 3.67% | 10.76% | 17.6% | 2,783 tok/s |
| RastNus-Small-5B-FP8 | FP8 | 8.4 GB | 3.62% | 10.73% | 17.6% | 3,521 tok/s |
| RastNus-Small-5B-NVFP4 | NVFP4 | 8.0 GB | 3.72% | 11.29% | 17.9% | 3,783 tok/s |
| RastNus-Tiny-1B | bf16 | 2.1 GB | 4.77% | 13.08% | 19.5% | 4,777 tok/s |
| RastNus-Tiny-1B-FP8 | FP8 | 1.6 GB | 4.81% | 13.00% | 19.5% | 5,148 tok/s |
| RastNus-Tiny-1B-NVFP4 | NVFP4 | 1.5 GB | 4.56% | 12.67% | 18.5% | 5,891 tok/s |
Throughput = generated tokens per second for the whole test set as a batch on a single NVIDIA RTX PRO 5000 Blackwell (vLLM 0.33).
llama.cpp — GGUF
| Model | Quant | Size | CER ↓ | WER ↓ | Exact match ↑ | Same output as BF16 ↑ |
|---|---|---|---|---|---|---|
| RastNus-Base-12B-GGUF | BF16 |
23.83 GB | 3.69% | 11.65% | 20.7% | 100% |
| RastNus-Base-12B-GGUF | Q8_0 ¹ |
12.67 GB | 4.36% | 12.13% | 21.3% | 86.0% |
| RastNus-Base-12B-GGUF | Q6_K ¹ |
9.79 GB | 4.43% | 12.33% | 20.4% | 78.7% |
| RastNus-Base-12B-GGUF | Q4_K_M |
7.38 GB | 3.43% | 11.01% | 20.4% | 62.9% |
| RastNus-Base-12B-GGUF | Q2_K ⚠️ |
4.83 GB | 8.20% | 19.19% | 13.4% | 22.2% |
| RastNus-Small-5B-GGUF | BF16 |
9.27 GB | 3.68% | 10.82% | 17.3% | 100% |
| RastNus-Small-5B-GGUF | Q8_0 |
4.95 GB | 3.63% | 10.76% | 17.3% | 90.6% |
| RastNus-Small-5B-GGUF | Q6_K |
3.83 GB | 3.64% | 10.77% | 17.3% | 86.3% |
| RastNus-Small-5B-GGUF | Q4_K_M |
3.42 GB | 3.67% | 10.87% | 18.2% | 68.7% |
| RastNus-Small-5B-GGUF | Q2_K ⚠️ |
2.98 GB | 5.33% | 14.80% | 14.0% | 32.2% |
| RastNus-Tiny-1B-GGUF | BF16 |
2.06 GB | 4.53% | 12.69% | 18.5% | 100% |
| RastNus-Tiny-1B-GGUF | Q8_0 |
1.11 GB | 4.56% | 12.70% | 18.8% | 91.8% |
| RastNus-Tiny-1B-GGUF | Q6_K |
0.86 GB | 4.51% | 12.61% | 18.2% | 88.4% |
| RastNus-Tiny-1B-GGUF | Q4_K_M |
0.70 GB | 4.58% | 12.58% | 18.2% | 72.9% |
| RastNus-Tiny-1B-GGUF | Q2_K ⚠️ |
0.55 GB | 6.57% | 15.58% | 15.5% | 44.4% |
Measured with llama.cpp (CUDA) on the same GPU. Same output as BF16 = share of examples where the quantized file produces exactly the same text as the lossless BF16 GGUF — the most direct measure of quantization loss. ⚠️ = not recommended (no better than leaving the text uncorrected). ¹ = one test input containing many blank lines makes the model loop on newlines until the length limit; without that single example these files score at the BF16 level (≈3.7% CER). Small differences between the vLLM and llama.cpp bf16 scores come from numerical differences between the two engines, not from the weights.
Example
Input
خویندنەوەی کتیب زور گرنگە بو مروف ، چونکە بیروهۆش دەکاتەوە ئەو کەسە ی کتێب دەخوێنەوە؛ ئاستی رۆشنبیری یان بەرز ترە لەوانی تر
Output (Q4_K_M)
خوێندنەوەی کتێب زۆر گرنگە بۆ مرۆڤ، چونکە بیر و هۆش دەکاتەوە. ئەو کەسەی کتێب دەخوێنێتەوە، ئاستی ڕۆشنبیرییان بەرزترە لەوانی تر.
Prompt Format & Control Tokens
| Token | Effect |
|---|---|
<|punct:on|> / <|punct:off|> |
Add & normalize punctuation, or leave punctuation untouched |
<|pure:on|> / <|pure:off|> |
Replace non-Kurdish loanwords with pure Kurdish equivalents, or keep original vocabulary |
<bos><|task:kurdish_correction|><|punct:on|><|pure:off|><|input|>
{your Kurdish text}
<|output|>
{corrected text}<eos>
The embedded chat template builds this for you. Defaults are punctuationEnabled=True and kurdishPureMode=False.
Usage
llama.cpp server (OpenAI-compatible)
llama-server -hf PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M --jinja -ngl 99
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="none")
resp = client.chat.completions.create(
model="RastNus-Base-12B",
messages=[{"role": "user", "content": "خویندنەوەی کتیب زور گرنگە بو مروف"}],
extra_body={"chat_template_kwargs": {
"punctuationEnabled": True,
"kurdishPureMode": False,
}},
temperature=0.0,
)
print(resp.choices[0].message.content)
Note:
--jinjais required so thatchat_template_kwargs(the punctuation / pure-Kurdish switches) are applied.
Ollama
ollama run hf.co/PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M
Type or paste Kurdish text and the corrected text comes back. The repository ships an Ollama template and params, so the RastNus prompt format and greedy decoding are applied automatically.
Modes. By default punctuation correction is on and pure-Kurdish mode is off. To change the mode, send a system message:
| System message | Punctuation | Pure Kurdish |
|---|---|---|
| (none) | on | off |
pure:on |
on | on |
punct:off |
off | off |
punct:off pure:on |
off | on |
curl http://localhost:11434/api/chat -d '{
"model": "hf.co/PawanKrd/RastNus-Base-12B-GGUF:Q4_K_M",
"stream": false,
"messages": [
{"role": "system", "content": "pure:on"},
{"role": "user", "content": "خویندنەوەی کتیب زور گرنگە بو مروف"}
]
}'
In the interactive CLI, use /set system pure:on. The same system messages work through Ollama's OpenAI-compatible endpoint (/v1/chat/completions).
Important: run the
hf.co/...model directly. Don't wrap it in a custom model withollama create/ a Modelfile — Ollama then attaches its built-in Gemma 4 chat format, which RastNus doesn't use, and the output breaks.
Raw completion (any GGUF runtime)
If your runtime does not support chat-template arguments, send the prompt directly (without <bos> — llama.cpp adds it automatically):
curl http://localhost:8080/completion -d '{
"prompt": "<|task:kurdish_correction|><|punct:on|><|pure:off|><|input|>\nخویندنەوەی کتیب زور گرنگە بو مروف\n<|output|>\n",
"temperature": 0, "n_predict": 512
}'
Recommended Generation Settings
- Greedy decoding (
temperature=0) — correction is a deterministic task - These defaults are built in: every GGUF file carries greedy sampling metadata (
temperature 0,top_k 1) and the repository includes Ollamatemplateandparamsfiles, so no extra settings are needed - Max new tokens ≈ 1.5 × input token count
- For long documents, split at sentence/paragraph boundaries (≈1,500–2,000 characters per chunk), correct each chunk, and rejoin
Intended Use & Limitations
Intended uses: local, offline proofreading of Sorani text on consumer GPUs, laptops and servers.
These GGUF files are text-only; the image/audio inputs of the original checkpoint are not included.
Limitations:
- Targets Central Kurdish (Sorani) in Arabic script; not designed for Kurmanji (Latin script), Zazaki, or other varieties.
- A correction model, not a chat assistant.
- Can over-correct rare proper nouns, dialect-specific spellings, or intentional stylistic choices.
- Human review is recommended for high-stakes publishing.
About PawanKrd
Pawan.Krd (PKRD, LLC) builds AI and cloud infrastructure — LLMs, ASR, TTS, and related APIs — with a focus on reliability and open models. RastNus is part of that work: first-class writing tools for Kurdish, released openly.
- 🌐 Website: https://pawan.krd
- 🤗 Hugging Face: https://huggingface.co/PawanKrd
License & Attribution
Released under the Apache License, Version 2.0. See LICENSE and NOTICE.
RastNus-Base-12B-GGUF contains quantized conversions of RastNus-Base-12B, itself a derivative work of Gemma 4 12B by Google LLC / Google DeepMind (Apache 2.0). Gemma is a trademark of Google LLC, used here only to describe the origin of the base model. This project is not affiliated with, endorsed by, or sponsored by Google.
This model is provided "AS IS", without warranties or conditions of any kind.
Citation
@misc{rastnus2026,
title = {RastNus-Base-12B: A Central Kurdish (Sorani) Grammar and Spelling Correction Model},
author = {PawanKrd},
year = {2026},
url = {https://huggingface.co/PawanKrd/RastNus-Base-12B},
note = {Derivative of Google Gemma 4 12B, Apache License 2.0}
}
- Downloads last month
- 105
Model tree for PawanKrd/RastNus-Base-12B-GGUF
Base model
PawanKrd/RastNus-Base-12B