| --- |
| license: other |
| license_name: parakon-community-license |
| license_link: https://huggingface.co/Parakon/parakon-runtime/blob/main/LICENSE_PARAKON |
| tags: |
| - parakon |
| - llama.cpp |
| - cuda |
| - compression |
| --- |
| |
| # Parakon Runtime |
|
|
| The inference runtime for [Parakon compressed models](https://huggingface.co/Parakon): |
| a fork of [llama.cpp](https://github.com/ggml-org/llama.cpp) with custom CUDA kernels |
| for Parakon's 1-bit storage formats. Stock llama.cpp cannot load |
| these artifacts; this runtime runs them at full speed with the weights kept |
| compressed in GPU memory end to end — they are never expanded to fp16. |
|
|
| ## What it runs |
|
|
| | Model | Artifact | Hardware measured | Decode speed | |
| |---|---|---|---| |
| | [Parakon-122B](https://huggingface.co/Parakon/Parakon-122B) | 23.9 GB | 4× NVIDIA A10G (25.0 GB VRAM total) | 64.5 tok/s | |
| | [Parakon-30B](https://huggingface.co/Parakon/Parakon-30B) | 5.9 GB | 1× NVIDIA A10G (6.5 GB VRAM) | 162.9 tok/s | |
|
|
| All figures measured with this repository at 4096 context, greedy decoding. |
| Metal (Apple silicon) and CPU execution of Parakon types are not in this |
| repository yet; this release is CUDA. A CUDA-capable GPU is required. |
| The 122B fits and runs on a single g5.12xlarge; the 30B fits on any 8 GB GPU. |
|
|
| ## Build (CUDA) |
|
|
| ```bash |
| cmake -B build -DGGML_CUDA=ON |
| cmake --build build --config Release -j |
| ``` |
|
|
| ## Run |
|
|
| ```bash |
| ./build/bin/llama-completion -m parakon-30b.gguf -ngl 99 -c 4096 \ |
| -p "Explain why the sky is blue." |
| ``` |
|
|
| Multi-GPU works out of the box; layers are split automatically. Sampling |
| recommendations are on each model card. |
|
|
| ## How it works |
|
|
| Parakon models store MoE expert weights in 1-bit codebook formats. The decode |
| path uses fused mat-vec kernels that read the quantized blocks directly from |
| VRAM and accumulate against int8-quantized activations via dp4a — no |
| dequantized copy of the weights ever exists in memory. Prompt processing at |
| large batch sizes falls back to a dequant-per-tile GEMM path automatically. |
|
|
| ## Attribution |
|
|
| - Fork of [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) (MIT — see `LICENSE`). |
| Upstream README preserved as `README_llama.cpp.md`; all upstream copyright notices retained. |
| - The 1-bit block formats follow the design of |
| [Unsloth's dynamic 1-bit quantization](https://huggingface.co/unsloth) family. |
| The kernels, the fused decode path, and the compression pipeline that produces |
| Parakon artifacts are Parakon's own work. |
|
|
| ## License |
|
|
| The llama.cpp base remains MIT (`LICENSE`). Parakon's additions (files prefixed |
| `parakon`, and the Parakon dispatch changes) are released under the Parakon |
| Community License (`LICENSE_PARAKON`): free for research, evaluation and |
| personal use, free commercial use for organizations under 100 employees and |
| $1M annual revenue; larger organizations need a commercial agreement — contact |
| the Parakon team via the org page. |
|
|
