---
license: other
license_name: parakon-community-license
license_link: https://huggingface.co/Parakon/parakon-runtime/blob/main/LICENSE_PARAKON
tags:
- parakon
- llama.cpp
- cuda
- compression
---

# Parakon Runtime

The inference runtime for [Parakon compressed models](https://huggingface.co/Parakon):
a fork of [llama.cpp](https://github.com/ggml-org/llama.cpp) with custom CUDA kernels
for Parakon's 1-bit storage formats. Stock llama.cpp cannot load
these artifacts; this runtime runs them at full speed with the weights kept
compressed in GPU memory end to end — they are never expanded to fp16.

## What it runs

| Model | Artifact | Hardware measured | Decode speed |
|---|---|---|---|
| [Parakon-122B](https://huggingface.co/Parakon/Parakon-122B) | 23.9 GB | 4× NVIDIA A10G (25.0 GB VRAM total) | 64.5 tok/s |
| [Parakon-30B](https://huggingface.co/Parakon/Parakon-30B) | 5.9 GB | 1× NVIDIA A10G (6.5 GB VRAM) | 162.9 tok/s |

All figures measured with this repository at 4096 context, greedy decoding.
Metal (Apple silicon) and CPU execution of Parakon types are not in this
repository yet; this release is CUDA. A CUDA-capable GPU is required.
The 122B fits and runs on a single g5.12xlarge; the 30B fits on any 8 GB GPU.

## Build (CUDA)

```bash
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
```

## Run

```bash
./build/bin/llama-completion -m parakon-30b.gguf -ngl 99 -c 4096 \
  -p "Explain why the sky is blue."
```

Multi-GPU works out of the box; layers are split automatically. Sampling
recommendations are on each model card.

## How it works

Parakon models store MoE expert weights in 1-bit codebook formats. The decode
path uses fused mat-vec kernels that read the quantized blocks directly from
VRAM and accumulate against int8-quantized activations via dp4a — no
dequantized copy of the weights ever exists in memory. Prompt processing at
large batch sizes falls back to a dequant-per-tile GEMM path automatically.

## Attribution

- Fork of [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) (MIT — see `LICENSE`).
  Upstream README preserved as `README_llama.cpp.md`; all upstream copyright notices retained.
- The 1-bit block formats follow the design of
  [Unsloth's dynamic 1-bit quantization](https://huggingface.co/unsloth) family.
  The kernels, the fused decode path, and the compression pipeline that produces
  Parakon artifacts are Parakon's own work.

## License

The llama.cpp base remains MIT (`LICENSE`). Parakon's additions (files prefixed
`parakon`, and the Parakon dispatch changes) are released under the Parakon
Community License (`LICENSE_PARAKON`): free for research, evaluation and
personal use, free commercial use for organizations under 100 employees and
$1M annual revenue; larger organizations need a commercial agreement — contact
the Parakon team via the org page.