Blame for README.md

---
license: mit
tags:
  - entrit
  - d2
  - llama.cpp
  - gguf
  - quantization
---

# Run Entrit d2 models with llama.cpp

This installs **llama.cpp**, the free program that runs AI models on your own computer, with
Entrit's **d2 kernel** added. Our d2 models store each weight as one of 9 levels, and stock
llama.cpp can't read that format. The install script downloads llama.cpp, adds our kernel, and
builds it for your graphics card. With no graphics card it still builds and runs, but on the
processor alone a 27B model takes minutes per word, so plan on a graphics card for these models.

You'll type a few commands into a terminal. You don't need to know how to program.

## Which models it runs

| Model | File to download | Runs here? |
|---|---|---|
| [Qwen3.8-27B d2](https://huggingface.co/Entrit/Qwen3.8-27B-d2-GGUF) | `Qwen3.8-27B-d2-D2NL.gguf` (15.3 GB) | yes |
| [Muse-Glimmer-30B d2](https://huggingface.co/Entrit/Muse-Glimmer-30B-d2) | — | no: llama.cpp doesn't support this architecture yet. Use `transformers`. |
| [DeepSeek-V4-Flash d2](https://huggingface.co/Entrit/DeepSeek-V4-Flash-d2swap-mp2) | — | no: needs DeepSeek's own inference code. |

## What you need

- A Mac with Apple Silicon (M1 or newer), or a Linux PC.
  Windows works through WSL (see the end of this page).
- About 5 GB of free disk space for the program, plus room for the model.
- Enough memory for the model. Qwen3.8-27B needs a graphics card with **24 GB** (RTX 3090, 4090,
  or similar), or a Mac with at least **32 GB**.
- Internet, for the first install.

## Step 1: Open a terminal

- **Mac:** press Cmd-Space, type `Terminal`, press Enter.
- **Linux:** press Ctrl-Alt-T.

Every grey box below is a command. Copy it, paste it into the terminal, and press Enter.

## Step 2: Install the build tools (once)

**Mac:**
```bash
xcode-select --install
```
A window pops up. Click Install and wait for it to finish. Then install Homebrew (a tool installer)
if you don't have it, from https://brew.sh, and run:
```bash
brew install cmake
```

**Linux (Ubuntu or Debian):**
```bash
sudo apt update && sudo apt install -y build-essential cmake curl patch
```
Then add the part for your graphics card:

| Your graphics card | Extra command |
|---|---|
| NVIDIA | Install the CUDA Toolkit from https://developer.nvidia.com/cuda-downloads |
| AMD or Intel | `sudo apt install -y libvulkan-dev glslc` |
| None | nothing; the processor version works but is far too slow for a 27B model |

## Step 3: Download and install

```bash
mkdir ~/entrit-llama && cd ~/entrit-llama
curl -fsSLO https://huggingface.co/Entrit/llama.cpp-d2nl/resolve/main/install.sh
bash install.sh
```
The first two lines make a folder and download the installer into it. The installer then downloads
the rest of our files and llama.cpp, finds your graphics card, and builds for it. The first build
takes 5 to 20 minutes. At the end you should see:
```
==> Self-test: Entrit kernel on ... vs the CPU
PASS
==> Done
```
**PASS** means our kernel gives the same answers on your graphics card as on the processor.

## Step 4: Download a model

```bash
./get-model.sh Entrit/Qwen3.8-27B-d2-GGUF Qwen3.8-27B-d2-D2NL.gguf
```
It's 15 GB, so give it a while. If the download stops, run the same command again and it picks up
where it left off.

## Step 5: Chat with it

```bash
./run.sh models/Qwen3.8-27B-d2-D2NL.gguf
```
Wait until the terminal says `server is listening`. Then open **http://localhost:8080** in your web
browser and type a message. To stop, go back to the terminal and press Ctrl-C.

Other programs can talk to it too. It answers the OpenAI chat API at `http://localhost:8080/v1`.

## Updating

When we release a new version of the kernel:
```bash
cd ~/entrit-llama
./install.sh --update
```
It downloads the newest files and rebuilds. Your downloaded models stay where they are.

## If something goes wrong

| What you see | What to do |
|---|---|
| `'cmake' is missing` (or c++, curl, patch) | Do Step 2 again. |
| `could not download ...` | Check your internet connection and try again. |
| `Vulkan needs glslc` | Run `sudo apt install -y libvulkan-dev glslc`, or install the Vulkan SDK from https://vulkan.lunarg.com |
| `configure failed` or `build failed` | Open `build-config.log` or `build.log`. The last 30 lines usually say what is missing. |
| `the kernel patch did not apply` | Run `./install.sh --update` to get matching files. |
| The self-test doesn't say PASS | Open an issue on this page's Community tab with `build.log` and the name of your graphics card. |
| `out of memory`, or the model loads very slowly | The model is too big for your card. Add `-ngl 30` to put only part of it on the graphics card: `./run.sh models/<file>.gguf -ngl 30` |
| It picked the wrong backend | Choose one yourself: `./install.sh --backend vulkan` (or cuda, metal, hip, cpu) |

## Windows

Install WSL, which gives you Linux inside Windows. Open PowerShell as administrator and run
`wsl --install`, restart, then open "Ubuntu" from the Start menu. From there, follow the Linux steps.
NVIDIA cards work inside WSL with NVIDIA's normal Windows driver.

## How it works

d2 stores each weight as one of 9 levels, `{0, ±1, ±2.83, ±5.20, ±8}`, times one scale per group of
64 weights. The kernel adds a GGUF type, **D2NL**, that holds the level index in the same 4-bit block
llama.cpp uses for its IQ4_NL type, and expands it through a 16-entry int8 table:

```
level   0    ±1    ±2.828   ±5.196   ±8
int8    0   ±11    ±31      ±57      ±88      (10.981 × level, within 0.19% of each level)
```

So every IQ4_NL code path serves D2NL with the table swapped in: dp4a and int8 tensor cores on
NVIDIA, Vulkan shaders on AMD and Intel, Metal on Apple, and plain C on the processor. The AMD ROCm
route (`--backend hip`) builds from the CUDA code but hasn't been tested; use `--backend vulkan` if it
gives you trouble.

| File | What it does |
|---|---|
| `install.sh` | downloads llama.cpp, adds our kernel, builds, runs the self-test |
| `get-model.sh` | downloads a model from Hugging Face into `models/` |
| `run.sh` | starts the chat server |
| `patches/entrit-d2nl.diff` | our changes to llama.cpp |
| `LLAMA_CPP_VERSION` | the exact llama.cpp version our changes apply to (from [PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp)) |

After installing, the folder also holds `llama.cpp/` (the source), `build/` (the program) and
`models/`. You can delete `llama.cpp/` and `build/` at any time; `./install.sh` makes them again.

## License and notice

The patch and scripts are released under the MIT License, the same as llama.cpp. The d2 codec is the
subject of pending patent applications by Entrit Systems. Contact: eric@entrit.io