| --- |
| license: mit |
| tags: |
| - entrit |
| - d2 |
| - llama.cpp |
| - gguf |
| - quantization |
| --- |
| |
| # Run Entrit d2 models with llama.cpp |
|
|
| This installs **llama.cpp**, the free program that runs AI models on your own computer, with |
| Entrit's **d2 kernel** added. Our d2 models store each weight as one of 9 levels, and stock |
| llama.cpp can't read that format. The install script downloads llama.cpp, adds our kernel, and |
| builds it for your graphics card. With no graphics card it still builds and runs, but on the |
| processor alone a 27B model takes minutes per word, so plan on a graphics card for these models. |
|
|
| You'll type a few commands into a terminal. You don't need to know how to program. |
|
|
| ## Which models it runs |
|
|
| | Model | File to download | Runs here? | |
| |---|---|---| |
| | [Qwen3.8-27B d2](https://huggingface.co/Entrit/Qwen3.8-27B-d2-GGUF) | `Qwen3.8-27B-d2-D2NL.gguf` (15.3 GB) | yes | |
| | [Muse-Glimmer-30B d2](https://huggingface.co/Entrit/Muse-Glimmer-30B-d2) | — | no: llama.cpp doesn't support this architecture yet. Use `transformers`. | |
| | [DeepSeek-V4-Flash d2](https://huggingface.co/Entrit/DeepSeek-V4-Flash-d2swap-mp2) | — | no: needs DeepSeek's own inference code. | |
|
|
| ## What you need |
|
|
| - A Mac with Apple Silicon (M1 or newer), or a Linux PC. |
| Windows works through WSL (see the end of this page). |
| - About 5 GB of free disk space for the program, plus room for the model. |
| - Enough memory for the model. Qwen3.8-27B needs a graphics card with **24 GB** (RTX 3090, 4090, |
| or similar), or a Mac with at least **32 GB**. |
| - Internet, for the first install. |
|
|
| ## Step 1: Open a terminal |
|
|
| - **Mac:** press Cmd-Space, type `Terminal`, press Enter. |
| - **Linux:** press Ctrl-Alt-T. |
|
|
| Every grey box below is a command. Copy it, paste it into the terminal, and press Enter. |
|
|
| ## Step 2: Install the build tools (once) |
|
|
| **Mac:** |
| ```bash |
| xcode-select --install |
| ``` |
| A window pops up. Click Install and wait for it to finish. Then install Homebrew (a tool installer) |
| if you don't have it, from https://brew.sh, and run: |
| ```bash |
| brew install cmake |
| ``` |
|
|
| **Linux (Ubuntu or Debian):** |
| ```bash |
| sudo apt update && sudo apt install -y build-essential cmake curl patch |
| ``` |
| Then add the part for your graphics card: |
|
|
| | Your graphics card | Extra command | |
| |---|---| |
| | NVIDIA | Install the CUDA Toolkit from https://developer.nvidia.com/cuda-downloads | |
| | AMD or Intel | `sudo apt install -y libvulkan-dev glslc` | |
| | None | nothing; the processor version works but is far too slow for a 27B model | |
|
|
| ## Step 3: Download and install |
|
|
| ```bash |
| mkdir ~/entrit-llama && cd ~/entrit-llama |
| curl -fsSLO https://huggingface.co/Entrit/llama.cpp-d2nl/resolve/main/install.sh |
| bash install.sh |
| ``` |
| The first two lines make a folder and download the installer into it. The installer then downloads |
| the rest of our files and llama.cpp, finds your graphics card, and builds for it. The first build |
| takes 5 to 20 minutes. At the end you should see: |
| ``` |
| ==> Self-test: Entrit kernel on ... vs the CPU |
| PASS |
| ==> Done |
| ``` |
| **PASS** means our kernel gives the same answers on your graphics card as on the processor. |
|
|
| ## Step 4: Download a model |
|
|
| ```bash |
| ./get-model.sh Entrit/Qwen3.8-27B-d2-GGUF Qwen3.8-27B-d2-D2NL.gguf |
| ``` |
| It's 15 GB, so give it a while. If the download stops, run the same command again and it picks up |
| where it left off. |
|
|
| ## Step 5: Chat with it |
|
|
| ```bash |
| ./run.sh models/Qwen3.8-27B-d2-D2NL.gguf |
| ``` |
| Wait until the terminal says `server is listening`. Then open **http://localhost:8080** in your web |
| browser and type a message. To stop, go back to the terminal and press Ctrl-C. |
|
|
| Other programs can talk to it too. It answers the OpenAI chat API at `http://localhost:8080/v1`. |
|
|
| ## Updating |
|
|
| When we release a new version of the kernel: |
| ```bash |
| cd ~/entrit-llama |
| ./install.sh --update |
| ``` |
| It downloads the newest files and rebuilds. Your downloaded models stay where they are. |
|
|
| ## If something goes wrong |
|
|
| | What you see | What to do | |
| |---|---| |
| | `'cmake' is missing` (or c++, curl, patch) | Do Step 2 again. | |
| | `could not download ...` | Check your internet connection and try again. | |
| | `Vulkan needs glslc` | Run `sudo apt install -y libvulkan-dev glslc`, or install the Vulkan SDK from https://vulkan.lunarg.com | |
| | `configure failed` or `build failed` | Open `build-config.log` or `build.log`. The last 30 lines usually say what is missing. | |
| | `the kernel patch did not apply` | Run `./install.sh --update` to get matching files. | |
| | The self-test doesn't say PASS | Open an issue on this page's Community tab with `build.log` and the name of your graphics card. | |
| | `out of memory`, or the model loads very slowly | The model is too big for your card. Add `-ngl 30` to put only part of it on the graphics card: `./run.sh models/<file>.gguf -ngl 30` | |
| | It picked the wrong backend | Choose one yourself: `./install.sh --backend vulkan` (or cuda, metal, hip, cpu) | |
|
|
| ## Windows |
|
|
| Install WSL, which gives you Linux inside Windows. Open PowerShell as administrator and run |
| `wsl --install`, restart, then open "Ubuntu" from the Start menu. From there, follow the Linux steps. |
| NVIDIA cards work inside WSL with NVIDIA's normal Windows driver. |
|
|
| ## How it works |
|
|
| d2 stores each weight as one of 9 levels, `{0, ±1, ±2.83, ±5.20, ±8}`, times one scale per group of |
| 64 weights. The kernel adds a GGUF type, **D2NL**, that holds the level index in the same 4-bit block |
| llama.cpp uses for its IQ4_NL type, and expands it through a 16-entry int8 table: |
| |
| ``` |
| level 0 ±1 ±2.828 ±5.196 ±8 |
| int8 0 ±11 ±31 ±57 ±88 (10.981 × level, within 0.19% of each level) |
| ``` |
| |
| So every IQ4_NL code path serves D2NL with the table swapped in: dp4a and int8 tensor cores on |
| NVIDIA, Vulkan shaders on AMD and Intel, Metal on Apple, and plain C on the processor. The AMD ROCm |
| route (`--backend hip`) builds from the CUDA code but hasn't been tested; use `--backend vulkan` if it |
| gives you trouble. |
|
|
| | File | What it does | |
| |---|---| |
| | `install.sh` | downloads llama.cpp, adds our kernel, builds, runs the self-test | |
| | `get-model.sh` | downloads a model from Hugging Face into `models/` | |
| | `run.sh` | starts the chat server | |
| | `patches/entrit-d2nl.diff` | our changes to llama.cpp | |
| | `LLAMA_CPP_VERSION` | the exact llama.cpp version our changes apply to (from [PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp)) | |
|
|
| After installing, the folder also holds `llama.cpp/` (the source), `build/` (the program) and |
| `models/`. You can delete `llama.cpp/` and `build/` at any time; `./install.sh` makes them again. |
|
|
| ## License and notice |
|
|
| The patch and scripts are released under the MIT License, the same as llama.cpp. The d2 codec is the |
| subject of pending patent applications by Entrit Systems. Contact: eric@entrit.io |
|
|
