| --- |
| license: agpl-3.0 |
| tags: |
| - koboldcpp |
| --- |
| |
| # koboldcpp builds with machine code for Ampere and Ada GPUs |
|
|
| Builds of [koboldcpp](https://github.com/LostRuins/koboldcpp)'s CUDA backend from unmodified release tags, with one |
| change to the compiler flags. |
|
|
| ## Why |
|
|
| The official release compiles its GPU code with `-gencode arch=compute_NN,code=compute_NN`. That emits only PTX |
| (portable intermediate code), so the NVIDIA driver compiles it the first time a model is loaded on a machine and |
| caches the result in `~/.nv/ComputeCache`. On a freshly created cloud GPU instance that cache is empty: measured on |
| an NVIDIA A40 with a 31 GB model, the first launch spent about 55 s in "Starting model warm up", against 0.3 s on |
| later launches. These builds add machine code, so there is nothing to compile: 11 s to ready instead of 69 s, with |
| the same generation speed. |
|
|
| ## What is different from the release |
|
|
| Only the CUDA 12 flag block of the `Makefile`: |
|
|
| ``` |
| NVCCFLAGS += -Wno-deprecated-gpu-targets \ |
| -gencode arch=compute_80,code=compute_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_89,code=sm_89 \ |
| -DKCPP_LIMIT_CUDA_MAX_ARCH=800 |
| ``` |
|
|
| - `sm_86` machine code: A40, RTX A6000, RTX 30 series. |
| - `sm_89` machine code: L40, L40S, RTX 6000 Ada, RTX 40 series. |
| - `compute_80` PTX: fallback for any other card, which then compiles on first use as the release does. |
| - The release's `compute_50`, `compute_61` and `compute_75` targets are dropped. |
|
|
| Each tarball records the source commit (`SOURCE_COMMIT`) and the exact flags (`BUILD_FLAGS`). |
|
|
| ## Files |
|
|
| `koboldcpp-<version>-cu12-sm86-sm89.tar.gz` unpacks to `koboldcpp/` with `koboldcpp.py`, `koboldcpp_cublas.so` and |
| the files the release bundle carries. Only the CUDA backend is included. Run it with: |
|
|
| ``` |
| python3 koboldcpp/koboldcpp.py --usecuda --model model.gguf |
| ``` |
|
|
| Needs at run time: Python 3 with `jinja2` and `psutil`, the CUDA 12 runtime and cuBLAS libraries |
| (`libcublas-12-8` on Ubuntu), and an NVIDIA driver. |
|
|
| ## Reproduce |
|
|
| `build.sh` in this repository is the complete build. It runs as a Hugging Face Job: |
|
|
| ``` |
| hf jobs run --flavor cpu-upgrade --timeout 2h --secrets HF_TOKEN \ |
| -e KCPP_VERSION=1.121 -e HF_REPO=<user>/<repo> \ |
| nvidia/cuda:12.8.1-devel-ubuntu22.04 \ |
| bash -c 'apt-get update -qq && apt-get install -y -qq curl ca-certificates >/dev/null && curl -fsSL https://huggingface.co/<user>/<repo>/resolve/main/build.sh | bash' |
| ``` |
|
|
| ## Licence |
|
|
| koboldcpp is licensed under AGPL-3.0; its bundled GGML, llama.cpp and stable-diffusion.cpp parts are MIT. Both |
| licence texts are inside each tarball. The corresponding source is the upstream repository at the commit in |
| `SOURCE_COMMIT`, plus `build.sh` here. |
|
|
