---
license: agpl-3.0
tags:
  - koboldcpp
---

# koboldcpp builds with machine code for Ampere and Ada GPUs

Builds of [koboldcpp](https://github.com/LostRuins/koboldcpp)'s CUDA backend from unmodified release tags, with one
change to the compiler flags.

## Why

The official release compiles its GPU code with `-gencode arch=compute_NN,code=compute_NN`. That emits only PTX
(portable intermediate code), so the NVIDIA driver compiles it the first time a model is loaded on a machine and
caches the result in `~/.nv/ComputeCache`. On a freshly created cloud GPU instance that cache is empty: measured on
an NVIDIA A40 with a 31 GB model, the first launch spent about 55 s in "Starting model warm up", against 0.3 s on
later launches. These builds add machine code, so there is nothing to compile: 11 s to ready instead of 69 s, with
the same generation speed.

## What is different from the release

Only the CUDA 12 flag block of the `Makefile`:

```
NVCCFLAGS += -Wno-deprecated-gpu-targets \
             -gencode arch=compute_80,code=compute_80 -gencode arch=compute_86,code=sm_86 -gencode arch=compute_89,code=sm_89 \
             -DKCPP_LIMIT_CUDA_MAX_ARCH=800
```

- `sm_86` machine code: A40, RTX A6000, RTX 30 series.
- `sm_89` machine code: L40, L40S, RTX 6000 Ada, RTX 40 series.
- `compute_80` PTX: fallback for any other card, which then compiles on first use as the release does.
- The release's `compute_50`, `compute_61` and `compute_75` targets are dropped.

Each tarball records the source commit (`SOURCE_COMMIT`) and the exact flags (`BUILD_FLAGS`).

## Files

`koboldcpp-<version>-cu12-sm86-sm89.tar.gz` unpacks to `koboldcpp/` with `koboldcpp.py`, `koboldcpp_cublas.so` and
the files the release bundle carries. Only the CUDA backend is included. Run it with:

```
python3 koboldcpp/koboldcpp.py --usecuda --model model.gguf
```

Needs at run time: Python 3 with `jinja2` and `psutil`, the CUDA 12 runtime and cuBLAS libraries
(`libcublas-12-8` on Ubuntu), and an NVIDIA driver.

## Reproduce

`build.sh` in this repository is the complete build. It runs as a Hugging Face Job:

```
hf jobs run --flavor cpu-upgrade --timeout 2h --secrets HF_TOKEN \
  -e KCPP_VERSION=1.121 -e HF_REPO=<user>/<repo> \
  nvidia/cuda:12.8.1-devel-ubuntu22.04 \
  bash -c 'apt-get update -qq && apt-get install -y -qq curl ca-certificates >/dev/null && curl -fsSL https://huggingface.co/<user>/<repo>/resolve/main/build.sh | bash'
```

## Licence

koboldcpp is licensed under AGPL-3.0; its bundled GGML, llama.cpp and stable-diffusion.cpp parts are MIT. Both
licence texts are inside each tarball. The corresponding source is the upstream repository at the commit in
`SOURCE_COMMIT`, plus `build.sh` here.