Presets
Preconfigured presets and Docker-based build/run scripts for llama.cpp and turboquant.
Hardware
Linux host has:
- RTX 2080 SUPER: CUDA device 0, compute capability
sm_75, ~8 GB VRAM - GTX 980: CUDA device 1, compute capability
sm_52, ~4 GB VRAM - Proprietary NVIDIA driver 580
llama.cpp CUDA builds target both GPUs with:
CMAKE_CUDA_ARCHITECTURES=52;75
The host keeps CUDA 13.4. The llama.cpp build uses CUDA 12.9 inside an Ubuntu 24.04 container because CUDA 12.9 is incompatible with host Ubuntu 26 headers.
Preset files
Two preset files define model configurations with server parameters, quantization, and speculative decoding settings:
| File | Purpose |
|---|---|
preset.ini |
Standard llama.cpp presets |
turboquant.ini |
TurboQuant presets with KV cache tuning (cache-type-k, cache-type-v) |
GPU settings in preset.ini:
device = CUDA0,CUDA1
split-mode = layer
main-gpu = 0
fit = on
Automatic fitting handles model VRAM overflow. layer split minimizes PCIe traffic; the GTX 980 is connected at PCIe x1.
Configuration
Scripts read .env. Copy template for another machine:
cp .env.example .env
Edit paths, image names, ports, CUDA architectures, and benchmark repetitions in .env. Keep .env local; commit .env.example only.
Key files:
| File | Purpose |
|---|---|
.env |
Local machine configuration; ignored by Git |
.env.example |
Portable configuration template |
Docker setup
Host requirements:
- Docker
- NVIDIA Container Toolkit
- Proprietary NVIDIA driver 580
Build image once after changing Dockerfile:
sudo docker compose build
Files:
| File | Purpose |
|---|---|
Dockerfile |
CUDA 12.9.1 build image with CMake, OpenSSL, GCC, and ccache |
docker-compose.yml |
GPU-enabled llama.cpp build service |
build-llama.cpp.sh |
Pull source and run containerized build |
The image includes libssl-dev, so llama.cpp can download Hugging Face models over HTTPS.
Scripts
Build (Linux)
sudo docker compose build # First time or after Dockerfile changes
./build-llama.cpp.sh # Build standard llama.cpp
./build-llama.cpp-tq.sh # Build turboquant fork
Build (Windows)
build-llama.cpp.bat # Build standard llama.cpp
build-llama.cpp-tq.bat # Build turboquant fork
Server
./server.sh # Start llama.cpp server with preset.ini
./server-tq.sh # Start turboquant server with turboquant.ini
server.sh uses the prebuilt CUDA 12.9 image, exposes port 8080, and serves both GPUs. From another computer:
http://<host-ip>:8080
The server runs in router mode and loads models on demand. Set an API key before exposing it beyond a trusted LAN; current presets intentionally have no key configured.
Benchmark
Targeted Qwen3.6 dual-GPU benchmark:
./bench-qwen3.6-35b.sh
It tests:
- GPU layer offload (
-ngl) - CPU thread counts (
-t) - Generation lengths (
-n) - Load modes (
auto,mmap,mlock,mmap+mlock,dio) - Lazy loading (
auto,on,off) - Batch and microbatch sizes
- Layer and row split modes
- Manual tensor split ratios
- CPU MoE placement
- KV cache precision
The benchmark uses CUDA 12.9 in the prebuilt container and targets both GPUs. llama-bench device syntax is slash-separated:
-dev CUDA0/CUDA1
This differs from server/preset syntax, which is comma-separated:
device = CUDA0,CUDA1
Results save to bench-results/. Analyze completed dual-GPU results:
./analyze-bench.py
The analyzer scans all result CSVs, matches exact preset model names, compares dual-GPU results against CUDA0-only results, and prints the complete current model section merged with recommendations. It never mixes Qwen3.6-35B results with Qwen3.6-35B-MTP results. Run all models or select one:
./analyze-bench.py
./analyze-bench.py --model Qwen3.6-35B-MTP
Apply recommendations. Existing keys update in place; missing keys append once, without duplicates.
A .bak backup is always created before writing:
./analyze-bench.py --apply
./analyze-bench.py --model Qwen3.6-35B-MTP --apply
Preview changes without writing:
./analyze-bench.py --dry-run
./analyze-bench.py --model Qwen3.6-35B-MTP --apply --dry-run
Recommendations use a 1 tok/s tolerance. Tensor-split benchmarks use llama-bench slash syntax (1/0.57); server preset device lists use commas (CUDA0,CUDA1). The benchmark resumes completed tests by model, GPU suite, and test name rather than requiring timestamp tracking. Recommendations use a 1 tok/s tolerance: when a higher-quality cache quantization is within 1 tok/s of the fastest result, analyzer prefers it. Preference order is f16/bf16, q8, q6, q5, then q4.
General benchmark script remains available:
./bench-presets.sh --list
./bench-presets.sh --model Qwen3.6-35B-MTP
./bench-presets.sh --all
Quick Start
cd /mnt/MODELS/presets
sudo docker compose build
./build-llama.cpp.sh
./server.sh
Then use the OpenAI-compatible API on port 8080.