| # dllm-server (span-02 engine) |
|
|
| Rust + hand-written CUDA inference server for span-02 (Gemma-4 26B-A4B; NVFP4 experts, FP8 dense layers). It serves the |
| TypeSafe System One / Decision Index `/v1/systemone` API (JSON, multipart assets, zstd / gzip bodies) |
| and `/health`; the request and response format is in the model card (`../README.md`, section API). |
|
|
| Crates: |
| * `core`: model config, safetensors loader, tokenizer and prompt construction, answer policy (gate, thought budget, blend). |
| * `kernels`: the CUDA kernels (`cuda/*.cu`, compiled for sm_120a and embedded in the binary) and their launch glue. |
| * `model`: the Gemma-4 decoder on the GPU (paged KV with a sliding-window pool, CUDA graphs), the MTP draft head, |
| speculative decoding, the vision tower. |
| * `vision`: image decode and the Gemma-4 image preprocessing (bit-exact with the HF processor). |
| * `runtime`: the scheduler (continuous batching, radix prefix cache, earliest-deadline-first admission) and the |
| readout / gate / thought / readout loop. |
| * `server`: HTTP, wire format (`wire.rs`), body transport and TypeSafe image assets (`assets.rs`), command line. |
| |
| ## Requirements |
| * NVIDIA Blackwell GPU with compute capability 12.0 (sm_120a): RTX 5090 (32 GB) or RTX PRO 6000 Blackwell (96 GB). |
| The kernels use sm_120a block-scaled FP4/FP8 tensor-core instructions; other architectures are not supported. |
| * NVIDIA driver with CUDA 12.9 support (R575 or newer). |
| * Build: CUDA toolkit 12.8+ (`nvcc`; 12.9 used for the published measurements), stable Rust (1.99.0 used), |
| a C/C++ toolchain, `cmake` and `nasm` (for libjpeg-turbo), `pkg-config`, `clang`. |
| |
| ## Build |
| ``` |
| cd engine |
| NVCC=/usr/local/cuda/bin/nvcc cargo build --release -p dllm-server |
| # -> target/release/dllm-server (the sm_120a cubins are embedded; no GPU is needed at build time) |
| ``` |
| Without `nvcc` the crates still compile (for the CPU tests), but the binary has no device code and fails at start. |
| |
| ## Run |
| From the repo root, `bash launch.sh` starts the server with the submission settings (see the model card). By hand: |
| ``` |
| engine/target/release/dllm-server --bind 0.0.0.0:8000 --model <repo dir> --spec-draft <dir of google/gemma-4-26B-A4B-it-assistant> \ |
| --max-len 16384 --budget-ms 0 --tau 0.8 --cap 1024 --think-ms 750 --think-cost <repo dir>/think_cost.json \ |
| --image-soft-tokens 560 --serial --no-prefix-cache --spec-max-branches 16 |
| ``` |
| `dllm-server --help` lists every option. Throughput-oriented serving drops `--serial --no-prefix-cache` (cross-request |
| batching and the prefix cache) and may use a wall-clock thought budget (`--budget-ms`) instead of `--think-ms`. |
| |
| ## Tests |
| ``` |
| cargo test --workspace # CPU: policy, prompts, wire format, scheduler, image preprocessing |
| # GPU (one CUDA device; the model tests need the weights and the MTP draft, ~24 GB of GPU memory): |
| NVCC=/usr/local/cuda/bin/nvcc cargo test --release -p dllm-kernels -- --ignored |
| DLLM_MODEL_DIR=<repo dir> DLLM_DRAFT_DIR=<draft dir> NVCC=/usr/local/cuda/bin/nvcc \ |
| cargo test --release -p dllm-model -p dllm-server -- --ignored --test-threads 1 |
| ``` |
| The GPU tests check the kernels against analytic inputs and an f32 attention reference, that speculative thoughts equal |
| one-row greedy decoding token for token (alone and batched), and that the server with the submission settings returns |
| exactly the recorded responses of `crates/server/tests/golden/` (choices, probabilities, per-answer reasoning tokens and traces). |
| |
| License: Apache-2.0; see `NOTICE` at the repo root for the projects whose algorithms are ported. |
| |
