Architecture
Rust owns host orchestration. Native CUDA/MMQ stays the GPU backend. v0.1.0 defines the first independent Rust-host release with a stable host/runtime boundary and a native performance-observability workflow. Its production qualification is tracked separately from the pre-split parity evidence.
Design principles
Keep model mechanics explicit: shapes, tensors, state, prompt protocols and stops are family contracts. Add only the abstractions shared by proven families, using direct dispatch and narrow interfaces. Hardware-specific hot paths stay native; the Rust host owns lifecycle and policy.
A family addition validates the GGUF shape and tensor inventory, tokenizer and input/output protocol, native state lifecycle and serving eligibility. Promotion requires loader, forward, API, state/KV and measured performance gates. Preserve the C oracles and original goldens rather than refreshing away drift.
Ownership and execution
CLI / API clients / coding agent
|
Rust host: model/session ownership, scheduling, serving,
KV/state policy, distributed orchestration
|
ds4-core (safe host API)
|
ds4-sys (native inference FFI)
|
native/bridge/ds4_bridge.h
|
Native: CUDA / MMQ / VMM / graphs / device state
Metal and CPU reference backends retained
Rust's Model, Session, snapshots and batch wrappers own opaque native
handles. Sessions borrow their model; thread-affine handles stay on the
execution owner. Their Drop paths release the matching native resources.
HTTP readers and output writers do not take ownership of GPU state.
Rust owns catalog identification/validation, tensor bind plans and tokenizer behavior on the production host path. Native code owns weight upload, allocation, numerical execution and GPU payloads. GGUF access stays mmap-backed; do not read the entire model into a Rust byte vector. A resident VMM weight owner can share allocations with fresh workers through the existing manifest and broker contract. See the memory guard.
Crates and binaries
For Jinja-backed artifacts, input grammar runs the model's official Jinja
behind the local ds4-core::chat_template adapter. API normalization,
tokenizer/media processing and output protocols remain separate. REPL reuses
the server's host-only output parser. See
template assets and continuation, including
the DeepSeek V4 encoder exception. New families with official Jinja do not need
another imperative input renderer.
| Crate | Responsibility |
|---|---|
ds4-cli |
CLI, benchmark, coding agent |
ds4-server |
HTTP parsing/rendering, admission, scheduling, streaming, tools |
ds4-core |
Safe model/session API, catalog, tokenizer, host state |
ds4-kv |
KVC metadata, persistence policy, indexes and codecs |
ds4-dist |
Distributed protocol and blocking orchestration |
ds4-web |
Blocking agent web helpers |
ds4-sys |
ds4 native inference bindings and reviewed OS adapters |
ds4-perf |
Inspect, scout, compare and bounded performance experiments |
ds4-perf-gpu |
Optional CUDA calibration and CUPTI collection via cudarc |
make builds Rust production names ds4, ds4-server, ds4-bench, and
ds4-agent when a backend build is selected. Cargo retains the *-rs binary
names; Make provides those names as compatibility aliases. ds4-c,
ds4-server-c, ds4-bench-c, and ds4-agent-c remain C behavior oracles.
ds4-eval and native family/kernel tests also remain native.
ds4_cuda.cu, cuda/mmq/, and other hardware kernels are not Rust migration
targets. Native helper and oracle retention is intentional; it is not a claim
that inference has no C/CUDA dependency. Retire a C host oracle only after
its Rust replacement has unit/live parity, performance and soak evidence;
keep the baseline reachable in Git.
Native sources and tests
| Path | Responsibility |
|---|---|
native/bridge |
Opaque Rust/native inference boundary |
ds4.c |
Native model engine, GPU state and compatibility helpers |
ds4_cuda.cu, cuda/mmq/ |
CUDA backend and vendored MMQ primitives |
ds4_metal.m, metal/ |
Inherited Metal backend and kernels |
tests/parity |
C behavior oracles for Rust host checks |
docs/releases |
Release definitions and qualification gates |
docs/rust-migration |
Current boundary contracts and frozen migration evidence |
Host concurrency and state
The host uses blocking sockets, std::thread, channels, mutexes and condition
variables. ds4-server owns serial, continuous and static scheduling over the
configured/native-fitted bank capacity. Native calls execute the GPU work;
Rust controls admission, placement, cancellation and continuation policy.
Family-specific lane restrictions remain explicit in the
API surface matrix.
The four generation surfaces are Chat Completions, Completions, Messages and Responses. Their shared scheduling does not erase prompt/tool/stop contracts. Distributed frames use explicit integer codecs, not Rust/C memory layouts. KVC metadata and cache policy are Rust-owned; opaque numerical snapshot and payload state remains native.
Performance observability
ds4-perf -> machine/calibration -> scout/fit/NCU -> compare/decision
|
+-- optional ds4-perf-gpu: CUDA calibration and CUPTI collection
ds4-cli -- optional perf-nvtx --> NVIDIA nvtx
|
ds4-core -> ds4-sys -> native CUDA / MMQ / VMM
NVTX is a Rust-host side dependency. ds4-cli uses NVIDIA's LocalRange
directly around measured ds4.prefill and ds4.decode operations. Profiling
handles do not cross ds4-core, ds4-sys, or the native bridge. Ordinary
inference builds do not enable the SDK. ds4-perf is not an inference dependency.
CUDA profiling calls and CUPTI callbacks use maintained cudarc bindings in
ds4-perf-gpu. Its unsafe adapter is separate from native inference; the
orchestration executable and inference packages do not expose device handles.
The profiling guide defines the build, capability checks, versioned evidence, comparison policy and bounded automatic experiments. Nsight Systems is structural evidence; it does not establish memory-bound versus compute-bound behavior, correctness, or end-to-end improvement.
Boundary changes
Follow FFI_CONTRACT.md. Keep model/session/device structures opaque; no CUDA streams, device pointers, graph handles or VMM handles in application code. Host-owned metadata descriptors may cross the documented ABI; native internal layouts may not.
A native ABI change updates the header, implementation, Rust declarations, safe wrappers and relevant parity tests together. Do not add another CUDA context, allocation stack or kernel runtime behind the Rust host. Preserve explicit numerical and wire contracts; change them only with scoped evidence.
The migration index retains the frozen C baseline, parity matrix and genesis decision. Those records describe their original commits and workloads, not the current release candidate's qualification.