| # Architecture |
|
|
| Rust owns host orchestration. Native CUDA/MMQ stays the GPU backend. |
| [v0.1.0](../releases/v0.1.0.md) defines the first independent Rust-host release |
| with a stable host/runtime boundary and a native performance-observability |
| workflow. Its production qualification is tracked separately from the |
| [pre-split parity evidence](PARITY_MATRIX.md). |
|
|
| ## Design principles |
|
|
| Keep model mechanics explicit: shapes, tensors, state, prompt protocols and |
| stops are family contracts. Add only the abstractions shared by proven families, |
| using direct dispatch and narrow interfaces. Hardware-specific hot paths stay |
| native; the Rust host owns lifecycle and policy. |
|
|
| A family addition validates the GGUF shape and tensor inventory, tokenizer and |
| input/output protocol, native state lifecycle and serving eligibility. Promotion |
| requires loader, forward, API, state/KV and measured performance gates. Preserve |
| the C oracles and original goldens rather than refreshing away drift. |
|
|
| ## Ownership and execution |
|
|
| ```text |
| CLI / API clients / coding agent |
| | |
| Rust host: model/session ownership, scheduling, serving, |
| KV/state policy, distributed orchestration |
| | |
| ds4-core (safe host API) |
| | |
| ds4-sys (native inference FFI) |
| | |
| native/bridge/ds4_bridge.h |
| | |
| Native: CUDA / MMQ / VMM / graphs / device state |
| Metal and CPU reference backends retained |
| ``` |
|
|
| Rust's `Model`, `Session`, snapshots and batch wrappers own opaque native |
| handles. Sessions borrow their model; thread-affine handles stay on the |
| execution owner. Their `Drop` paths release the matching native resources. |
| HTTP readers and output writers do not take ownership of GPU state. |
|
|
| Rust owns catalog identification/validation, tensor bind plans and tokenizer |
| behavior on the production host path. Native code owns weight upload, |
| allocation, numerical execution and GPU payloads. GGUF access stays mmap-backed; |
| do not read the entire model into a Rust byte vector. A resident VMM weight |
| owner can share allocations with fresh workers through the existing manifest |
| and broker contract. See the [memory guard](../host-memory-guard.md). |
|
|
| ## Crates and binaries |
|
|
| For Jinja-backed artifacts, input grammar runs the model's official Jinja |
| behind the local `ds4-core::chat_template` adapter. API normalization, |
| tokenizer/media processing and output protocols remain separate. REPL reuses |
| the server's host-only output parser. See |
| [template assets and continuation](../chat-templates.md), including |
| the DeepSeek V4 encoder exception. New families with official Jinja do not need |
| another imperative input renderer. |
|
|
| | Crate | Responsibility | |
| |---|---| |
| | `ds4-cli` | CLI, benchmark, coding agent | |
| | `ds4-server` | HTTP parsing/rendering, admission, scheduling, streaming, tools | |
| | `ds4-core` | Safe model/session API, catalog, tokenizer, host state | |
| | `ds4-kv` | KVC metadata, persistence policy, indexes and codecs | |
| | `ds4-dist` | Distributed protocol and blocking orchestration | |
| | `ds4-web` | Blocking agent web helpers | |
| | `ds4-sys` | ds4 native inference bindings and reviewed OS adapters | |
| | `ds4-perf` | Inspect, scout, compare and bounded performance experiments | |
| | `ds4-perf-gpu` | Optional CUDA calibration and CUPTI collection via cudarc | |
|
|
| `make` builds Rust production names `ds4`, `ds4-server`, `ds4-bench`, and |
| `ds4-agent` when a backend build is selected. Cargo retains the `*-rs` binary |
| names; Make provides those names as compatibility aliases. `ds4-c`, |
| `ds4-server-c`, `ds4-bench-c`, and `ds4-agent-c` remain C behavior oracles. |
| `ds4-eval` and native family/kernel tests also remain native. |
|
|
| `ds4_cuda.cu`, `cuda/mmq/`, and other hardware kernels are not Rust migration |
| targets. Native helper and oracle retention is intentional; it is not a claim |
| that inference has no C/CUDA dependency. Retire a C host oracle only after |
| its Rust replacement has unit/live parity, performance and soak evidence; |
| keep the baseline reachable in Git. |
|
|
| ### Native sources and tests |
|
|
| | Path | Responsibility | |
| |---|---| |
| | `native/bridge` | Opaque Rust/native inference boundary | |
| | `ds4.c` | Native model engine, GPU state and compatibility helpers | |
| | `ds4_cuda.cu`, `cuda/mmq/` | CUDA backend and vendored MMQ primitives | |
| | `ds4_metal.m`, `metal/` | Inherited Metal backend and kernels | |
| | `tests/parity` | C behavior oracles for Rust host checks | |
| | `docs/releases` | Release definitions and qualification gates | |
| | `docs/rust-migration` | Current boundary contracts and frozen migration evidence | |
|
|
| ## Host concurrency and state |
|
|
| The host uses blocking sockets, `std::thread`, channels, mutexes and condition |
| variables. `ds4-server` owns serial, continuous and static scheduling over the |
| configured/native-fitted bank capacity. Native calls execute the GPU work; |
| Rust controls admission, placement, cancellation and continuation policy. |
| Family-specific lane restrictions remain explicit in the |
| [API surface matrix](../ds4-api-surface-matrix.md). |
|
|
| The four generation surfaces are Chat Completions, Completions, Messages and |
| Responses. Their shared scheduling does not erase prompt/tool/stop contracts. |
| Distributed frames use explicit integer codecs, not Rust/C memory layouts. |
| KVC metadata and cache policy are Rust-owned; opaque numerical snapshot and |
| payload state remains native. |
|
|
| ## Performance observability |
|
|
| ```text |
| ds4-perf -> machine/calibration -> scout/fit/NCU -> compare/decision |
| | |
| +-- optional ds4-perf-gpu: CUDA calibration and CUPTI collection |
| |
| ds4-cli -- optional perf-nvtx --> NVIDIA nvtx |
| | |
| ds4-core -> ds4-sys -> native CUDA / MMQ / VMM |
| ``` |
|
|
| NVTX is a Rust-host side dependency. `ds4-cli` uses NVIDIA's `LocalRange` |
| directly around measured `ds4.prefill` and `ds4.decode` operations. Profiling |
| handles do not cross `ds4-core`, `ds4-sys`, or the native bridge. Ordinary |
| inference builds do not enable the SDK. `ds4-perf` is not an inference dependency. |
|
|
| CUDA profiling calls and CUPTI callbacks use maintained cudarc bindings in |
| `ds4-perf-gpu`. Its unsafe adapter is separate from native inference; the |
| orchestration executable and inference packages do not expose device handles. |
|
|
| The [profiling guide](../ds4-perf.md) defines the build, capability checks, |
| versioned evidence, comparison policy and bounded automatic experiments. |
| Nsight Systems is structural evidence; it does not establish memory-bound |
| versus compute-bound behavior, correctness, or end-to-end improvement. |
|
|
| ## Boundary changes |
|
|
| Follow [FFI_CONTRACT.md](FFI_CONTRACT.md). Keep model/session/device structures |
| opaque; no CUDA streams, device pointers, graph handles or VMM handles in |
| application code. Host-owned metadata descriptors may cross the documented |
| ABI; native internal layouts may not. |
|
|
| A native ABI change updates the header, implementation, Rust declarations, |
| safe wrappers and relevant parity tests together. Do not add another CUDA |
| context, allocation stack or kernel runtime behind the Rust host. Preserve |
| explicit numerical and wire contracts; change them only with scoped evidence. |
|
|
| The [migration index](README.md) retains the frozen C baseline, parity matrix |
| and genesis decision. Those records describe their original commits and |
| workloads, not the current release candidate's qualification. |
|
|
