# Ripple on B200

BF16 Ripple-65B + native DFlash2 on two NVIDIA B200 GPUs, using **SGLang 0.5.19**.
Four files here: this README, `model.py`, `ripple_format.py`, and `serve.py`. Both
inference folders share `chat_template.jinja` at the repository root; keep that layout.
Uses the tools-aware ChatML template from our evals.

Requires Linux, CUDA Toolkit 13, a C++ compiler, and [uv](https://docs.astral.sh/uv/).
Use complete, unquantized BF16 target and matching `DFlash2DraftModel` checkpoints.
Run the commands below from `inference/server`.

```bash
export CUDA_HOME=/usr/local/cuda-13.0
export PATH="$CUDA_HOME/bin:$PATH"

CUDA_VISIBLE_DEVICES=0,1 uv run --script serve.py \
  --model-path /path/to/target \
  --speculative-draft-model-path /path/to/draft
```

Dependencies install automatically. The server listens on `127.0.0.1:30000`.
It uses the checkpoint's tokenizer and the repository-root chat template.

```bash
curl http://127.0.0.1:30000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"ripple","messages":[{"role":"user","content":"Hello"}],"max_tokens":128}'
```

Defaults: TP=2, BF16 weights and KV cache, TRT-LLM target attention, Triton draft
attention, 16 draft tokens, and a **1Mi-token target context** (`--context-length 1048576`,
including prompt and generated tokens). Standard SGLang CLI flags apply.
The DFlash2 draft's last layer is global attention without RoPE. It was trained on
16Ki-token rows, so the draft keeps a **16Ki-token window** (`--speculative-draft-window-size 16384`);
smaller windows are refused.
The launcher defaults `SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1` so stock SGLang can
extend the draft's positional cache beyond its 128Ki checkpoint limit. This does not
increase the draft attention window. The target checkpoint supports up to 2Mi positions.
For four GPUs, run a second copy with `CUDA_VISIBLE_DEVICES=2,3` and `--port 30001`.
This gives two independent replicas, each using two GPUs. Send eight concurrent
requests to each port for sixteen total.

The launcher enables Ripple's parsers from `ripple_format.py` (`--reasoning-parser ripple`
and `--tool-call-parser ripple`). Responses separate `message.reasoning_content`,
final-answer `message.content`, and structured `message.tool_calls`. Streaming uses the
corresponding `delta` fields. Allow enough output tokens for both reasoning and the answer.

Pass OpenAI-style `tools` and `tool_choice: "auto"` to request calls:

```json
{
  "model": "ripple",
  "messages": [{"role": "user", "content": "Use add_numbers to calculate 23 + 19."}],
  "tools": [{"type": "function", "function": {
    "name": "add_numbers", "description": "Add two integers.", "strict": true,
    "parameters": {"type": "object", "properties": {
      "a": {"type": "integer"}, "b": {"type": "integer"}
    }, "required": ["a", "b"], "additionalProperties": false}
  }}],
  "tool_choice": "auto",
  "temperature": 0,
  "max_tokens": 1024
}
```

Your application executes the requested function. Append the returned assistant
message unchanged, then a `tool` message containing its `tool_call_id` and a string
`content` with the result, and send the history again. The template re-renders an
unchanged message to exactly the tokens the model generated. Previous
`reasoning_content` is retained by default (`chat_template_kwargs.clear_thinking=false`).
The template does not implement a thinking on/off switch or reasoning-effort levels.

For `tool_choice: "required"`, use `function.strict=true` and
`parameters.additionalProperties=false`, as above. Non-strict required calls
produced extra argument fields in validation with this SGLang version.