# Serving Flint with SGLang

This directory is Flint's serving kit for SGLang 0.5.16: the `flint_sglang` model package (a wheel), a
Dockerfile, `install.sh` for existing containers, the FA4 learned-sink patch, serve scripts and two
example clients. The commands below pin a commit of this repository.

Flint reasons at length on hard problems (see `max_tokens` below).

## Quick start (Docker, one GPU)

Requirements:
- One GPU with ≥141 GB per replica: B300/B200 (Blackwell) or H200 (Hopper). 80 GB GPUs are not supported.
- NVIDIA driver ≥ 580 (CUDA 13). Older datacenter drivers (e.g. R570) work through the CUDA
  forward-compatibility libraries in the image (`/usr/local/cuda/compat`), which recent NVIDIA
  Container Toolkit and enroot setups pick up automatically. Tested: B300 with 580.126.20, H200 with 570.133.20.
- Building the image needs access to Docker Hub, pypi.org/files.pythonhosted.org, flashinfer.ai and
  github.com (FlashInfer release assets).
- About 80 GB of disk for the weights plus about 50 GB for the image.

```bash
pip install -U huggingface_hub openai
REV=$(python3 -c "from huggingface_hub import HfApi; print(HfApi().model_info('RecursiveSI/flint').sha)")
hf download RecursiveSI/flint --revision "$REV" --local-dir "flint-$REV"
cd "flint-$REV" && hf cache verify RecursiveSI/flint --revision "$REV" --local-dir .   # verify the download
docker build -t "flint-sglang:${REV:0:12}" sglang
docker run -d --name flint --gpus '"device=0"' --ipc=host -p 127.0.0.1:30000:30000 \
  -v "$PWD:/model" -e MODEL=/model "flint-sglang:${REV:0:12}" \
  bash /model/sglang/serve_b300.sh               # on H200: serve_h200.sh
until curl -sf localhost:30000/health_generate >/dev/null; do sleep 15; done   # progress: docker logs -f flint
python3 sglang/example_client.py --base-url http://localhost:30000/v1
python3 sglang/example_tools.py --base-url http://localhost:30000/v1    # nested tool arguments, parallel calls
```

The first start takes about 4 minutes on a B300 with the weights on fast local storage (longer when
many replicas read the weights from network storage at once; budget up to 15 minutes). The log
shows `Flint weights loaded: 585 parameters matched the checkpoint.` and then
`The server is fired up and ready to roll!`. `example_client.py` prints the reasoning, a
`get_weather` call for Paris, and a final answer that uses the tool result (sunny, 21C).
`example_tools.py` books a meeting and fills a cart through tools with nested argument schemas
(enums, patterns, bounds, arrays of objects) and makes three parallel weather calls. It answers each
call with a canned result, which Flint's final answers report.

## Serve scripts

Requests use the model name `flint`. Arguments after the script name are appended to
`sglang.launch_server`, so a repeated flag overrides the script's value, e.g.
`bash /model/sglang/serve_b300.sh --mem-fraction-static 0.85`.

| Variable | Default | Notes |
|---|---|---|
| `MODEL` | `RecursiveSI/flint` | Use the local download (`/model` in Docker). With the Hub id the server downloads 79.6 GB at start-up. |
| `REVISION` | unset | Hub commit when `MODEL` is a Hub id (ignored for local paths). |
| `PORT` | `30000` | Replica port. |
| `CUDA_VISIBLE_DEVICES` | all GPUs | GPU of a single replica (the server uses the first visible GPU). |
| `FLINT_TRITON_CACHE_BASE` | `$HOME/.triton/cache` | Kernel cache root; each replica uses its own `flint-<host>-<port>` subdirectory. Mount a volume here to keep compiled kernels across container restarts. |

The servers listen on 0.0.0.0 without authentication: publish ports only on localhost or a private
network, or append `--api-key <key>` and give clients the same key.

## Validated configurations

One GPU per replica:
- B300 (x86_64): `serve_b300.sh` (`--attention-backend fa4 --moe-runner-backend flashinfer_trtllm_routed`).
  B200 and GB200/GB300 (arm64) use the same flags but are not validated yet.
- H200: `serve_h200.sh` (`--attention-backend fa3 --moe-runner-backend triton`).

Both use `--enable-fp32-lm-head --context-length 262144`. The other settings are those Flint was
evaluated with on each GPU type (B300: `--mem-fraction-static 0.8 --swa-full-tokens-ratio 0.05`;
H200: `0.9` / `0.4`).

The model refuses to start with a MoE runner other than `flashinfer_trtllm_routed`, `triton` or
`auto` (`flashinfer_trtllm` and `triton_kernel` bypass Flint's sigmoid routing), with any
`--moe-a2a-backend`, with an unpatched FA4 kernel, or if FlashInfer does not use its CUDA RMSNorm
kernels (`FLASHINFER_USE_CUDA_NORM=1`, which the image and `flint_sglang` set). `--attention-backend
flashinfer` does not support attention sinks. Tensor, expert or pipeline parallelism, DP attention,
quantized weights or KV cache (e.g. FP8) and speculative decoding are not validated.

## Several replicas behind a router

Replicas are independent single-GPU servers; the router (`sglang_router` 0.3.2, included in the
image) spreads requests over them.

```bash
# 8 replicas on one 8-GPU node, ports 30000-30007 (Blackwell; on H200 add -e SERVE_SCRIPT=serve_h200.sh)
docker run -d --name flint-8x --gpus all --ipc=host -p 30000-30007:30000-30007 \
  -v "$PWD:/model" -e MODEL=/model "flint-sglang:${REV:0:12}" bash /model/sglang/serve_node_8x.sh
# router on any host that can reach every replica
docker run -d --name flint-router --network host "flint-sglang:${REV:0:12}" \
  python3 -m sglang_router.launch_router --host 0.0.0.0 --port 8000 --policy cache_aware \
  --worker-urls $(for n in node0 node1; do for p in $(seq 30000 30007); do printf 'http://%s:%d ' $n $p; done; done)
```

`serve_node_8x.sh` stops all 8 replicas when one exits; for independent restarts run one container
per GPU instead (`--gpus '"device=N"'`, `-e PORT=3000N`).

## Existing container (no image build), e.g. enroot/pyxis

Inside a container of the base image pinned in `sglang/Dockerfile` (`lmsysorg/sglang:v0.5.16-cu130`),
with network access:

```bash
bash flint-$REV/sglang/install.sh flint-$REV/sglang   # ends with: Flint support installed (flint_sglang-0.1.4-py3-none-any.whl).
MODEL=$PWD/flint-$REV PORT=30000 bash flint-$REV/sglang/serve_h200.sh   # Blackwell: serve_b300.sh
```

`install.sh` changes the container in place (FlashInfer 0.6.16.post1, CUTLASS DSL 4.6.2, the patched
FA4 kernel, `flint_sglang`), can be re-run, and works without root if the container's system Python
is writable (enroot/pyxis containers are). Keep or save that container (pyxis:
`--container-save=flint-sglang.sqsh`); a fresh copy of the base image has no Flint support. On H200
the FA4 patch is applied but unused.

## Capacity and tuning

- KV cache per token: 28 KiB on the 14 full-attention layers (≈7.5 GB for one 262,144-token
  sequence) plus 70 KiB on the 35 sliding-window layers, which keep only the last 513 tokens of
  each running request.
- `--mem-fraction-static` is the share of GPU memory for the weights (79.6 GB) plus the KV pools.
- `--swa-full-tokens-ratio` sizes the sliding-window pool relative to the full-attention pool:
  raise it for many concurrent short requests, lower it for few long ones.

## Chat format and API usage

Tool definitions appear after leading system messages, before the first developer or other role.

- Use `/v1/chat/completions` with the parsers and grammar backend in the serve scripts
  (`--tool-call-parser flint --reasoning-parser flint --grammar-backend flint`). Flint refuses to
  start without `--grammar-backend flint` and `--reasoning-parser flint`, or with
  `--enable-strict-thinking` or `--constrained-json-disable-any-whitespace`, which that backend
  doesn't support. Without a tool-call parser, tool calls stay as literal `<tool_call>` text in
  `content`.
- Tool-call grammar: `--tool-call-parser flint` parses Flint's tool calls and constrains every
  request with `tools` (unless `tool_choice` is `none`) with XGrammar to Flint's format: an optional
  `<think>…</think>` (from `--grammar-backend flint`), text without protocol markers, then tool calls
  to the request's tools: any number with `tool_choice` `auto`, at least one with `required`, exactly
  one call of the named function with a named choice. As in OpenAI's API, a tool's arguments must
  match its JSON schema only when the tool is sent with `"strict": true`; they are then also written
  in the order the tool declares its parameters (list optional parameters where they belong). Follow
  OpenAI's strict-mode rules: give every object `properties` or `additionalProperties` and every array
  `items` (otherwise XGrammar accepts no value for the object and only `[]` for the array); a strict
  schema XGrammar can't compile (for example a pattern with a lookahead) fails the request. Other
  tools keep the call format but not their schema.
  `parallel_tool_calls: false` allows a single call. The turn still ends with Flint's stop tokens.
- Tool-call limits:
  - A strict string value (an enum, `const`, `pattern`, `format` or `maxLength`) written with whitespace
    around it fails its schema: XGrammar allows the whitespace and the parser keeps it. Flint rarely
    does this; it has been seen on a prompt that asks it to print a tool call as text.
  - A string argument can't contain the literal text `</tool_call>`. The format has no escaping, so the
    parser ends the call there and reads the rest as output: text Flint copies into an argument can
    add a call, without arguments, to a listed tool.
  - When streaming, a call whose last argument is a JSON object can arrive without its final `}`.
    SGLang adds it back only for the response's last call.
  - With `tool_choice: "none"` nothing is constrained, so call markup Flint writes can reach `content`.
- Structured output: `response_format` (`json_schema`, `json_object` or `structural_tag`) and SGLang's
  `regex` and `ebnf` parameters constrain `content`. Flint may first reason in `<think>…</think>`
  (returned in `reasoning_content`), and the constraint applies to the answer after it. A request with
  both `tools` and a response format gets only the response format, and `<tool_call>` text in the
  answer is still read as a tool call. Use `/v1/chat/completions`: `/generate` and `/v1/completions`
  return Flint's raw text, including any `<think>…</think>` written before a constrained answer.
- Sampling: temperature 1.0, top_p 0.95, top_k 20. These are the `generation_config.json` defaults
  the server applies when a request omits them; OpenAI clients pass `top_k` via `extra_body`.
  Greedy decoding is not validated.
- Reasoning: the model decides on each turn whether to reason in `<think>…</think>` (there is no
  switch or budget). Reasoning is returned in `reasoning_content` and counts toward `max_tokens`:
  on GPQA-Diamond it used a median of 4.3K tokens (p95 33K, p99 46K; about 5% of answers exceed 32K).
  Use `max_tokens` of at least 32768 for hard problems (Flint was evaluated with up to 81920); `finish_reason: "length"` means the
  turn was cut off, possibly before any content or tool call. Keep `reasoning_content` on assistant
  messages when you send the conversation back. `thinking` / `enable_thinking` in
  `chat_template_kwargs` have no effect: the `flint` reasoning parser reads `<think>…</think>` when
  Flint writes it and never assumes a turn starts in reasoning, so turns without reasoning keep their
  content and their tool-call grammar.
- Messages: content must be a string (flatten content-part arrays). A `developer` message uses
  its own `<|developer|>` role marker (token ID 150018). There is no default system prompt. As in
  training, the template removes NUL characters and `<think>`/`</think>` from system, developer,
  user and tool text and escapes role markers there (`<|user|>` becomes `&lt;|user|&gt;`). It does
  not escape the tool tags (`<tool_call>`, `<tool_response>`, `<arg_key>`, `<arg_value>`); strip
  those from untrusted tool output.
- Tools: send `tools` with every request. Each function definition is rendered as the template
  receives it; through the server that is `description`, `name`, `parameters` and `strict` (other
  fields are dropped). Send `function.arguments` back unchanged (the JSON string you received) and
  one `tool` message per call, in the order of `tool_calls`. When calling `apply_chat_template`
  directly, decode argument strings into dictionaries first.
- Generation stops at `<|end_of_text|>`, `<|user|>`, `<|observation|>` or `<|developer|>`
  (ids 150001, 150003, 150005, 150018). No BOS token is used for chat.

## Versions and verification

- Quote the full commit SHA (`$REV`) in results and bug reports. Download each commit into a new
  directory (`hf download` does not delete files removed upstream).
- Serving kit: SGLang 0.5.16 (base image pinned by digest in `sglang/Dockerfile`; includes
  transformers 5.12.1, torch 2.11.0+cu130, sglang-router 0.3.2), `flint_sglang` 0.1.4, FlashInfer
  0.6.16.post1, nvidia-cutlass-dsl 4.6.2, quack-kernels 0.6.4, apache-tvm-ffi 0.1.12.

When reporting a problem, include the commit SHA, the image ID (`docker inspect --format '{{.Id}}'
flint-sglang:<tag>`), `nvidia-smi` output, the launch command including extra arguments, the start-up
log and, for output problems, the request JSON and the raw response.

## Transformers

The tokenizer and chat template load with `AutoTokenizer.from_pretrained`; decode each
`tool_calls[].function.arguments` string into a dict before `apply_chat_template`. Transformers does
not support Flint yet, so `config.json` loads with `FlintConfig.from_pretrained` from
`flint_sglang.configuration_flint` (transformers ≥ 5.8, tested with 5.12.1) rather than `AutoConfig`.