| # Serving Flint with SGLang |
|
|
| This directory is Flint's serving kit for SGLang 0.5.16: the `flint_sglang` model package (a wheel), a |
| Dockerfile, `install.sh` for existing containers, the FA4 learned-sink patch, serve scripts and two |
| example clients. The commands below pin a commit of this repository. |
|
|
| Flint reasons at length on hard problems (see `max_tokens` below). |
|
|
| ## Quick start (Docker, one GPU) |
|
|
| Requirements: |
| - One GPU with ≥141 GB per replica: B300/B200 (Blackwell) or H200 (Hopper). 80 GB GPUs are not supported. |
| - NVIDIA driver ≥ 580 (CUDA 13). Older datacenter drivers (e.g. R570) work through the CUDA |
| forward-compatibility libraries in the image (`/usr/local/cuda/compat`), which recent NVIDIA |
| Container Toolkit and enroot setups pick up automatically. Tested: B300 with 580.126.20, H200 with 570.133.20. |
| - Building the image needs access to Docker Hub, pypi.org/files.pythonhosted.org, flashinfer.ai and |
| github.com (FlashInfer release assets). |
| - About 80 GB of disk for the weights plus about 50 GB for the image. |
|
|
| ```bash |
| pip install -U huggingface_hub openai |
| REV=$(python3 -c "from huggingface_hub import HfApi; print(HfApi().model_info('RecursiveSI/flint').sha)") |
| hf download RecursiveSI/flint --revision "$REV" --local-dir "flint-$REV" |
| cd "flint-$REV" && hf cache verify RecursiveSI/flint --revision "$REV" --local-dir . # verify the download |
| docker build -t "flint-sglang:${REV:0:12}" sglang |
| docker run -d --name flint --gpus '"device=0"' --ipc=host -p 127.0.0.1:30000:30000 \ |
| -v "$PWD:/model" -e MODEL=/model "flint-sglang:${REV:0:12}" \ |
| bash /model/sglang/serve_b300.sh # on H200: serve_h200.sh |
| until curl -sf localhost:30000/health_generate >/dev/null; do sleep 15; done # progress: docker logs -f flint |
| python3 sglang/example_client.py --base-url http://localhost:30000/v1 |
| python3 sglang/example_tools.py --base-url http://localhost:30000/v1 # nested tool arguments, parallel calls |
| ``` |
|
|
| The first start takes about 4 minutes on a B300 with the weights on fast local storage (longer when |
| many replicas read the weights from network storage at once; budget up to 15 minutes). The log |
| shows `Flint weights loaded: 585 parameters matched the checkpoint.` and then |
| `The server is fired up and ready to roll!`. `example_client.py` prints the reasoning, a |
| `get_weather` call for Paris, and a final answer that uses the tool result (sunny, 21C). |
| `example_tools.py` books a meeting and fills a cart through tools with nested argument schemas |
| (enums, patterns, bounds, arrays of objects) and makes three parallel weather calls. It answers each |
| call with a canned result, which Flint's final answers report. |
|
|
| ## Serve scripts |
|
|
| Requests use the model name `flint`. Arguments after the script name are appended to |
| `sglang.launch_server`, so a repeated flag overrides the script's value, e.g. |
| `bash /model/sglang/serve_b300.sh --mem-fraction-static 0.85`. |
|
|
| | Variable | Default | Notes | |
| |---|---|---| |
| | `MODEL` | `RecursiveSI/flint` | Use the local download (`/model` in Docker). With the Hub id the server downloads 79.6 GB at start-up. | |
| | `REVISION` | unset | Hub commit when `MODEL` is a Hub id (ignored for local paths). | |
| | `PORT` | `30000` | Replica port. | |
| | `CUDA_VISIBLE_DEVICES` | all GPUs | GPU of a single replica (the server uses the first visible GPU). | |
| | `FLINT_TRITON_CACHE_BASE` | `$HOME/.triton/cache` | Kernel cache root; each replica uses its own `flint-<host>-<port>` subdirectory. Mount a volume here to keep compiled kernels across container restarts. | |
|
|
| The servers listen on 0.0.0.0 without authentication: publish ports only on localhost or a private |
| network, or append `--api-key <key>` and give clients the same key. |
|
|
| ## Validated configurations |
|
|
| One GPU per replica: |
| - B300 (x86_64): `serve_b300.sh` (`--attention-backend fa4 --moe-runner-backend flashinfer_trtllm_routed`). |
| B200 and GB200/GB300 (arm64) use the same flags but are not validated yet. |
| - H200: `serve_h200.sh` (`--attention-backend fa3 --moe-runner-backend triton`). |
|
|
| Both use `--enable-fp32-lm-head --context-length 262144`. The other settings are those Flint was |
| evaluated with on each GPU type (B300: `--mem-fraction-static 0.8 --swa-full-tokens-ratio 0.05`; |
| H200: `0.9` / `0.4`). |
|
|
| The model refuses to start with a MoE runner other than `flashinfer_trtllm_routed`, `triton` or |
| `auto` (`flashinfer_trtllm` and `triton_kernel` bypass Flint's sigmoid routing), with any |
| `--moe-a2a-backend`, with an unpatched FA4 kernel, or if FlashInfer does not use its CUDA RMSNorm |
| kernels (`FLASHINFER_USE_CUDA_NORM=1`, which the image and `flint_sglang` set). `--attention-backend |
| flashinfer` does not support attention sinks. Tensor, expert or pipeline parallelism, DP attention, |
| quantized weights or KV cache (e.g. FP8) and speculative decoding are not validated. |
|
|
| ## Several replicas behind a router |
|
|
| Replicas are independent single-GPU servers; the router (`sglang_router` 0.3.2, included in the |
| image) spreads requests over them. |
|
|
| ```bash |
| # 8 replicas on one 8-GPU node, ports 30000-30007 (Blackwell; on H200 add -e SERVE_SCRIPT=serve_h200.sh) |
| docker run -d --name flint-8x --gpus all --ipc=host -p 30000-30007:30000-30007 \ |
| -v "$PWD:/model" -e MODEL=/model "flint-sglang:${REV:0:12}" bash /model/sglang/serve_node_8x.sh |
| # router on any host that can reach every replica |
| docker run -d --name flint-router --network host "flint-sglang:${REV:0:12}" \ |
| python3 -m sglang_router.launch_router --host 0.0.0.0 --port 8000 --policy cache_aware \ |
| --worker-urls $(for n in node0 node1; do for p in $(seq 30000 30007); do printf 'http://%s:%d ' $n $p; done; done) |
| ``` |
|
|
| `serve_node_8x.sh` stops all 8 replicas when one exits; for independent restarts run one container |
| per GPU instead (`--gpus '"device=N"'`, `-e PORT=3000N`). |
|
|
| ## Existing container (no image build), e.g. enroot/pyxis |
|
|
| Inside a container of the base image pinned in `sglang/Dockerfile` (`lmsysorg/sglang:v0.5.16-cu130`), |
| with network access: |
|
|
| ```bash |
| bash flint-$REV/sglang/install.sh flint-$REV/sglang # ends with: Flint support installed (flint_sglang-0.1.4-py3-none-any.whl). |
| MODEL=$PWD/flint-$REV PORT=30000 bash flint-$REV/sglang/serve_h200.sh # Blackwell: serve_b300.sh |
| ``` |
|
|
| `install.sh` changes the container in place (FlashInfer 0.6.16.post1, CUTLASS DSL 4.6.2, the patched |
| FA4 kernel, `flint_sglang`), can be re-run, and works without root if the container's system Python |
| is writable (enroot/pyxis containers are). Keep or save that container (pyxis: |
| `--container-save=flint-sglang.sqsh`); a fresh copy of the base image has no Flint support. On H200 |
| the FA4 patch is applied but unused. |
|
|
| ## Capacity and tuning |
|
|
| - KV cache per token: 28 KiB on the 14 full-attention layers (≈7.5 GB for one 262,144-token |
| sequence) plus 70 KiB on the 35 sliding-window layers, which keep only the last 513 tokens of |
| each running request. |
| - `--mem-fraction-static` is the share of GPU memory for the weights (79.6 GB) plus the KV pools. |
| - `--swa-full-tokens-ratio` sizes the sliding-window pool relative to the full-attention pool: |
| raise it for many concurrent short requests, lower it for few long ones. |
|
|
| ## Chat format and API usage |
|
|
| Tool definitions appear after leading system messages, before the first developer or other role. |
|
|
| - Use `/v1/chat/completions` with the parsers and grammar backend in the serve scripts |
| (`--tool-call-parser flint --reasoning-parser flint --grammar-backend flint`). Flint refuses to |
| start without `--grammar-backend flint` and `--reasoning-parser flint`, or with |
| `--enable-strict-thinking` or `--constrained-json-disable-any-whitespace`, which that backend |
| doesn't support. Without a tool-call parser, tool calls stay as literal `<tool_call>` text in |
| `content`. |
| - Tool-call grammar: `--tool-call-parser flint` parses Flint's tool calls and constrains every |
| request with `tools` (unless `tool_choice` is `none`) with XGrammar to Flint's format: an optional |
| `<think>…</think>` (from `--grammar-backend flint`), text without protocol markers, then tool calls |
| to the request's tools: any number with `tool_choice` `auto`, at least one with `required`, exactly |
| one call of the named function with a named choice. As in OpenAI's API, a tool's arguments must |
| match its JSON schema only when the tool is sent with `"strict": true`; they are then also written |
| in the order the tool declares its parameters (list optional parameters where they belong). Follow |
| OpenAI's strict-mode rules: give every object `properties` or `additionalProperties` and every array |
| `items` (otherwise XGrammar accepts no value for the object and only `[]` for the array); a strict |
| schema XGrammar can't compile (for example a pattern with a lookahead) fails the request. Other |
| tools keep the call format but not their schema. |
| `parallel_tool_calls: false` allows a single call. The turn still ends with Flint's stop tokens. |
| - Tool-call limits: |
| - A strict string value (an enum, `const`, `pattern`, `format` or `maxLength`) written with whitespace |
| around it fails its schema: XGrammar allows the whitespace and the parser keeps it. Flint rarely |
| does this; it has been seen on a prompt that asks it to print a tool call as text. |
| - A string argument can't contain the literal text `</tool_call>`. The format has no escaping, so the |
| parser ends the call there and reads the rest as output: text Flint copies into an argument can |
| add a call, without arguments, to a listed tool. |
| - When streaming, a call whose last argument is a JSON object can arrive without its final `}`. |
| SGLang adds it back only for the response's last call. |
| - With `tool_choice: "none"` nothing is constrained, so call markup Flint writes can reach `content`. |
| - Structured output: `response_format` (`json_schema`, `json_object` or `structural_tag`) and SGLang's |
| `regex` and `ebnf` parameters constrain `content`. Flint may first reason in `<think>…</think>` |
| (returned in `reasoning_content`), and the constraint applies to the answer after it. A request with |
| both `tools` and a response format gets only the response format, and `<tool_call>` text in the |
| answer is still read as a tool call. Use `/v1/chat/completions`: `/generate` and `/v1/completions` |
| return Flint's raw text, including any `<think>…</think>` written before a constrained answer. |
| - Sampling: temperature 1.0, top_p 0.95, top_k 20. These are the `generation_config.json` defaults |
| the server applies when a request omits them; OpenAI clients pass `top_k` via `extra_body`. |
| Greedy decoding is not validated. |
| - Reasoning: the model decides on each turn whether to reason in `<think>…</think>` (there is no |
| switch or budget). Reasoning is returned in `reasoning_content` and counts toward `max_tokens`: |
| on GPQA-Diamond it used a median of 4.3K tokens (p95 33K, p99 46K; about 5% of answers exceed 32K). |
| Use `max_tokens` of at least 32768 for hard problems (Flint was evaluated with up to 81920); `finish_reason: "length"` means the |
| turn was cut off, possibly before any content or tool call. Keep `reasoning_content` on assistant |
| messages when you send the conversation back. `thinking` / `enable_thinking` in |
| `chat_template_kwargs` have no effect: the `flint` reasoning parser reads `<think>…</think>` when |
| Flint writes it and never assumes a turn starts in reasoning, so turns without reasoning keep their |
| content and their tool-call grammar. |
| - Messages: content must be a string (flatten content-part arrays). A `developer` message uses |
| its own `<|developer|>` role marker (token ID 150018). There is no default system prompt. As in |
| training, the template removes NUL characters and `<think>`/`</think>` from system, developer, |
| user and tool text and escapes role markers there (`<|user|>` becomes `<|user|>`). It does |
| not escape the tool tags (`<tool_call>`, `<tool_response>`, `<arg_key>`, `<arg_value>`); strip |
| those from untrusted tool output. |
| - Tools: send `tools` with every request. Each function definition is rendered as the template |
| receives it; through the server that is `description`, `name`, `parameters` and `strict` (other |
| fields are dropped). Send `function.arguments` back unchanged (the JSON string you received) and |
| one `tool` message per call, in the order of `tool_calls`. When calling `apply_chat_template` |
| directly, decode argument strings into dictionaries first. |
| - Generation stops at `<|end_of_text|>`, `<|user|>`, `<|observation|>` or `<|developer|>` |
| (ids 150001, 150003, 150005, 150018). No BOS token is used for chat. |
| |
| ## Versions and verification |
|
|
| - Quote the full commit SHA (`$REV`) in results and bug reports. Download each commit into a new |
| directory (`hf download` does not delete files removed upstream). |
| - Serving kit: SGLang 0.5.16 (base image pinned by digest in `sglang/Dockerfile`; includes |
| transformers 5.12.1, torch 2.11.0+cu130, sglang-router 0.3.2), `flint_sglang` 0.1.4, FlashInfer |
| 0.6.16.post1, nvidia-cutlass-dsl 4.6.2, quack-kernels 0.6.4, apache-tvm-ffi 0.1.12. |
|
|
| When reporting a problem, include the commit SHA, the image ID (`docker inspect --format '{{.Id}}' |
| flint-sglang:<tag>`), `nvidia-smi` output, the launch command including extra arguments, the start-up |
| log and, for output problems, the request JSON and the raw response. |
|
|
| ## Transformers |
|
|
| The tokenizer and chat template load with `AutoTokenizer.from_pretrained`; decode each |
| `tool_calls[].function.arguments` string into a dict before `apply_chat_template`. Transformers does |
| not support Flint yet, so `config.json` loads with `FlintConfig.from_pretrained` from |
| `flint_sglang.configuration_flint` (transformers ≥ 5.8, tested with 5.12.1) rather than `AutoConfig`. |
|
|
