Inkling-2-Small on four DGX Sparks: the serving variant with a reworked draft path
This repository holds no weights. It serves the official checkpoint,
thinkingmachines/Inkling-2-Small (594B parameters, 17B
active per token, NVFP4, 394 GB) at revision b0e7afb8892a07897c78ce1c5b801a0d9cbd5e7b, as it is, with the draft
model that ships inside it (mtp.safetensors). The variant is in how the model is loaded and run: Python hooks
(MIT) that are mounted into the unmodified SGLang image lmsysorg/sglang:dev-inkling-2-small, plus a launcher and
a watchdog for four NVIDIA DGX Spark nodes (GB10, 128 GB each) on a RoCE fabric. Those files and the way to run
them are what you download here.
Status: experimental. Four days of serving on one fleet, a development image as the base, and open problems (listed in RECIPE.md).
- Setting up the nodes, the fabric and a client, step by step: the recipe on the blog (more recipes at aidenle.com).
- The full recipe text is RECIPE.md; the lab notes with every number are docs/results.md.
- The same files are kept at github.com/aidendle94/sparkrun-recipes.
Terms used below. Draft: the small model of the engine's speculative decoding (DSpark); it proposes up to 7 tokens and the big model, the target, verifies them in one forward. Pool: the KV cache, counted in tokens, shared by all running and cached requests. Step: one draft-and-verify round of decode.
What is different
The draft
The draft's weights are the checkpoint's. What changes is how it is stored and used at run time:
| the image as it is | this variant | what it gives | what it costs | |
|---|---|---|---|---|
| draft weights in memory | BF16 | int8 copies of its dense weights, made at load (DENSE_I8=draft,dhead,dfree, default) |
+5 % decode | nothing for the target (held-out NLL identical) |
| draft rows the target verifies | all of them | only the rows the draft is confident of (GATE=0.4, default) |
+15 % decode | greedy text moves at near-ties only, by less than a second request in the batch moves it |
| draft KV cache | BF16 (the engine resets it to the model dtype) | FP8 (e4m3) with a reader of its own, patches/draft_decode.py (SPARK_DRAFT_KV_FP8=1, off unless set) |
793,000 pool tokens where 716,000 fitted; a faster draft behind long contexts (60 tok/s behind 500K tokens, 56β58 before) | about 1 % of step time for one short stream, nothing at 4 and 8 streams; accepted tokens a step unchanged |
The FP8 draft cache was measured on 2026-11-16: the reader is within 2.3e-3 (relative L2) of a dense FP64 computation on the stored values; the draft's K and V stay far below what e4m3 can hold (largest 31 against 448), with a rounding error of 2.7 %; the target's arithmetic is not touched.
The target and the rest
| hook (launcher variable) | what it does |
|---|---|
patches/paged_decode.py (PAGED_DECODE=1) |
on GB10 the image's decode kernel reads other requests' cache slots as soon as two requests share a batch; this looks the page up per key. Without it, concurrent requests produce garbage |
patches/fi_moe.py (FI_MOE=1) |
the 40 NVFP4 expert layers on FlashInfer's SM120 grouped GEMM (the image's NVFP4 kernels are SM100-only): prefill from about 440 to about 2,000 tok/s |
patches/roce_collectives.py (ROCE_AR=1) |
the all-reduces over RoCE between four boxes |
patches/fp8_kv.py (KV_DTYPE=fp8_e4m3) |
FP8 KV cache for the target with a reader for SM12x: 1.9x the pool at the same speed |
patches/dense_p12.py (DENSE_P12=attn,head,free) |
the target's attention weights packed into 12 bits without rounding (exact): β4.5 % step time for one stream |
patches/tp_shard.py (TP_SHARD=half,fp8) |
every node holds all 384 experts at a quarter of their width instead of 96 whole ones: β18 % step time. Not bit-identical to the stock placement (rounding only; mean NLL unchanged) |
patches/reasoning_cache.py (REASONING_CACHE=1) |
puts an answer's reasoning back into history turns that a client sends without it |
patches/wmark_boost.py (SPARK_WMARK_BOOST=1, off unless set) |
sizes the pool without the kernel's temporary watermark boost, so every boot gets the same pool |
Each one has an off switch; RECIPE.md has the table.
Measured
Four Sparks over a switched RoCE fabric, 2026-11-13 to 2026-11-16. Every number is one boot on one fleet.
| what | measured |
|---|---|
| decode, one stream, greedy, coding questions of 400 tokens | 59β61 tok/s (37 with the image's own decode step) |
| decode, one stream, temperature 0.7 | 56.6 tok/s |
| decode, four / eight streams at a time, greedy | 95β106 / 130β143 tok/s for the batch, depending on the boot |
| prefill, ~15K-token prompts | 2,400β2,500 tok/s |
| cold prefill of 130K / 260K / 500K tokens | 61 / 142 / 348 s |
| decode behind 130K / 500K tokens of context | 68 / 60 tok/s |
| context length | 262,144 by default; 524,288 with the settings below |
| KV pool | 793,000 tokens with the settings below; sized at each boot (493Kβ805K) with the defaults |
| tool calls, 283 cases over five wire formats, two trials | 250 right, 20 soft, 13 failed (10 are tool_choice: none, which does not work; 3 slips on the sampled trial); 12 of 12 agent loops |
| boot to healthy | about 5 min with a warm kernel cache, about 17 min the first time |
Run it
You need four DGX Sparks with a RoCE fabric between them, the checkpoint on the local disk of every node (394 GB each), Docker with the NVIDIA runtime, and key-based ssh from the node you launch on (rank 0) to the other three. RECIPE.md has the whole list.
# On every node: the checkpoint and the image (arm64).
hf download thinkingmachines/Inkling-2-Small --revision b0e7afb8892a07897c78ce1c5b801a0d9cbd5e7b
docker pull lmsysorg/sglang:dev-inkling-2-small
# The hooks were written against lmsysorg/sglang@sha256:424c1da97f6d6eef1612006c77b2ca791657a8184c1fbdbc682ebf9485d3b32c
# (2026-11-12). If the tag has moved and a boot fails in a hook, pull that digest and set INKLING_IMAGE to it.
# On rank 0: this repository.
hf download aidendle94/Inkling-2-Small-DGX-Spark --local-dir inkling2-spark
cd inkling2-spark && chmod +x launch/*.sh bench/*.sh
# The site file: nodes, users, RDMA devices. Every line of the example says how to find its value.
cp launch/fleet.env.example launch/fleet.env
# Print the four docker run commands without starting anything (needs ssh to every rank).
launch/launch-inkling2.sh --dry-run
# Launch with the defaults (262,144 context, the pool sized at boot): copies patches/ to every node, starts the four
# ranks, waits for /health, arms the watchdog.
launch/relaunch-inkling2.sh
# First request (port 8210 on rank 0).
curl -s http://<rank-0>:8210/v1/chat/completions -H 'content-type: application/json' \
-d '{"model":"inkling-2-small","messages":[{"role":"user","content":"Is 91 prime?"}],"max_tokens":400}'
# Read the text of overlapping requests. Do this after every boot of a changed configuration: a broken decode
# path looks fast and healthy in every counter.
mkdir -p logs && python3 bench/concurrency_check.py && python3 bench/reasoning_cache_check.py --expect on
With the FP8 draft cache, 524,288 tokens of context and the 793,000-token pool (what the author's fleet serves with since 2026-11-16; not the launcher's defaults):
CTX=524288 MEMFRAC=0.94 NCCL_EXTRA="-e SPARK_WMARK_BOOST=1 -e SPARK_DRAFT_KV_FP8=1" \
EXTRA_ARGS="--max-total-tokens 793000 --swa-full-tokens-ratio 0.05 --prefill-decode-interval 2" \
launch/relaunch-inkling2.sh
NCCL_EXTRA is the launcher's pass-through for extra -e NAME=value arguments of the containers.
--swa-full-tokens-ratio 0.05 halves the cache of the 40 windowed layers, and --prefill-decode-interval 2 is a
scheduling setting still under test. The cap goes with the settings: the same memory holds 716,000 tokens without
SPARK_DRAFT_KV_FP8=1, and what fits depends on what the rank-0 node has free at boot.
To stop: kill the watchdog (its pid is in the last watchdog started line of ~/inkling2-watchdog.log), then
launch/launch-inkling2.sh --stop.
Using it as a client
- A chat request without a
temperatureis sampled at 0.7. For agent work and lookups in long contexts, send 0. - Send an earlier turn's reasoning back with the history (
reasoning_content, or the thinking blocks on/v1/messages). The server restores it for clients that drop it, on chat completions and/v1/messages. tool_choice: nonedoes not work: leave the tools out of the request instead.- Give long-context turns 4,000 output tokens or more; the reasoning uses them.
Files
| path | what |
|---|---|
launch/ |
the launcher (one container a node), the relaunch script, the watchdog, the example site file |
patches/ |
the hooks; sitecustomize.py installs them at interpreter start in every rank |
bench/ |
checks and measurements (text under concurrency, tool calls, long contexts, step time, kernel tests); each says in its docstring whether it is safe on a serving fleet |
docs/results.md |
the lab notes |
RECIPE.md, recipe.yaml |
the full recipe and its machine-readable summary |
Licence
The files in this repository are MIT (LICENSE); adapted third-party parts are listed in NOTICE. The model
weights are not included: they are Thinking Machines' and come under their own terms (Apache-2.0 and the
acceptable use policy). The SGLang image is used
unmodified under its own licences.