Inkling-2-Small on four DGX Sparks: the serving variant with a reworked draft path

This repository holds no weights. It serves the official checkpoint, thinkingmachines/Inkling-2-Small (594B parameters, 17B active per token, NVFP4, 394 GB) at revision b0e7afb8892a07897c78ce1c5b801a0d9cbd5e7b, as it is, with the draft model that ships inside it (mtp.safetensors). The variant is in how the model is loaded and run: Python hooks (MIT) that are mounted into the unmodified SGLang image lmsysorg/sglang:dev-inkling-2-small, plus a launcher and a watchdog for four NVIDIA DGX Spark nodes (GB10, 128 GB each) on a RoCE fabric. Those files and the way to run them are what you download here.

Status: experimental. Four days of serving on one fleet, a development image as the base, and open problems (listed in RECIPE.md).

Terms used below. Draft: the small model of the engine's speculative decoding (DSpark); it proposes up to 7 tokens and the big model, the target, verifies them in one forward. Pool: the KV cache, counted in tokens, shared by all running and cached requests. Step: one draft-and-verify round of decode.

What is different

The draft

The draft's weights are the checkpoint's. What changes is how it is stored and used at run time:

the image as it is this variant what it gives what it costs
draft weights in memory BF16 int8 copies of its dense weights, made at load (DENSE_I8=draft,dhead,dfree, default) +5 % decode nothing for the target (held-out NLL identical)
draft rows the target verifies all of them only the rows the draft is confident of (GATE=0.4, default) +15 % decode greedy text moves at near-ties only, by less than a second request in the batch moves it
draft KV cache BF16 (the engine resets it to the model dtype) FP8 (e4m3) with a reader of its own, patches/draft_decode.py (SPARK_DRAFT_KV_FP8=1, off unless set) 793,000 pool tokens where 716,000 fitted; a faster draft behind long contexts (60 tok/s behind 500K tokens, 56–58 before) about 1 % of step time for one short stream, nothing at 4 and 8 streams; accepted tokens a step unchanged

The FP8 draft cache was measured on 2026-11-16: the reader is within 2.3e-3 (relative L2) of a dense FP64 computation on the stored values; the draft's K and V stay far below what e4m3 can hold (largest 31 against 448), with a rounding error of 2.7 %; the target's arithmetic is not touched.

The target and the rest

hook (launcher variable) what it does
patches/paged_decode.py (PAGED_DECODE=1) on GB10 the image's decode kernel reads other requests' cache slots as soon as two requests share a batch; this looks the page up per key. Without it, concurrent requests produce garbage
patches/fi_moe.py (FI_MOE=1) the 40 NVFP4 expert layers on FlashInfer's SM120 grouped GEMM (the image's NVFP4 kernels are SM100-only): prefill from about 440 to about 2,000 tok/s
patches/roce_collectives.py (ROCE_AR=1) the all-reduces over RoCE between four boxes
patches/fp8_kv.py (KV_DTYPE=fp8_e4m3) FP8 KV cache for the target with a reader for SM12x: 1.9x the pool at the same speed
patches/dense_p12.py (DENSE_P12=attn,head,free) the target's attention weights packed into 12 bits without rounding (exact): βˆ’4.5 % step time for one stream
patches/tp_shard.py (TP_SHARD=half,fp8) every node holds all 384 experts at a quarter of their width instead of 96 whole ones: βˆ’18 % step time. Not bit-identical to the stock placement (rounding only; mean NLL unchanged)
patches/reasoning_cache.py (REASONING_CACHE=1) puts an answer's reasoning back into history turns that a client sends without it
patches/wmark_boost.py (SPARK_WMARK_BOOST=1, off unless set) sizes the pool without the kernel's temporary watermark boost, so every boot gets the same pool

Each one has an off switch; RECIPE.md has the table.

Measured

Four Sparks over a switched RoCE fabric, 2026-11-13 to 2026-11-16. Every number is one boot on one fleet.

what measured
decode, one stream, greedy, coding questions of 400 tokens 59–61 tok/s (37 with the image's own decode step)
decode, one stream, temperature 0.7 56.6 tok/s
decode, four / eight streams at a time, greedy 95–106 / 130–143 tok/s for the batch, depending on the boot
prefill, ~15K-token prompts 2,400–2,500 tok/s
cold prefill of 130K / 260K / 500K tokens 61 / 142 / 348 s
decode behind 130K / 500K tokens of context 68 / 60 tok/s
context length 262,144 by default; 524,288 with the settings below
KV pool 793,000 tokens with the settings below; sized at each boot (493K–805K) with the defaults
tool calls, 283 cases over five wire formats, two trials 250 right, 20 soft, 13 failed (10 are tool_choice: none, which does not work; 3 slips on the sampled trial); 12 of 12 agent loops
boot to healthy about 5 min with a warm kernel cache, about 17 min the first time

Run it

You need four DGX Sparks with a RoCE fabric between them, the checkpoint on the local disk of every node (394 GB each), Docker with the NVIDIA runtime, and key-based ssh from the node you launch on (rank 0) to the other three. RECIPE.md has the whole list.

# On every node: the checkpoint and the image (arm64).
hf download thinkingmachines/Inkling-2-Small --revision b0e7afb8892a07897c78ce1c5b801a0d9cbd5e7b
docker pull lmsysorg/sglang:dev-inkling-2-small
#   The hooks were written against lmsysorg/sglang@sha256:424c1da97f6d6eef1612006c77b2ca791657a8184c1fbdbc682ebf9485d3b32c
#   (2026-11-12). If the tag has moved and a boot fails in a hook, pull that digest and set INKLING_IMAGE to it.

# On rank 0: this repository.
hf download aidendle94/Inkling-2-Small-DGX-Spark --local-dir inkling2-spark
cd inkling2-spark && chmod +x launch/*.sh bench/*.sh

# The site file: nodes, users, RDMA devices. Every line of the example says how to find its value.
cp launch/fleet.env.example launch/fleet.env

# Print the four docker run commands without starting anything (needs ssh to every rank).
launch/launch-inkling2.sh --dry-run

# Launch with the defaults (262,144 context, the pool sized at boot): copies patches/ to every node, starts the four
# ranks, waits for /health, arms the watchdog.
launch/relaunch-inkling2.sh

# First request (port 8210 on rank 0).
curl -s http://<rank-0>:8210/v1/chat/completions -H 'content-type: application/json' \
  -d '{"model":"inkling-2-small","messages":[{"role":"user","content":"Is 91 prime?"}],"max_tokens":400}'

# Read the text of overlapping requests. Do this after every boot of a changed configuration: a broken decode
# path looks fast and healthy in every counter.
mkdir -p logs && python3 bench/concurrency_check.py && python3 bench/reasoning_cache_check.py --expect on

With the FP8 draft cache, 524,288 tokens of context and the 793,000-token pool (what the author's fleet serves with since 2026-11-16; not the launcher's defaults):

CTX=524288 MEMFRAC=0.94 NCCL_EXTRA="-e SPARK_WMARK_BOOST=1 -e SPARK_DRAFT_KV_FP8=1" \
  EXTRA_ARGS="--max-total-tokens 793000 --swa-full-tokens-ratio 0.05 --prefill-decode-interval 2" \
  launch/relaunch-inkling2.sh

NCCL_EXTRA is the launcher's pass-through for extra -e NAME=value arguments of the containers. --swa-full-tokens-ratio 0.05 halves the cache of the 40 windowed layers, and --prefill-decode-interval 2 is a scheduling setting still under test. The cap goes with the settings: the same memory holds 716,000 tokens without SPARK_DRAFT_KV_FP8=1, and what fits depends on what the rank-0 node has free at boot.

To stop: kill the watchdog (its pid is in the last watchdog started line of ~/inkling2-watchdog.log), then launch/launch-inkling2.sh --stop.

Using it as a client

  • A chat request without a temperature is sampled at 0.7. For agent work and lookups in long contexts, send 0.
  • Send an earlier turn's reasoning back with the history (reasoning_content, or the thinking blocks on /v1/messages). The server restores it for clients that drop it, on chat completions and /v1/messages.
  • tool_choice: none does not work: leave the tools out of the request instead.
  • Give long-context turns 4,000 output tokens or more; the reasoning uses them.

Files

path what
launch/ the launcher (one container a node), the relaunch script, the watchdog, the example site file
patches/ the hooks; sitecustomize.py installs them at interpreter start in every rank
bench/ checks and measurements (text under concurrency, tool calls, long contexts, step time, kernel tests); each says in its docstring whether it is safe on a serving fleet
docs/results.md the lab notes
RECIPE.md, recipe.yaml the full recipe and its machine-readable summary

Licence

The files in this repository are MIT (LICENSE); adapted third-party parts are listed in NOTICE. The model weights are not included: they are Thinking Machines' and come under their own terms (Apache-2.0 and the acceptable use policy). The SGLang image is used unmodified under its own licences.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support