FINDINGS.md

Kimi-K3 on 2× A40 — measured findings

Everything here was measured on RunPod pod cumv30dk07dfdy: 2× NVIDIA A40 (2 × 46068 MiB), 93 GB container RAM, 16 container vCPU, 240 GB ephemeral disk, $0.92/h. Model: mmnga-o/Kimi-K3-REAP50-Width50-UD-gguf UD-IQ1_S, 181 GiB. Runtime: mmnga/llama.cpp branch kimi-k3-width-support, CUDA 12.8, sm_86.

Numbers without a measurement behind them are marked as estimates.

Throughput

Configuration	decode	prefill	static VRAM
autofit, 16 experts, greedy	2.94 tok/s avg	1.2–4.1 tok/s	87.7 GB
--cpu-moe --moe-cache on (unconfigured)	2.81 tok/s	0.9–2.4 tok/s	60.2 GB
autofit, 8 experts	3.15 tok/s	1.88 tok/s	87.4 GB

Prefill never exceeding decode by much is the tell: this deployment is not compute-bound on the GPU, it is dominated by whatever runs on the host.

Result: 7.3 tok/s

Step	decode	prefill	cost
starting point (previous effort, L40S)	2.32	—	—
2× A40 after the fixes below	2.94	1.2–4.1	$0.92/h
5× A40, identical placement	5.44	6.8–8.0	$2.20/h
5× A40, everything VRAM-resident	7.35	27.3	$2.20/h

3.2× on decode, 27× on prefill. Two independent factors, both of which this investigation initially got wrong: host vCPU (which RunPod scales with GPU count) and full VRAM residency. Neither is a tuning parameter.

Output quality at 7.3 tok/s: English prose and reasoning are correct ("The trip duration is 3.5 hours"), code is correct (fibonacci(n-1) + fibonacci(n-2)), and the model continues fluently in Chinese — Moonshot's other strong language. French generation stays broken, which is the Width50 build, not the hardware.

DSpark speculative decoding gives nothing. Once the draft finally loaded — after clearing four separate blockers — throughput was 7.32/7.33 against 7.33/7.35 without it, and llama-server exposed no draft_n / draft_n_accepted telemetry to explain why. The predicted floor of 1.2× was still too optimistic; the measured figure is 1.0×.

The answer: host vCPU, not VRAM

The same configuration, on the same model, using the same two GPUs — three further GPUs sitting idle at 267 MiB — runs at 1.9× the speed on a bigger host:

Pod	GPUs	vCPU	RAM	decode
cumv30dk07dfdy	2× A40	16	93 GB	2.31 / 2.98 / 2.88
okinf3ln9k8tsp	5× A40	41	234 GB	4.86 / 5.28 / 5.44 / 5.64

Prefill moves even harder: 0.99 → 8.01 tok/s.

RunPod scales vCPU with GPU count, so renting more GPUs buys host parallelism as much as it buys VRAM. Every placement experiment below kept 70–90 GB of weights being processed by 16 saturated threads, which is why moving that work between CPU and GPU changed nothing: the CPU side was the wall in all of them.

This is the mechanism the section below says was missing. It was found by accident — a 5-GPU pod picked up the 2-GPU placement script before the guard was published, and ran the identical configuration on a larger host.

Why the placement experiments all looked the same

~3 tok/s is the ceiling for this model on 2× A40, and the reason is not the one that seems obvious.

Throughput is invariant across configurations that differ enormously:

Configuration	VRAM used	attention	CPU-side bytes/token	decode
autofit, 16 experts	87.7 GB	48 layers on host	~33 GB	2.46 / 3.00
--cpu-moe + moe-cache	60.2 GB	all in VRAM	~4 GB	2.21 / 2.68
--n-cpu-moe 86	69.9 GB	all in VRAM	~4 GB	2.51 / 3.06
manual balanced -ot	87.8 GB	all in VRAM, 44.3/43.5 split	~4 GB	2.31 / 2.98
autofit, 8 experts	87.4 GB	48 layers on host	~17 GB	3.15
-ngl 0 (no GPU)	1.5 GB	all on host	all	0.65

The manual placement is the cleanest of these: explicit -ot ...=CUDA0/CUDA1 layer ranges fill both cards to 96%/94% with every layer's attention resident and only 72 layers' routed experts on the host. It took three calibration rounds, each guided by the shortfall rather than a guess — 45256 MiB over, then 1038 MiB over, then fitting. It performs exactly like everything else.

Cutting host traffic by 8× moved nothing. Halving expert compute moved +28%.

No validated mechanism. Three hypotheses were proposed and each was refuted by measurement:

1. Routed-expert CPU compute dominates — refuted: halving experts per token gave +28%, not the ~100% that would follow.
2. Attention re-read from host RAM dominates — refuted: --n-cpu-moe 86 puts every layer's attention in VRAM and cuts host bytes ~8×, and changed nothing.
3. CPU↔GPU boundary crossings dominate — refuted by comparing the two: autofit splits into contiguous GPU-then-CPU blocks (1–2 crossings per token) while --n-cpu-moe 86 crosses at 86 layers (~172 per token). If crossings dominated, autofit would be far faster. The two are within 2% of each other.

What stands is empirical, not theoretical: no rebalancing of CPU against GPU within 92 GB of VRAM improves anything, across placements spanning 60–88 GB of VRAM and 4–33 GB of host traffic per token.

The GPUs are not decorative, though — verified. Running with -ngl 0, so that essentially nothing sits in VRAM (1265 MiB and 269 MiB), gives 0.65 tok/s against 2.51 for the same prompt with autofit. The two A40s are worth 3.9×. That check was run specifically because, if the GPUs had contributed nothing, "buy more VRAM" would have been the wrong advice.

So the direction — fit more of the model in VRAM — is sound, and a ggml microbenchmark says the magnitude is large. test-backend-ops timing MUL_MAT_ID on the K3 expert shapes (MXFP4, 3584→3072) on an L40S:

 8 experts, C=1    29.25 µs
16 experts, C=1    59.83 µs
 8 experts, C=7   159.53 µs
16 experts, C=7   318.97 µs      ~7.73 TFLOPS

At 59.83 µs per MoE tensor and 92 MoE layers, the routed-expert work costs 5.5 ms/token when the experts are VRAM-resident — a ~180 tok/s ceiling from that term alone. We measure 333 ms/token, so GPU MoE compute is about 1.6% of the time; the other 98% is the host path.

That makes the naive linear extrapolation (5.4 tok/s at full residency) almost certainly far too pessimistic: full residency does not scale the existing regime, it deletes the term that dominates it.

What did NOT work, and why

Halving experts per token (16 → 8). Bought only +28% (2.46 → 3.15 on the same prompt) and destroyed the output — # # # # after six tokens. REAP had already cut 896 experts to 448 with the router calibrated for 16 active. The small gain also disproved the hypothesis that routed-expert CPU compute was the bottleneck: halving it would otherwise have nearly doubled throughput.

The CUDA MoE expert cache (llama-k3-moe-cache.patch, from the Code145 build). Applies cleanly to mmnga's branch once tests/ is excluded, compiles for sm_86, and --moe-cache appears in --help. Measured 14% slower on decode and roughly half the prefill. Caveat: that run set none of the seven GGML_CUDA_MOE_CACHE_* variables its own production launcher uses, so the verdict is against an unconfigured cache, not the design.

DSpark speculative decoding. Three separate blockers, each found only after clearing the previous one:

1. --draft-max was removed; the flag is now --spec-draft-n-max.
2. The published GGUF declares general.architecture = "dflash-draft"; llama.cpp registers LLM_ARCH_DFLASH as plain "dflash". Fixed by fix_dspark_arch.py (renames the architecture and its 30 namespaced KV keys, verified by reopening and comparing every tensor's shape, dtype and byte size).
3. The rewritten file still fails: "DFlash model requires 'target_layers' in GGUF metadata". The third-party converter also names tensors dflash.dspark.markov.w1 where llama.cpp expects markov_w1. The clean fix is to convert from RadixArk/Kimi-K3-DSpark with llama.cpp's own converter.

Expected ceiling if it ever loads: 1.2–2×, not the ~3× vLLM reports. On a MoE each drafted token routes to its own experts, so batch verification amortises only attention and shared experts, not the routed reads.

GGML_OP_OFFLOAD_MIN_BATCH=1. ggml can copy only the selected experts of an offloaded MUL_MAT_ID to the GPU instead of computing them on the host, but the path is gated on batch size and the default of 32 means it never fires during decode. Forcing it on halved throughput: 2.46 → 1.13 and 3.00 → 1.38 tok/s. With 16 experts across 68 offloaded layers that is 1088 separate small transfers per token, so it is latency-bound, not bandwidth-bound, and loses to host compute. The upstream default is right.

--n-cpu-moe at any rung (62/68/74/80). Never loaded. Tensor overrides set model_params::tensor_buft_overrides, which aborts autofit exactly like -ngl and --tensor-split do — and once autofit is off, neither --split-mode layer nor --tensor-split distributes anything: every allocation lands on device 1 while device 0 stays empty. Rung 80 missed by 135 MiB (46203 requested against 46068 available) with all attention plus twelve expert layers on one card. Autofit is the only mechanism that actually uses both GPUs; balancing manually requires explicit -ot ...=CUDA0 / =CUDA1 layer ranges.

TurboQuant / KV-cache quantisation. Not applicable: 69 of 93 layers are KDA recurrent and carry no growing KV, so the whole cache is about 0.8 GB.

ONNX custom ops from earlier work target Qwen3.5 on SM75, not Kimi-K3 on sm_86, and ONNX Runtime has no kimi_k3 architecture.

Output quality

English and code are genuinely usable; French is not.

"The capital of Japan is"  -> Tokyo, Washington, Stockholm, Berlin   correct
"train 14:00 -> 17:30"     -> "The trip takes 3 hours 30 minutes."   correct
"def invert(d):"           -> return {v: k for k, v in d.items()}    correct
French free-form           -> "cunce", "s'fotrenci", "zocneine"      non-words

French knowledge survives — "La Revolution francaise a commence en" continues "1789" — but French generation is corrupted at the sub-word level. The prime suspect is Width50: this build halves moe_intermediate_size from 3072 to 1536.

Sampler, six configurations on one prompt

Setting	Result
greedy, rp 1.0	facts correct, then hard repetition loop
greedy, rp 1.05 / last_n 64	no loop, but "Germany is PLN" — the penalty pushes it off correct answers while enumerating
greedy, rp 1.15 / last_n 256	drifts into SQL-flavoured nonsense
temp 0.3, min_p 0.05	fluent, hallucinates ("Tokyo Island", "Sumaru")
temp 0.6, top_k 20	degrades
temp 0.3, min_p 0.1, rp 1.05	fluent, no loop, nothing glaringly wrong — chosen default

Repetition penalty measurably reduces factual accuracy on this model. Clients wanting maximum accuracy should still ask for temperature: 0.

Deployment mechanics

common_fit_params sizes GPU placement against actually-free VRAM across all devices, but aborts the moment any placement parameter is pinned:

-ngl 99        -> "n_gpu_layers already set by user, abort"
--tensor-split -> "model_params::tensor_split already set by user, abort"

Both aborts produced the same failure: 84790 MiB shoved onto device 1 alone, cudaMalloc OOM, device 0 untouched. With no placement flags, autofit fills both cards to 95%. --cpu-moe and --n-cpu-moe do not abort it.

Boot time went from 14m28s to ~4 minutes by publishing the compiled binaries to the Hub (46 MB tarball, 2 s to extract vs 8m34s to compile). The artifact name pins base image, CUDA version and GPU arch so a mismatched tarball can never be picked up silently, and it is smoke-tested by running llama-server --version before it is trusted.

RunPod costs. A stopped pod's Volume disk bills at $0.20/GB/month, double the running rate — a 1600 GB volume cost $0.44/h doing nothing, which was 99% of one day's spend. Container disk is erased on stop but still billed, so for re-downloadable data the rule is: always Terminate, never Stop. A stopped pod is also pinned to its host and may refuse to restart while the same GPU type shows High stock elsewhere.

Kimi-K3-256K-REAP-Code145

Finalised during this session. All 96 shards were already on the Hub (347.5 GB); only config.json, the index and the tokenizer were missing, and finalize_code145.py — which downloads only metadata, never the weights — had never been run. The repo is now loadable.

num_experts             145
num_experts_per_token    16
moe_intermediate_size  3072      full width, unlike the 1536 we serve
source_index_keys    497 220
dropped_expert_keys  414 552
output_index_keys     82 668     = 497220 - 414552
renamed_expert_keys   80 040     = 92 layers x 145 experts x 6 tensors

Full width makes it the serious candidate for working French. It does not fit 2× A40 at MXFP4 (347.5 GB against 185 GB of fast memory); either 4× A40 at $1.80/h, or an imatrix-guided quantisation down to ~1.7 bit.

---

Session 2 — what actually bounds decode, and what long context costs

Decode is serialisation-bound, not bandwidth-bound

Six placements had moved 60–88 GB of weights between CPU and GPU without changing throughput, and no mechanism had survived measurement. The question none of them asked was whether the hardware is idle between tokens. It is.

Aggregate throughput against concurrent streams, 5× A40, 64 tokens each, distinct prefixes so no stream rides another's prompt cache:

1 stream     5.56 tok/s aggregate    5.56 per stream
2 streams    9.85 tok/s aggregate    4.93 per stream   1.77x
4 streams   14.53 tok/s aggregate    3.63 per stream   2.61x

Concurrency scales, so single-stream decode leaves silicon idle. The bandwidth arithmetic agrees independently: 7 GB of weights are touched per token, and at 138 ms/token that is **51 GB/s against 696 GB/s available** on the active card — 7% of the roof. llama.cpp splits layers across GPUs, so at batch 1 exactly one card computes while the other four wait.

This retires the last open question from session 1 and explains why every kernel-level optimisation attempt returned nothing: the expert GEMM was never the constraint.

Sustained decode on real output: 7.22 tok/s

The 7.3 tok/s headline was measured on selftests, three of whose five prompts produce degenerate repetition (# [CS_ah], et d'et d') — and a repetition loop decodes cheap. Re-measured on 600 tokens of coherent prose: 7.22 tok/s. The number holds; it is now measured on output worth having.

Prefill: 167 tok/s cold, ~free warm

Every selftest prompt is 6–22 tokens, so its "prefill tok/s" is per-request overhead and says nothing. Measured properly, 3408-token prompt:

cold   3408 tok in 20.4 s = 167 tok/s     (23x the decode rate — prefill batches)
warm      4 tok in  0.53 s                (slot prompt cache reprocessed only the delta)

For a Claude Code client sending a 32k system prompt: ~3 min once, then only the changed tokens on every later turn.

Long context is nearly free on this architecture

From the config, not from assumption: kv_lora_rank 512, qk_rope_head_dim 64, full_attn_layers every 4th layer, max_position_embeddings 1048576.

24 of 93 layers cache at all (MLA); the other 69 are KDA — recurrent state,
constant regardless of context length.

KV per token = 24 x (512 + 64) x 2 = 27.6 KB

  65536 ctx ->  1.8 GB      262144 ->  7.2 GB      1048576 -> 29 GB

Against ~38 GB left free by the weights on a 5-GPU host. 16384 was never a hardware limit, just an untested default — and it could not accept a single Claude Code request. What constrains context here is the weights sharing the cards, never the cache.

Cost reality check

Runpod hosts an official Kimi-K3 endpoint (moonshot-kimi). Measured: 300 tokens in 10.281 s, billed $0.004797.

                    speed         $/1M output    quality
official          29.2 tok/s          ~16        full precision, French works
ours (1 stream)    7.2 tok/s          ~85        1.14 bpw, French broken
ours (4 streams)  14.5 tok/s          ~42

The official endpoint is 4x faster and 5.7x cheaper. Self-hosting buys privacy, no per-token billing, and context control — not price or quality. Worth restating whenever the effort seems to justify itself on cost.

No smaller GGUF exists

Ours is the most compact Kimi-K3 published, because it is the Width50 variant (moe_intermediate_size 1536 rather than 3072) — which is also why French is broken.

ours   REAP448 Width50 IQ1_S       181 GiB
       0xTank REAP568 UD-IQ1_S     246 GiB
       mmnga-o REAP50 UD-IQ1_S     305 GiB
       prometheusAIR REAP55 IQ1_M  319 GiB
       hellohazime REAP640ja       411 GiB

A "light" build for a single A40 (46 GB, ~37 GB of weights after a 256k KV) does not exist and would have to be produced — roughly a 64-expert REAP.

Under $1 per million tokens is not reachable with K3

At $0.44/h per A40, $1/1M output requires 122 tok/s aggregate per card. The 5× pod at $2.20/h would need 611 tok/s; it delivers 14.5. That is a 42x gap, not a tuning problem.

What does reach it, from HF configs (throughput figures are bandwidth-derived estimates, not measured):

Qwen3-Coder-30B-A3B  30B/3B active  ~30 GB FP8  KV 12 GiB@256k  1x A40  ~$0.10-0.25/1M
Qwen3-Coder-Next     80B/3B active  ~74 GB FP8  KV 12 GiB@256k  2x A40  ~$0.20-0.80/1M
Qwen3.8-27B          dense          ~26 GB FP8  KV 32 GiB@256k  256k does not fit on 1 card

Served by vLLM, not llama.cpp — tensor parallelism and continuous batching are exactly what the concurrency measurement above shows llama.cpp lacking.

Operational trap: bare wait kills the hot-reload watcher

llama-server is started with &, so a bare wait anywhere later in the script waits for it too, forever. The server keeps serving, the selftest never finishes, the watcher never starts, and the pod silently ignores every published update. Symptom: /props reports the old config long after a push. Always wait "$pid" with an explicit pid.

---

Session 3 — 256k validated, and a measured alternative that dominates it

Kimi-K3 at 262144 context: it works, and it costs nothing in decode

--ctx-size 262144, all-VRAM on 5x A40, 39.5-42.5 GB used per card:

n_ctx        262144        (was 16384 -- never a hardware limit, just a default)
decode          7.25 tok/s (unchanged from 16k: 7.2-7.3)

16x the context for no throughput cost, exactly as the KDA/MLA geometry predicted. The DSpark rungs OOM'd at this context and the ladder correctly fell through to all-VRAM without a draft — no loss, DSpark was measured at zero gain.

Prefill degrades gracefully, and the proxy times out before it finishes

A cold 38k prefill takes ~230 s. Runpod's HTTP proxy sits behind Cloudflare, which cuts the connection at ~125 s (error code: 524). The workaround is to build the prefix in chunks — each request extends the cached prefix — which also measures the degradation curve a single call would have hidden:

chunk 1  +8400 tok @  8k ctx   228.6 tok/s
chunk 2  +8404 tok @ 17k ctx   203.6 tok/s
chunk 3  +8404 tok @ 25k ctx   182.4 tok/s
chunk 4  +8605 tok @ 34k ctx   164.1 tok/s
chunk 5  +8704 tok @ 43k ctx   149.8 tok/s

-34% across 5x the context, far better than quadratic — 69 of 93 layers are KDA (linear), only 24 are MLA. 42,517 tokens prefilled in 3 min 55 total. Re-submitting a cached 32k prompt returns prompt_n=4 in 0.6 s.

vLLM's Kimi-K3 support is real and unreachable

vLLM and SGLang both ship day-0 K3 support, including the two things that failed here: a hybrid prefix cache over recurrent KDA state, and working DSpark block-diffusion speculative decoding. None of it is usable on A40s, because every vLLM-loadable K3 checkpoint is the full unpruned model:

nvidia/Kimi-K3-NVFP4        1499 GiB   + requires Blackwell (sm_100)
RedHatAI/Kimi-K3-NVFP4      1533 GiB   + requires Blackwell
RedHatAI/Kimi-K3-FP8-BLOCK  2626 GiB   + requires sm_89+

No AWQ, no GPTQ, no W4A16. The 181 GiB IQ1_S we run is llama.cpp-only, and llama.cpp is exactly what caps us. The unlock would be quantising a REAP'd K3 to W4A16 for vLLM — Code145 at 4-bit lands near 260 GB (6x A40 / 4x A100-80). That build, not any config change, is the only path to the good engine.

The measured alternative

Qwen3-Coder-30B-A3B-Instruct, AWQ 4-bit (16.9 GiB), vLLM 0.27.1, one A40 at $0.44/h, 262144 context with fp8 KV. Everything below is measured, not derived:

concurrency   tokens   duration   aggregate      per stream   $/1M output
     1          128      1.6 s      77.6 tok/s      77.6        $1.575
     4          512      1.9 s     265.3 tok/s      66.3        $0.461
     8         1024      2.1 s     496.8 tok/s      62.1        $0.246
    16         2048      2.6 s     782.4 tok/s      48.9        $0.156
    32         4096      3.5 s    1165.6 tok/s      36.4        $0.105

prefill  38,494 tok cold in 10.42 s = 3693 tok/s;  warm 1.07 s

Head to head:

                        Kimi-K3 5x A40   Qwen3-Coder 1x A40   K3 official API
hourly                      $2.20              $0.44           per-token
decode, 1 stream          7.2 tok/s         77.6 tok/s        29.2 tok/s
prefill @38k              ~165 tok/s        3693 tok/s             --
32k prompt, cold           3 min 55            10 s                --
context 262144            5 cards            1 card                --
$/1M output                  ~85         $1.58 .. $0.105          ~16
French                      broken            correct            correct

Qwen is 10.8x on decode, 22x on prefill, on one fifth of the hardware, and it answers in French. The under-$1/1M target is met from two concurrent streams and reaches $0.105 at 32. Kimi-K3 self-hosted on A40s is not competitive on any axis measured here; it remains interesting only where the specific K3 model is the requirement.

Two mistakes worth not repeating

• Creating a pod from create-pod without a start command: the schema has no args field, so the container starts with the image default. Use create-template with dockerStartCmd, then deploy with templateId.
• --disable-log-requests was removed in vLLM 0.27; it crash-loops the server. A pod's args cannot be edited after creation — only the template can — so a bad flag costs a full recreate.

The context ceiling on 5x A40, and why q8 KV is not the way out

ctx      16384    7.2-7.3 tok/s
ctx     262144    7.25 tok/s
ctx     524288    7.29 tok/s     <- ceiling, all-VRAM, no draft
ctx    1048576    OOM: 29 GB of f16 MLA cache does not fit alongside the weights
KV q8_0            rejected outright, at every context

Decode is flat across a 32x range of context. That is the KDA/MLA hybrid doing exactly what it is for: only 24 of 93 layers cache at all, and the other 69 hold a recurrent state whose size is independent of context.

q8_0 KV cannot be used here, and the failure signature says why. Every placement rung failed in 1.6-2.5 s with failed to create llama_context from model and no cudaMalloc line — a parameter rejection. Contrast the genuine OOM at 512k on the draft rung: 29.9 s, with cudaMalloc failed: out of memory printed. Fast failure with no allocator message means llama.cpp refused the configuration; slow failure with one means it tried and ran out. The 69 KDA layers carry a recurrent state llama.cpp will not quantise, so f16 is forced and the ceiling is set by memory: 512k fits, 1M does not.

TurboQuant (Google, ICLR 2026) would take the cache to ~3 bits and put 1M within reach, but as of this run it is a paper with an open SGLang feature request (#21618) and no merged implementation in any engine.

Qwen decode against context length

short context   77.6 tok/s
119k context    34.3 tok/s      (prompt served warm from prefix cache)

Halves across 119k, and still 4.7x Kimi's short-context rate. Tool calling was verified on the OpenAI endpoint: two parallel calls, correct arguments, finish_reason: tool_calls.

Runpod's HTTP proxy caps a single request at ~125 s

Cloudflare sits in front of the pod proxy and returns error code: 524. A cold prefill longer than that cannot complete in one call — measured at ~40k on Kimi and ~232k on Qwen. Chunk the prompt (each request extends the cached prefix) or expose a TCP port instead.

---

Session 4 — speculative decoding, and why the regime decides everything

n-gram speculative decoding on Qwen3-Coder-30B-A3B-AWQ, vLLM 0.27.1, 1x A40, --speculative-config '{"method":"ngram","num_speculative_tokens":5,...}'. Same model, same hardware, same quant, two pods running side by side so the comparison is simultaneous rather than sequential:

A. pure generation (no prompt/output overlap)
     baseline    102.21 tok/s
     ngram        58.49 tok/s      -43%

B. code editing (output re-emits the input)
     baseline    130.16 tok/s
     ngram       242.39 tok/s      +86%

Speculation is not a free win — it is a bet on repetition. With nothing to guess, every draft is rejected and the verification cost is pure loss; that is the -43%. Claude Code lives almost entirely in regime B (read a file, emit a modified version, repeat identifiers), so it is the right default there and the wrong default for prose.

vLLM's telemetry, aggregated over both regimes:

drafts 207   draft tokens 1035   accepted 706  = 68.2%
accepted per position: 179 / 156 / 131 / 120 / 120
mean accepted length 3.41  ->  ~4.4 tokens emitted per forward pass

This is the measurement llama.cpp never produced: DSpark there exposed no draft_n at all and moved throughput by 0.0%. The mechanism was never broken — the engine was.

The number that matters for the original goal

Single stream, code editing, one A40 at $0.44/h:

242.39 tok/s  ->  $0.50 per 1M output tokens

The under-$1/1M target is met on a single stream, without needing concurrency to amortise anything. For reference the same target on Kimi-K3 required 611 tok/s aggregate against 14.5 measured.

Note also that the baseline itself reads higher here (102-130 tok/s) than the 77.6 measured earlier: that earlier figure was a 128-token request whose wall clock was dominated by per-request overhead. Longer outputs amortise it. Quote 77.6 for short replies and ~130 for sustained generation.

---

Session 5 — Kimi-Linear-48B-A3B: genuine Kimi, 1M context, one A40

cyankiwi/Kimi-Linear-48B-A3B-Instruct-AWQ-4bit, 28.4 GiB, vLLM 0.27.1, one A40 at $0.44/h. KimiLinearForCausalLM, model_type: kimi_linear — the same architecture family as K3, with the same MLA geometry (kv_lora_rank 512, qk_rope_head_dim 64).

Why it fits where K3 could not

27 layers = 7 MLA (full_attn_layers [4,8,12,16,20,24,27]) + 20 KDA
KV/token  = 7 x (512 + 64) x 2 = 7.88 KB      (K3: 24 layers -> 27.6 KB)

weights AWQ4   28.4 GiB
1,048,576 ctx   7.88 GiB KV   =>  36.3 GB on a 46 GB card

vLLM confirmed it at boot: GPU KV cache size: 1,416,566 tokens on a single card. K3 needed five A40s at $2.20/h to reach 524,288.

Measured

flux    aggregate   per stream   $/1M
  1        67.4        67.4      1.814
  4       206.6        51.6      0.592
  8       352.4        44.0      0.347
 16       556.1        34.8      0.220
 32       779.9        24.4      0.157

prefill cold  6162 tok/s     (better than Qwen3-Coder-30B's 3957)
prefill warm  6430 tok/s     ratio 1.04 -> NO prefix caching
code editing  89.4 tok/s
tool calling  2 parallel calls, correct arguments
code          correct;  French fluent

n-gram speculation is BROKEN on this backend

Rejection sampling is supposed to be lossless, so any output difference between speculative and non-speculative decoding is a bug. Here it corrupts code, reproducibly and identically across runs:

                with speculation                    without
add()     return a -b):\n    return a -      return a - b\n\ndef multiply(a,
fibonacci fibonacci(n-1(n-1) + fibonacci(n-2) fibonacci(n-1) + fibonacci(n-2)

The signature is a just-emitted fragment re-spliced at the wrong place — a bad draft accept. Plain prose and verbatim copying stayed perfect, which is why it took targeted probes to find: code is dense in the short repeats (parentheses, operators) that n-gram matching latches onto.

It is also 3-4x slower, because vLLM disables CUDA graphs under speculation on TritonMLABackend:

flux      with spec   without    ratio
  1          23.1       71.2      3.1x
  4          58.7      249.6      4.3x
 32         203.9      781.0      3.8x
$/1M @32     0.600      0.157

Turn speculation off for Kimi-Linear. Worth reporting upstream.

Two fixes that made it work at all

• Tokenizer. tokenization_kimi.py imports bytes_to_unicode from transformers.convert_slow_tokenizer, removed in transformers >= 5.5.3 which every vLLM >= 0.24 requires. No Kimi-Linear repo ships a fast tokenizer.json, so the Python tokenizer is mandatory. Fixed by publishing patdev/kimi-linear-tokenizer-fix: same files with a local fallback definition, verified byte-identical to the transformers reference before deployment. Point vLLM at it with --tokenizer.
• Tool parser. hermes yields zero calls. The chat template uses <|tool_calls_section_begin|> / <|tool_call_begin|>, the K2 token scheme — so the parser is kimi_k2. (kimi_k3 is for <|open|>/<|close|>/<|sep|> and does not apply.) It also sets skip_special_tokens=False, which those markers need.
• Headroom. --gpu-memory-utilization 0.93 OOMs at the very last step, in the speculative rejection sampler asking for 160 MiB with 143 MiB left. 0.90 leaves room and still yields >1M tokens of KV.

Head to head

                    Kimi-Linear 48B    Qwen3-Coder-30B
hourly                   $0.44              $0.44
decode 1 stream        67-71 tok/s        77.6-79.2 tok/s
aggregate @32            780               1166
prefill cold            6162               3957
prefill warm            6430              63315
context              1,048,576            262,144
$/1M @32                0.157              0.105
tool calling            yes                 yes
French                  fluent             correct
brand                GENUINE KIMI          Qwen

Kimi-Linear wins on context (4x) and cold prefill (1.6x). It loses on decode and, decisively for an agent client, on prefix caching: 6430 vs 63315, ten times worse, because vLLM does not yet support prefix caching over recurrent hybrid state. A 32k system prompt costs ~5 s every turn instead of ~0.5 s.

Qwen3-Coder-Next 80B: no data

Two attempts, both infrastructure failures, zero requests served.

attempt 1   hung 22 min in NCCL peer-to-peer init on 2x A40
attempt 2   NCCL_P2P_DISABLE=1 cleared that, then looped ~10 min on
            "No available shared memory broadcast block found in 60 seconds"

NCCL_P2P_DISABLE=1 is required for tensor parallelism on Runpod A40 pairs. The second hang is unproven — the hypothesis is torch.compile on a 512-expert MoE, testable with --enforce-eager. Its boot log did confirm one thing: enable_prefix_caching=False, the same hybrid limitation as Kimi-Linear.