|
Download docs/prefill-decode-optimization-playbook.md from staccs/ds4-winhip: direct link, hf CLI and curl.
- Browser
- Download file 31.8 kB
-
https://huggingface.co/staccs/ds4-winhip/resolve/main/docs/prefill-decode-optimization-playbook.md
- Command line
-
hf download hf://staccs/ds4-winhip/docs/prefill-decode-optimization-playbook.md
-
curl -L -o prefill-decode-optimization-playbook.md https://huggingface.co/staccs/ds4-winhip/resolve/main/docs/prefill-decode-optimization-playbook.md
31.8 kB
| # Prefill and decode optimization playbook | |
| What the 2026-09 optimization campaigns on this engine taught, written down | |
| so the next campaign (another family, another host) starts from the method | |
| and the failure modes instead of rediscovering them. Everything below was | |
| measured on one DGX Spark (GB10, `sm_121`, 128 GB unified memory, ~240 GB/s | |
| LPDDR5X, ~90K random 4 KiB IOPS from the NVMe) with mixed-quant MoE models | |
| served from a resident VMM weight owner. The numbers are quoted so the | |
| weight of each lesson is visible; they do not transfer to other hardware, | |
| only the reasoning does. | |
| Campaigns this distills (all in `docs/` and `scratch/`): | |
| | campaign | model | result | | |
| |---|---|---| | |
| | 2026-09-01 / 09-02 prefill (9 + 5 rounds) | Qwen3.8-Flash-Next Q5 + SSD-PLE | 8K prompt 497 -> 604 -> 911 tok/s | | |
| | 2026-09-02 / 09-03 decode (8 rounds) | same | 93 -> 52 ms per MTP step, 18 -> 31 tok/s | | |
| | 2026-09-04 long context (5 rounds) | same | cold 64K 530 -> 1371 tok/s, cold 196K 1052 -> 1288 | | |
| | 2026-09-05 K2-Horizon-375B MQ87 (12 prefill + 4 decode rounds) | K2 IQ1/IQ2 experts | 8K prefill 287 -> 645 tok/s (+125 %), decode 5.5 -> 13.3 tok/s | | |
| | 2026-09-06 prefill (3 rounds) | Qwen | cold 8K 1215 -> 1362 tok/s (+12 %), 64K +4 % | | |
| | 2026-09-06-r4 (D2R K=2560 + HC q8 + o_proj K=6144) | Qwen | cold 8K 1353 -> 1432 tok/s, 64K 1431 -> 1555 | | |
| The one-line version: **almost every large win came from removing work the | |
| graph did not need (idle time, fallbacks, repeated transformations, redundant | |
| round trips), and the kernel-tuning rounds came last and paid least.** This | |
| is the order the engine's `AGENTS.md` prescribes and the campaigns confirmed | |
| it round after round. | |
| --- | |
| ## Local scout with ds4-perf | |
| Use the [ds4-perf guide](ds4-perf.md) for the four-command workflow, | |
| optional NVIDIA NVTX/CUPTI builds, workload contracts, bounded NCU capture, | |
| comparison gates, and automatic experiments. The production annotations remain | |
| `ds4.prefill` and `ds4.decode` at the measured Rust operations. | |
| Nsight reports diagnose execution structure; only fresh unprofiled samples | |
| support throughput comparisons. Keep the original model, prompt, token counts, | |
| cache protocol, and intended weight owner. Record prefill and decode separately. | |
| The heuristics prioritize host/idle, memory operations, short-launch fragmentation, | |
| and dominant kernels. Missing or ambiguous data stays unknown. GPU coverage is | |
| an interval union inside the Rust phase; summed kernel durations may overlap. | |
| GB10 validation (2026-09-07): Qwen MQ-Q5 SSD-PLE BF16, 2,048-token cold frontier | |
| and eight greedy tokens, three fresh unprofiled runs per build. All 248,320 | |
| frontier logits and token IDs were byte-identical. Optional NVTX changed median | |
| prefill from 1247.86 to 1246.19 tok/s and decode from 25.22 to 25.11 tok/s | |
| (-0.13% / -0.44%); this short smoke does not establish broader performance. | |
| Scout found prefill `HOST / IDLE` (73.1% GPU coverage) and decode `DOMINANT KERNEL` | |
| (`mul_mat_vec_q`, 89.0% coverage). Both range names and phase kernel totals were | |
| checked against a separate Nsight export of the raw report. | |
| Existing `speed-bench/` scripts remain valid for specialized server and | |
| deep-context profiling. | |
| --- | |
| ## 1. Method: one round | |
| A round is one change, measured against the previously adopted binary under | |
| identical conditions, kept only if the gain is stable and the numerics are | |
| accounted for. Rounds that skipped a step produced the wrong conclusion | |
| every time they did. | |
| 1. **Profile the whole request, not a kernel.** `nsys --trace=cuda` on the | |
| production shape (an 8K cold prefill, a 64K cold prefill, a decode window | |
| of ~10 tokens). Export to sqlite and reduce it yourself: per chunk, GPU | |
| busy vs idle, the *gap before each kernel* (idle is attributed to the | |
| kernel that was waiting), then kernel totals with launch counts and | |
| average duration. `scratch/qwen-prefill-opt-20260906/{nsys_chunks.py, | |
| kern.py}` are the two scripts; they run in seconds. Rank gaps and kernels | |
| together. A 1.06 s gap before the PLE gather kernel out of a 6.6 s | |
| prefill was the biggest single item of the last Qwen campaign and no | |
| kernel view shows it. | |
| 2. **Attribute before designing.** Launch counts and average durations tell | |
| you whether a bucket is one slow kernel or many small ones (334 dense MMQ | |
| launches per chunk, 514 quantize launches, 387 sanitize passes). For a | |
| kernel, `ncu` on a model-free probe (the fixtures have `DS4_*_PROFILE_*` | |
| modes that launch the production shape on synthetic weights) gives the | |
| stall reasons; on this host `ncu` runs only with the weight owner down. | |
| 3. **Change one thing.** Two changes in one round cost a full re-measure | |
| when one of them regresses (the 09-06 round 3 folded a SwiGLU epilogue and | |
| a fused sum together; the epilogue lost 0.7 % and hid the sum's +1 % | |
| until a kill-switch run separated them). | |
| 4. **Prove the numerics on a fixture first** (section 5), then run the | |
| full-model gate. Fixtures are fast (seconds) and catch layout bugs the | |
| full model only shows as an illegal address at boot. | |
| 5. **A/B on fresh processes, same hour, medians of three.** One fresh | |
| worker per run against the resident owner, cold KV directory, the same | |
| corpus slice, `thinking` disabled, non-streaming so `usage`/`timings` are | |
| present. Throughput drifts 1-4 % between hours on one binary (thermal, | |
| page cache, owner state), so the baseline is re-run in the same session | |
| whenever a difference is under ~2 %. Never build while timed runs are | |
| on; the compile steals the cores the page workers use. | |
| 6. **Keep a kill switch** for every new execution path (`DS4_*=0/1`, read | |
| once per process). It is the attribution tool: same binary, path on vs | |
| off, and it is what turns "the text changed" into "the text changed | |
| because of X" (section 5.4). | |
| 7. **Commit with the evidence**: what was slow and by how much, why, the | |
| contract (bit-identical / fp32 reorder / value parity), the A/B numbers | |
| with the shape and corpus, and the `Tests:` that actually ran. | |
| What counts as the metric matters as much as the protocol: | |
| - Prefill: tokens per second of a **cold** prefill of a fixed shape. A | |
| repeated prompt on a warm worker is not cold; the SSD-PLE page cache and | |
| the disk-KV cache both hide the cost you are trying to measure. Say which | |
| corpus: a prompt whose n-grams hit the page cache and one whose n-grams | |
| miss it differ by 30 % on the same binary. | |
| - Long context: the single-shot `--ctx-start N --ctx-max N` shape (one cold | |
| N-token prefill, the production TTFT of a long request) and the | |
| incremental sweep (2K steps on a warm session) answer different questions; | |
| the sweep cannot exercise the next-chunk lookahead because the next chunk | |
| is not known. | |
| - Decode: **milliseconds per step**, not tokens per second, whenever | |
| speculative decoding is on. tok/s = tokens-per-step / ms-per-step, and | |
| tokens-per-step moves with the draft acceptance rate, which moves with the | |
| text, which moves with any fp32 reorder (a near-tie token flips ~25 tokens | |
| into the standard prompt). 93 -> 52 ms per step is a kernel result; | |
| 18 -> 31 tok/s is that plus acceptance luck. | |
| --- | |
| ## 2. Prefill: where the time went and how it was removed | |
| Ordered as `AGENTS.md` orders it: unexpected fallback paths, repeated | |
| transformations, redundant memory traffic, routing and indexing overhead, | |
| launch fragmentation, fast-path utilization, and only then kernel tuning. | |
| ### 2.1 GPU idle time is the first target | |
| If the profile shows idle, no kernel work matters until it is gone. | |
| - **Find the wait, then the bug behind it.** The 09-04 trace put the GPU | |
| idle 3.8-12 s per 8K chunk before the PLE gather kernel. The obvious | |
| reading ("SSD is slow") was wrong: the page cache's victim scan stopped at | |
| the *first* evictable way instead of the least recently used one, so a | |
| prefetch's pages were evicted by the next request for the same set and a | |
| chunk re-read a third of its pages at gather time. True LRU + 16 ways | |
| took the 64K wait from 59.5 s to 3.8 s (530 -> 1163 tok/s). More I/O | |
| workers had made it *worse* (64 workers: 339 tok/s) because faster | |
| completion produced more READY victims for the broken scan. A knob that | |
| makes things worse when turned up is a bug, not a limit. | |
| - **Overlap I/O with compute at the granularity the pipeline allows.** The | |
| next chunk's pages are queued right after this chunk's rows are gathered | |
| (layer 1), so 47 layers of compute hide the reads (the "lookahead"). | |
| Cache sizing follows from the overlap: one chunk's working set per | |
| in-flight stream, two for two alternating banks. | |
| - **The first chunk has nothing to hide behind.** Opening every prompt | |
| with a short chunk (2,048 rows) exposes only its own reads and hides the | |
| full-size chunk behind it: cold 8K +9.1 %. Two boundaries: a short | |
| *trailing* chunk is a net loss (a 256-row chunk ran its 48 layers at low | |
| occupancy and hid few reads: -7 % on a 2,304-token prompt), so the split | |
| applies only when the trailing chunk is at least as long as the opening | |
| one; and on prompts whose pages are already cached the opening chunk costs | |
| its own inefficiency (~1.7 %). Chunk-size changes also change GEMM shapes | |
| (cuBLAS algorithm choice) and therefore near-tie tokens; that is expected. | |
| - **Queue depth is not the lever when the pipeline is.** A Linux-AIO | |
| rewrite of the page workers (depth 32) lost 14 % on the cold-PLE prompt | |
| although the raw device scales with depth: deeper queues made the | |
| head-of-line rows the gather waits on arrive later. Fix ordering | |
| (consumption order, targeted wake-ups) before adding concurrency. | |
| ### 2.2 Fallback paths: the biggest single wins | |
| A quantization format or a shape that the fast tier does not admit falls to | |
| a generic kernel, and the fallback is usually 2-10x slower than the tier it | |
| missed. Read the engine's "engaged path" log lines and grep the dispatch | |
| predicates before touching any kernel. | |
| - **Rectangular schedules over ragged work.** MoE prefill routes ~5-20 rows | |
| per expert; a `[expert, max-bucket]` grid launches mostly empty tiles. A | |
| compact worklist (one persistent kernel, grid = SM count, tiles of | |
| 8/16/32/64/128 columns chosen per expert bucket inside the kernel) is the | |
| right shape: K2 IQ2_XXS down 12.9 -> 7.0 ms, IQ1_S gate/up 9.8 -> 5.0 ms, | |
| IQ1_M gate/up from a per-token MMVQ loop to an MMQ tile +55 % end to end. | |
| - **Shapes that are "almost" the tile.** An expert-down with K = 640 is not | |
| a multiple of the 256-wide K-quant super-block, so the recipe stores a | |
| 512-column main + 128-column tail and the engine ran the tail as a separate | |
| F32 accumulate. Folding the tail into the MMQ K loop as one extra | |
| half-iteration (+14 %) and then running the dense K = 640 shared-expert | |
| down through the same fused-tail entry as a one-expert MoE (+17 %) were | |
| the two largest prefill rounds of the 09-02 campaign. When a dense | |
| operation can be expressed as a routed one with an identity map, the | |
| routed tier's kernels are usually the better ones. | |
| - **Every projection on the quantized-activation tier.** PLE key/value and | |
| QSA k/v pairs ran on Q8_0 DP4A kernels while every other projection used | |
| the Q8_1-activation MMQ tier; moving them (+11 %) changed numerics within | |
| the MMQ tolerance, so this is an arithmetic round with a contract, not a | |
| scheduling round. | |
| - **Chunk size is a dispatch parameter.** K2 at 512-token chunks saw ~21 | |
| rows per expert and 16x the per-chunk launches of an 8K prefill; 1,024 | |
| doubled the rows per expert (+20 %). It also changed the greedy IDs (56 | |
| of 64), which is why it was rejected once and adopted only with the | |
| revised gate (section 5.3). | |
| - **Tiers that exist but are never entered.** Qwen's dense Q8 projections | |
| missed the aligned D2R tier because the weight owner's repack rule | |
| admitted only `dims[0] % 1024 == 0` (K=2560 qkv/z/q). Admitting the | |
| D2R prefill contract moved them: cold 8K +2.8 %, 64K +5.1 %. o_proj | |
| (K=6144) already had the artifact via `%1024` and still missed the | |
| dispatch cap of 4096 (kept for DeepSeek K=8192, 0.81x vs mmq); raising | |
| the cap to 6144 only: +1.3 % / +2.5 %. The "engaged path" log line | |
| for a tier is worth checking against the tensor list once per family. | |
| ### 2.3 Repeated transformations | |
| The same activation quantized, converted or permuted more than once is pure | |
| waste and usually bit-identical to remove. | |
| - Pair kernels that share one expert map and one Q8 activation for gate/up | |
| (K2 +4 %, Qwen +11 % in combination with the worklist). | |
| - Quantize once per **token**, not once per assignment slot: the routed | |
| gate/up quantize gathered every token row top-k times (0.84 GB per 8K | |
| layer). Quantize the compact rows and scatter the 144-byte Q8 blocks into | |
| the expert-sorted layout the tiles stream (+1 %; see 2.7 for the variant | |
| that lost). | |
| - Hyper-connection rows converted to BF16 once by the norm instead of per | |
| consuming GEMM (+3.3 %), later the norm storing only BF16 rows plus one | |
| scale per lane, with F32 recomputed from the hyper input where needed. | |
| - The Qwen HC mix is one F32 tensor consumed by qkv+z or q+index; emitting | |
| its Q8 once through the existing norm-q8 registry retired the per-GEMM | |
| quantize (bit-identical, +0.8 % / +0.9 %). The D2R preq entry had to | |
| accept the same K % 128 as the launch; a leftover %1024 gate kept the | |
| published buffer unused. | |
| - Counting-sort expert id maps instead of the warp-scan builder | |
| (bit-identical, +7 % with the sanitize drop). | |
| - Trigonometry tables: K2's QK-norm/RoPE kernel computed `pow/fmod/cos/sin` | |
| in double per (head, pair) per launch; on GB10 FP64 runs at 1/64 rate. One | |
| (cos, sin) table per chunk position: 40 ms per 512-token chunk gone, | |
| 12 us per decode launch gone. | |
| ### 2.4 Redundant memory traffic | |
| At ~240 GB/s every [rows x hidden] F32 round trip at 8K rows costs ~0.35 ms | |
| per read or write; a MoE layer had a dozen of them. Count bytes per layer | |
| before and after; the trace's kernel time for bandwidth-bound kernels is | |
| bytes / 240 GB/s within 10 %. | |
| - **Sanitize passes.** A standalone non-finite pass over every MMQ output | |
| is a full read + write; when every consumer already zeroes non-finite | |
| values at read (`moe_sum`, the weighted SwiGLU), the pass is redundant. | |
| K2 Prefill 9 removed 58 launches per chunk; after D2R retired its own | |
| outputs Qwen still carries 303 dense ones on a 6,144-row chunk (0.8 %). | |
| - **Pack copies.** The expert-down read its 512-column input from a packed | |
| copy of the 640-wide SwiGLU rows although the same entry already read the | |
| 128-column tail in place with a row stride. Reading both halves in place | |
| removed 378 MB per layer (+0.8 %). Whenever a kernel takes a stride, | |
| strided reads are free; a pack kernel is a smell. | |
| - **Fused epilogues on elementwise chains** (safe class): `moe_sum` + | |
| shared-expert sigmoid gate + residual add in one kernel (two round trips | |
| per layer, +1 %); residual + next norm in one kernel; SwiGLU producing the | |
| quantized down input directly where the layout allows. | |
| - **Keep F32 intermediates out of memory.** The hyper-connection mix used | |
| to write [rows x 10240] F32 logits from a cuBLAS GEMM and re-read them in | |
| a mix kernel (~3 GB per sub-layer); forming the mix-up logits on the | |
| tensor cores inside the mix kernel and applying the sigmoid in the | |
| epilogue cut it to ~1.7 GB (+7 %). The 1.6 GB per-bank QSA score scratch | |
| went the same way: a fused attention kernel with an online softmax never | |
| writes scores (104 + 64 -> 67 ms per layer). | |
| - **Recurrent state in registers.** The GDN recurrence did two loads and a | |
| store per state element per token; keeping 32 rows per thread in registers | |
| for the whole chunk halved the kernel (33 -> 17 ms), with `fmaf` written | |
| explicitly so the contraction matched the old code bit for bit. | |
| ### 2.5 Launch fragmentation | |
| Per-launch cost on this host is ~5-10 us of GPU gap plus host time; a Qwen | |
| layer had ~62 launches at decode and an 8K prefill chunk ~4,000. Pairs | |
| (2.3), persistent worklists (2.2) and fused epilogues (2.4) all cut launches | |
| as a side effect; the rounds that targeted launches directly: | |
| - One launch per layer for the routed vec tier instead of one per token | |
| (Qwen R15, -7.6 % per decode step) — a per-token split introduced for | |
| row-stability had silently split the assignment-major down call too. | |
| - Paired 320-row and 4-row BF16 projections (R12), warp-parallel router | |
| selection replacing a serial kernel (R13). | |
| - Decode attention over the whole context in one launch was the wrong | |
| direction: split-K across the context (section 3.4) adds a combine launch | |
| and still wins 78 % because the single launch used 1 of 48 SMs. | |
| ### 2.6 Kernel tuning, last | |
| Only after the graph is clean. The signals that led to each tuning round: | |
| - **Shuffle-bound reductions.** The QSA block scorer reduced every lane's | |
| 4-dim partials with five shuffles per head (20 shuffles per 16 FMAs, ~12 % | |
| of FMA peak). Eight lanes per block holding 16-dim query slices in | |
| registers, a transpose tree for the four head sums, and a 132-float tile | |
| stride for conflict-free `LDS.128`: 12.0 -> 4.2 ms. | |
| - **Block barriers in a per-token loop.** The GDN recurrence paid four | |
| `__syncthreads` per token across four warps; one warp owning eight | |
| columns with xor-shuffle cross-group sums, a warp-private shared Q/K copy, | |
| and the next token's operands prepared one iteration ahead: 16.6 -> | |
| 7.1 ms (fp32 reorder; fixture 1.5e-8). | |
| - **Long-scoreboard stalls in a K loop.** ncu on the worklist MMQ kernels: | |
| 255 registers, one block per SM, 61-74 % issue slots idle, the dominant | |
| stall the global loads issued at the top of each K iteration and consumed | |
| immediately. Prefetching the next block's raw bytes into registers after | |
| expanding the current tile, and staging both activation halves with | |
| `cp.async` one iteration ahead (4 -> 2 barriers per iteration): +5.3 %, | |
| bit-identical, for tiles up to 64 wide (a 128-wide two-stage buffer no | |
| longer fits the 100 KB SM). | |
| - **Fragment loads.** `ldmatrix` / `ldmatrix.trans` for the HMMA attention | |
| fragments plus a BF16 next-tile register prefetch: +2.4 %. | |
| - **Occupancy over reuse for small per-thread state.** GQA decode attention | |
| sharing one KV row across six query heads spilled registers and lost | |
| 25 %; the same idea over two heads gained 25 %. Check the register count | |
| after every "reuse across heads/rows" change. | |
| ### 2.7 What did not work, and the rule each one taught | |
| | attempt | result | rule | | |
| |---|---|---| | |
| | per-column token map inside the MMQ tile so the y tile reads the token-compact activation directly | gate/up kernel +22 %, prefill -3 %, despite 0.6 GB less DRAM traffic per layer | the tile's contiguous 18 KB y loads are the hot loop; DRAM bytes saved outside the loop do not pay for an indirection inside it | | |
| | weighted SwiGLU applied in the up launch's write back (reads the stored gate at the same index) | up launch 4.49 -> 5.84 ms, more than the 0.095 s pass it removed | an MMQ write back may *store* scattered, it must not *read* scattered; fuse elementwise work where the reads are coalesced | | |
| | Linux AIO page workers, depth 32 | cold-PLE prompt -14 % | deeper queues without consumption-order scheduling delay the rows the consumer waits on | | |
| | QSA reduce over 6-head groups; generic Q8 MMVQ with 8 warps; IQ1_M 4-row tile | no stable gain or slower | occupancy and register pressure; measure before assuming reuse wins | | |
| | balanced halves for prompts between one and two opening chunks | -4 % vs the split, -8 % vs one chunk | a chunk under ~2K rows runs the MoE tiles at low occupancy; below that size the I/O overlap is worth less than the efficiency lost | | |
| | batched IQ1_M MMVQ across tokens (K2 Prefill 1) | isolated parity passed, first token changed at the frontier | an isolated-kernel parity test is not a graph gate; the frontier logits and greedy IDs are | | |
| --- | |
| ## 3. Decode: a different workload | |
| Decode at width 1-2 is bandwidth- and launch-bound, not compute-bound. The | |
| Qwen budget at the end of the campaign, per ~52 ms MTP step: dense GEMVs at | |
| ~240 GB/s (the 248,320-row LM head alone 2.7 ms per call), the two-row | |
| routed-expert traffic, ~62 launches per layer, host gaps ~2 ms per pass. | |
| Prefill kernels are the wrong kernels here and vice versa; dispatch on width. | |
| ### 3.1 Width-dependent dispatch | |
| - Decode widths (1..8 rows, including the two-row verify pass and the | |
| two-bank lane) belong on vector kernels (`mmvq`, warp-per-row aligned Q8), | |
| not on MMQ tiles: moving top-10 routing from ten-row MMQ tiles to the vec | |
| tier was -6 % per step (Qwen R9); K2's aligned Q8 dense-vec path serves | |
| every dense projection at width 1. | |
| - The vec tier's assignment cap is a policy enum, not a magic number | |
| (`DS4_ROUTED_VEC_MAX_ROWS`), and the split between "one launch per | |
| assignment" and "one launch per token" must be checked per call site | |
| (R15). | |
| - Tiny GEMVs deserve their own kernels: a warp per output row with 16-byte | |
| BF16x8 loads replaced a naive per-thread kernel on the hyper-connection | |
| path (1,067 launches per step, -8 %). | |
| ### 3.2 Row stability is a contract, not an optimization | |
| Speculative decoding verifies a draft by running [committed, draft] as one | |
| two-row pass and compares its row-0 arithmetic with what a one-row pass | |
| would have produced. That only works if every kernel's per-row result is | |
| independent of the batch width: | |
| - widths 2..8 use the same aligned Q8 NC kernel as width 1 (the raw `mmvq` | |
| partitions by column count and differs at 2e-7); | |
| - multi-expert rows on the routed vec tier launch one token at a time; | |
| - the F32 router uses a row-stable GEMV, not cuBLAS (9e-4 apart). | |
| The gate is `tests/test_qwen4exp_verify`: 48 accept/reject steps, logits | |
| and every recurrent state bit-exact against the serial oracle. Every decode | |
| round since has had to keep it, and it has caught a real regression (the | |
| per-token split in R14 that R15 undid was found by measurement, but the | |
| state parity of R14 itself came from this test). For MTP on compressed-KV | |
| families the contract is token-level at width > 1 (`AGENTS.md`, "MTP / | |
| compressed-KV decode rules"): the committed cache values are inherently | |
| fp-noisy across widths, the committed token stream and frontier counts are | |
| not. | |
| ### 3.3 Speculative decoding: never verify twice | |
| Before R14 the drafter's token was verified by a *second* one-row target | |
| pass, so MTP never saved a pass and an auto-quenched run was 5 % faster than | |
| the speculative one. The fix is protocol, not kernels: run the committed | |
| token and the draft as one two-row pass, checkpoint the recurrent frontiers | |
| (GDN state, PLE conv state) inside the same kernels after row 0 (a template | |
| instantiation, because a branch in the hot loop cost 3.5 % prefill), and | |
| roll back a rejection by pointer swap plus a hash re-derivation; position- | |
| indexed caches only reset their lengths. 67.5 -> 56.7 ms per step, -16 %. | |
| Measure the speculative path with the quench heuristic in mind: under nsys | |
| the per-launch overhead trips it after 12 cycles. | |
| ### 3.4 Attention at decode | |
| - **Split the context across blocks.** One block per head over a 32K | |
| context used 1 of 48 SMs; split-K across context chunks with a combine | |
| kernel: K2 decode +78 % (the largest decode round anywhere in these | |
| campaigns). Qwen's sparse-attention reduce got the same treatment at | |
| rows <= 8 (128-slot chunks + combine, -11 % per step). | |
| - **Share KV across GQA heads only as far as registers allow** (2.6). | |
| - **Run the decode-attention pair kernel per chunk**, not once over the | |
| whole context, when the context is split (K2 Decode 3). | |
| ### 3.5 The rest of the step | |
| - Two-bank lanes: run the two decode-ready banks as one two-row operation | |
| wherever the arithmetic is bank-independent (embedding, hyper-connection, | |
| output projection, GDN recurrent grid, PLE gather, Q5 tail, routed main); | |
| each was +1-9 % on a two-request microbenchmark and each has a fixture | |
| that shows the two-row result bit-identical to two one-row calls. | |
| - Host gaps: ~2 ms per pass is launch issue and synchronization; captured | |
| graphs help only if every captured kernel reads live substrate state | |
| (`AGENTS.md`, "CUDA captured-decode rules") — by-value arguments replay | |
| stale. | |
| - The LM head (248K rows) is 2.7 ms per call at bandwidth; the only lever | |
| there is the artifact (aligned Q8) and not calling it twice. | |
| --- | |
| ## 4. Memory: the constraint that shapes everything on a unified-memory host | |
| - One resident model, one owner, one worker. The owner holds the weights | |
| (80 GiB here) through VMM/IPC; workers import them and add their graphs | |
| (24-30 GiB for a 196K-262K two-bank Qwen worker). Every measurement | |
| worker is a fresh process against the same owner, so the A/B cost is | |
| seconds, not the 5-minute weight load. | |
| - Additive derived artifacts (aligned Q8 repacks) live in the owner and are | |
| cheap (0.94 GiB for 160 tensors); artifacts that *replace* raw residency | |
| (IQ2/Q2K SoA) change the memory plan. | |
| - Score/scratch buffers are sized to the width that uses them: the QSA | |
| score scratch shrank from prefill-width to decode-width once prefill went | |
| fused (graph 9.86 -> 8.33 GiB at 64K), which is what made the 262K | |
| two-bank worker fit. | |
| - Bounded caches for SSD sidecars are sized from the overlap policy (one | |
| in-flight chunk per stream), not from free memory; oversizing hides | |
| eviction bugs instead of fixing them. | |
| --- | |
| ## 5. Numerics: the contract per class of change | |
| Correctness before speed is only useful if "correct" is defined per change. | |
| Three classes were used; every commit names its class. | |
| ### 5.1 Bit-identical (scheduling, fusion of elementwise chains, layout) | |
| Same quantized values, same tiles, same dot-product order, same expression | |
| in the fused kernel: the fixture compares byte for byte (`memcmp`), and the | |
| full-model gate (logits + greedy IDs) is exact. Things that break | |
| bit-identity silently: | |
| - **Contraction.** `a + b * c` in one kernel is an FMA; the same math in | |
| two kernels is a rounded product then an add. Write `__fmul_rn` / | |
| `__fadd_rn` (or `fmaf` explicitly, if the old code was contracted) in the | |
| fused kernel and prove it on the fixture. | |
| - **Different translation units.** `expf` and division under | |
| `--use_fast_math` are the same intrinsic only if both TUs use the same | |
| flags; the Makefile does, but check when the epilogue moves from | |
| `ds4_cuda.cu` into the vendored MMQ TU. | |
| - **Batch width.** `mmvq` at N=1 and N=2 partition by column count and | |
| differ at 2e-7; MMQ tile widths do not change the per-element order. | |
| - **Chunk size / row count.** cuBLAS picks algorithms per shape; the | |
| opening chunk changed the greedy continuation of a near-tie prompt while | |
| every kernel stayed bit-identical (the kill switch reproduced the old | |
| text). This is not drift, but it is a shape change to document. | |
| ### 5.2 fp32 reorder (accumulation order changes, fused attention) | |
| Fixture tolerance against the CPU/F64 reference of the order of 1e-7 (max | |
| abs) to 1e-8, self-parity of the chunked vs single-shot path bit-exact, the | |
| two-row verify gate exact. Examples: the fused QSA attention (1.3e-7), the | |
| warp-independent GDN loop (1.5e-8), the block scorer (identical top-512 | |
| selection). | |
| ### 5.3 Value parity (a different quantization path or tier) | |
| For rounds that change what gets computed (Q8_0 DP4A -> Q8_1 MMQ | |
| activations, D2R vs mmq fold order, chunk-size changes on K2): a kernel | |
| parity row against F32 at the tier's known tolerance (~2.5e-3 rel RMS), | |
| the same-binary kill-switch A/B, the comparator band on the frontier logits | |
| (same argmax, top-10 overlap >= 8/10, KL <= 0.05, rel RMS <= 0.11), a | |
| byte-identical repeat of the new path, and the contract written into the | |
| doc. K2's chunk-size round was rejected under the byte-identical gate and | |
| accepted under this one with its drift measured (rel RMS 0.0636). | |
| ### 5.4 Reading "the text changed" | |
| A greedy continuation that differs after N tokens is a *symptom* with three | |
| causes: a bug, a class-5.3 change, or a near-tie token under a class-5.1/5.2 | |
| change. Tell them apart with the kill switch on the same binary: if the | |
| old text comes back with the switch off and the fixture is bit-identical, | |
| it is the shape (5.1); if the fixture shows the tier tolerance, it is 5.3 | |
| and needs the band; if neither, it is a bug. Do not compare texts across | |
| sessions or artifacts, and never quote tok/s from a run whose text differs | |
| without saying so. | |
| ### 5.5 Fixtures: make them look like production | |
| - The Qwen MoE fixture routed 1,200 rows over 8 experts (150 rows each: | |
| 128-wide tiles plus a 32-wide tail) and passed a kernel that faulted in | |
| production, where 512 experts see ~5 rows each and the 8- and 16-wide | |
| tiles carry the work. Add the production routing regime (a few rows per | |
| expert) to every MoE fixture. | |
| - Model-free fixtures register weights as host memory; a kernel that | |
| re-reads a weight tile per row block runs 2.5x slower there than in the | |
| served model. Cache the fixture weights on the device | |
| (`ds4_gpu_cache_model_range`) before timing. | |
| - Include a non-finite input in every fixture for a kernel that guards at | |
| read; the guard is part of the contract. | |
| - A fixture that passes is not a full-model gate. Boot the real model | |
| (the 256-token prewarm chunk found the narrow-tile bug in seconds). | |
| --- | |
| ## 6. Host and tooling pitfalls that cost hours | |
| - `pgrep -f`/`pkill -f` with a pattern that appears in the calling shell's | |
| own command line matches the shell. Keep patterns out of inline heredocs; | |
| check the PID's `ps` line before acting on it. | |
| - The engine's instance lock refuses a second `ds4` process: stop the | |
| production worker before any `ds4-bench` run against the same owner. | |
| - tmux starts commands from the server's environment; per-run engine | |
| switches ride in the command string (`EXTRA_ENV`), and a switch that | |
| silently did not reach the worker re-measures the baseline. | |
| - Unix socket paths are limited to ~108 bytes; a manifest under a long | |
| scratch path makes the weight owner exit at "broker socket path is too | |
| long". Use `/tmp`. | |
| - Never grep a worker log for "failed" as a completion signal (engine | |
| lines contain "failed 0"); wait on a marker you print yourself. | |
| - `nsys` traces of the first chunk include the bench's warm-up; identify | |
| chunks by the kernel that runs once per layer (the GDN recurrence, 36 | |
| launches per chunk) rather than by time. | |
| - `ncu` with the resident owner is a documented OOM on this host. | |
| - After `git checkout` of a file, `make` rebuilds its object even if the | |
| bytes are unchanged; a 13-minute MMQ compile can be the price of a | |
| cosmetic revert, and a build started while `make` is still evaluating an | |
| earlier goal picks up edits made in between. Do not edit sources while | |
| a build runs; keep patches as scripts in scratch and apply them to a | |
| clean tree. | |
| - A compile that runs during a timed run perturbs it; run traces (which are | |
| compared by kernel time) during builds, timed A/Bs never. | |
| --- | |
| ## 7. Checklist per round | |
| 1. Trace the production shape; rank gaps and kernels with launch counts. | |
| 2. Name the class of the change (5.1 / 5.2 / 5.3) before writing it. | |
| 3. Fixture with the production regime; bit-exact or a stated tolerance. | |
| 4. Full-model boot (prewarm chunk) and the family's resident gates. | |
| 5. Same-hour A/B, fresh workers, medians of three, both shapes (short | |
| prompt for TTFT, long prompt for the depth term), decode unchanged. | |
| 6. Kill switch on the same binary; attribute any text change. | |
| 7. Commit: bottleneck and its share, why the old path was inefficient, the | |
| contract, the numbers with shape and corpus, `Tests:`. | |
| 8. Update the doc with what was rejected and why; the next campaign starts | |
| there. | |