ds4-winhip / docs /prefill-decode-optimization-playbook.md
staccs's picture
ds4-winhip: Windows/HIP 7.2 build tree, launcher, supervisor, bench + notes (part 5)
bb74494 verified
|
Raw History Blame Contribute Delete
31.8 kB
# Prefill and decode optimization playbook
What the 2026-09 optimization campaigns on this engine taught, written down
so the next campaign (another family, another host) starts from the method
and the failure modes instead of rediscovering them. Everything below was
measured on one DGX Spark (GB10, `sm_121`, 128 GB unified memory, ~240 GB/s
LPDDR5X, ~90K random 4 KiB IOPS from the NVMe) with mixed-quant MoE models
served from a resident VMM weight owner. The numbers are quoted so the
weight of each lesson is visible; they do not transfer to other hardware,
only the reasoning does.
Campaigns this distills (all in `docs/` and `scratch/`):
| campaign | model | result |
|---|---|---|
| 2026-09-01 / 09-02 prefill (9 + 5 rounds) | Qwen3.8-Flash-Next Q5 + SSD-PLE | 8K prompt 497 -> 604 -> 911 tok/s |
| 2026-09-02 / 09-03 decode (8 rounds) | same | 93 -> 52 ms per MTP step, 18 -> 31 tok/s |
| 2026-09-04 long context (5 rounds) | same | cold 64K 530 -> 1371 tok/s, cold 196K 1052 -> 1288 |
| 2026-09-05 K2-Horizon-375B MQ87 (12 prefill + 4 decode rounds) | K2 IQ1/IQ2 experts | 8K prefill 287 -> 645 tok/s (+125 %), decode 5.5 -> 13.3 tok/s |
| 2026-09-06 prefill (3 rounds) | Qwen | cold 8K 1215 -> 1362 tok/s (+12 %), 64K +4 % |
| 2026-09-06-r4 (D2R K=2560 + HC q8 + o_proj K=6144) | Qwen | cold 8K 1353 -> 1432 tok/s, 64K 1431 -> 1555 |
The one-line version: **almost every large win came from removing work the
graph did not need (idle time, fallbacks, repeated transformations, redundant
round trips), and the kernel-tuning rounds came last and paid least.** This
is the order the engine's `AGENTS.md` prescribes and the campaigns confirmed
it round after round.
---
## Local scout with ds4-perf
Use the [ds4-perf guide](ds4-perf.md) for the four-command workflow,
optional NVIDIA NVTX/CUPTI builds, workload contracts, bounded NCU capture,
comparison gates, and automatic experiments. The production annotations remain
`ds4.prefill` and `ds4.decode` at the measured Rust operations.
Nsight reports diagnose execution structure; only fresh unprofiled samples
support throughput comparisons. Keep the original model, prompt, token counts,
cache protocol, and intended weight owner. Record prefill and decode separately.
The heuristics prioritize host/idle, memory operations, short-launch fragmentation,
and dominant kernels. Missing or ambiguous data stays unknown. GPU coverage is
an interval union inside the Rust phase; summed kernel durations may overlap.
GB10 validation (2026-09-07): Qwen MQ-Q5 SSD-PLE BF16, 2,048-token cold frontier
and eight greedy tokens, three fresh unprofiled runs per build. All 248,320
frontier logits and token IDs were byte-identical. Optional NVTX changed median
prefill from 1247.86 to 1246.19 tok/s and decode from 25.22 to 25.11 tok/s
(-0.13% / -0.44%); this short smoke does not establish broader performance.
Scout found prefill `HOST / IDLE` (73.1% GPU coverage) and decode `DOMINANT KERNEL`
(`mul_mat_vec_q`, 89.0% coverage). Both range names and phase kernel totals were
checked against a separate Nsight export of the raw report.
Existing `speed-bench/` scripts remain valid for specialized server and
deep-context profiling.
---
## 1. Method: one round
A round is one change, measured against the previously adopted binary under
identical conditions, kept only if the gain is stable and the numerics are
accounted for. Rounds that skipped a step produced the wrong conclusion
every time they did.
1. **Profile the whole request, not a kernel.** `nsys --trace=cuda` on the
production shape (an 8K cold prefill, a 64K cold prefill, a decode window
of ~10 tokens). Export to sqlite and reduce it yourself: per chunk, GPU
busy vs idle, the *gap before each kernel* (idle is attributed to the
kernel that was waiting), then kernel totals with launch counts and
average duration. `scratch/qwen-prefill-opt-20260906/{nsys_chunks.py,
kern.py}` are the two scripts; they run in seconds. Rank gaps and kernels
together. A 1.06 s gap before the PLE gather kernel out of a 6.6 s
prefill was the biggest single item of the last Qwen campaign and no
kernel view shows it.
2. **Attribute before designing.** Launch counts and average durations tell
you whether a bucket is one slow kernel or many small ones (334 dense MMQ
launches per chunk, 514 quantize launches, 387 sanitize passes). For a
kernel, `ncu` on a model-free probe (the fixtures have `DS4_*_PROFILE_*`
modes that launch the production shape on synthetic weights) gives the
stall reasons; on this host `ncu` runs only with the weight owner down.
3. **Change one thing.** Two changes in one round cost a full re-measure
when one of them regresses (the 09-06 round 3 folded a SwiGLU epilogue and
a fused sum together; the epilogue lost 0.7 % and hid the sum's +1 %
until a kill-switch run separated them).
4. **Prove the numerics on a fixture first** (section 5), then run the
full-model gate. Fixtures are fast (seconds) and catch layout bugs the
full model only shows as an illegal address at boot.
5. **A/B on fresh processes, same hour, medians of three.** One fresh
worker per run against the resident owner, cold KV directory, the same
corpus slice, `thinking` disabled, non-streaming so `usage`/`timings` are
present. Throughput drifts 1-4 % between hours on one binary (thermal,
page cache, owner state), so the baseline is re-run in the same session
whenever a difference is under ~2 %. Never build while timed runs are
on; the compile steals the cores the page workers use.
6. **Keep a kill switch** for every new execution path (`DS4_*=0/1`, read
once per process). It is the attribution tool: same binary, path on vs
off, and it is what turns "the text changed" into "the text changed
because of X" (section 5.4).
7. **Commit with the evidence**: what was slow and by how much, why, the
contract (bit-identical / fp32 reorder / value parity), the A/B numbers
with the shape and corpus, and the `Tests:` that actually ran.
What counts as the metric matters as much as the protocol:
- Prefill: tokens per second of a **cold** prefill of a fixed shape. A
repeated prompt on a warm worker is not cold; the SSD-PLE page cache and
the disk-KV cache both hide the cost you are trying to measure. Say which
corpus: a prompt whose n-grams hit the page cache and one whose n-grams
miss it differ by 30 % on the same binary.
- Long context: the single-shot `--ctx-start N --ctx-max N` shape (one cold
N-token prefill, the production TTFT of a long request) and the
incremental sweep (2K steps on a warm session) answer different questions;
the sweep cannot exercise the next-chunk lookahead because the next chunk
is not known.
- Decode: **milliseconds per step**, not tokens per second, whenever
speculative decoding is on. tok/s = tokens-per-step / ms-per-step, and
tokens-per-step moves with the draft acceptance rate, which moves with the
text, which moves with any fp32 reorder (a near-tie token flips ~25 tokens
into the standard prompt). 93 -> 52 ms per step is a kernel result;
18 -> 31 tok/s is that plus acceptance luck.
---
## 2. Prefill: where the time went and how it was removed
Ordered as `AGENTS.md` orders it: unexpected fallback paths, repeated
transformations, redundant memory traffic, routing and indexing overhead,
launch fragmentation, fast-path utilization, and only then kernel tuning.
### 2.1 GPU idle time is the first target
If the profile shows idle, no kernel work matters until it is gone.
- **Find the wait, then the bug behind it.** The 09-04 trace put the GPU
idle 3.8-12 s per 8K chunk before the PLE gather kernel. The obvious
reading ("SSD is slow") was wrong: the page cache's victim scan stopped at
the *first* evictable way instead of the least recently used one, so a
prefetch's pages were evicted by the next request for the same set and a
chunk re-read a third of its pages at gather time. True LRU + 16 ways
took the 64K wait from 59.5 s to 3.8 s (530 -> 1163 tok/s). More I/O
workers had made it *worse* (64 workers: 339 tok/s) because faster
completion produced more READY victims for the broken scan. A knob that
makes things worse when turned up is a bug, not a limit.
- **Overlap I/O with compute at the granularity the pipeline allows.** The
next chunk's pages are queued right after this chunk's rows are gathered
(layer 1), so 47 layers of compute hide the reads (the "lookahead").
Cache sizing follows from the overlap: one chunk's working set per
in-flight stream, two for two alternating banks.
- **The first chunk has nothing to hide behind.** Opening every prompt
with a short chunk (2,048 rows) exposes only its own reads and hides the
full-size chunk behind it: cold 8K +9.1 %. Two boundaries: a short
*trailing* chunk is a net loss (a 256-row chunk ran its 48 layers at low
occupancy and hid few reads: -7 % on a 2,304-token prompt), so the split
applies only when the trailing chunk is at least as long as the opening
one; and on prompts whose pages are already cached the opening chunk costs
its own inefficiency (~1.7 %). Chunk-size changes also change GEMM shapes
(cuBLAS algorithm choice) and therefore near-tie tokens; that is expected.
- **Queue depth is not the lever when the pipeline is.** A Linux-AIO
rewrite of the page workers (depth 32) lost 14 % on the cold-PLE prompt
although the raw device scales with depth: deeper queues made the
head-of-line rows the gather waits on arrive later. Fix ordering
(consumption order, targeted wake-ups) before adding concurrency.
### 2.2 Fallback paths: the biggest single wins
A quantization format or a shape that the fast tier does not admit falls to
a generic kernel, and the fallback is usually 2-10x slower than the tier it
missed. Read the engine's "engaged path" log lines and grep the dispatch
predicates before touching any kernel.
- **Rectangular schedules over ragged work.** MoE prefill routes ~5-20 rows
per expert; a `[expert, max-bucket]` grid launches mostly empty tiles. A
compact worklist (one persistent kernel, grid = SM count, tiles of
8/16/32/64/128 columns chosen per expert bucket inside the kernel) is the
right shape: K2 IQ2_XXS down 12.9 -> 7.0 ms, IQ1_S gate/up 9.8 -> 5.0 ms,
IQ1_M gate/up from a per-token MMVQ loop to an MMQ tile +55 % end to end.
- **Shapes that are "almost" the tile.** An expert-down with K = 640 is not
a multiple of the 256-wide K-quant super-block, so the recipe stores a
512-column main + 128-column tail and the engine ran the tail as a separate
F32 accumulate. Folding the tail into the MMQ K loop as one extra
half-iteration (+14 %) and then running the dense K = 640 shared-expert
down through the same fused-tail entry as a one-expert MoE (+17 %) were
the two largest prefill rounds of the 09-02 campaign. When a dense
operation can be expressed as a routed one with an identity map, the
routed tier's kernels are usually the better ones.
- **Every projection on the quantized-activation tier.** PLE key/value and
QSA k/v pairs ran on Q8_0 DP4A kernels while every other projection used
the Q8_1-activation MMQ tier; moving them (+11 %) changed numerics within
the MMQ tolerance, so this is an arithmetic round with a contract, not a
scheduling round.
- **Chunk size is a dispatch parameter.** K2 at 512-token chunks saw ~21
rows per expert and 16x the per-chunk launches of an 8K prefill; 1,024
doubled the rows per expert (+20 %). It also changed the greedy IDs (56
of 64), which is why it was rejected once and adopted only with the
revised gate (section 5.3).
- **Tiers that exist but are never entered.** Qwen's dense Q8 projections
missed the aligned D2R tier because the weight owner's repack rule
admitted only `dims[0] % 1024 == 0` (K=2560 qkv/z/q). Admitting the
D2R prefill contract moved them: cold 8K +2.8 %, 64K +5.1 %. o_proj
(K=6144) already had the artifact via `%1024` and still missed the
dispatch cap of 4096 (kept for DeepSeek K=8192, 0.81x vs mmq); raising
the cap to 6144 only: +1.3 % / +2.5 %. The "engaged path" log line
for a tier is worth checking against the tensor list once per family.
### 2.3 Repeated transformations
The same activation quantized, converted or permuted more than once is pure
waste and usually bit-identical to remove.
- Pair kernels that share one expert map and one Q8 activation for gate/up
(K2 +4 %, Qwen +11 % in combination with the worklist).
- Quantize once per **token**, not once per assignment slot: the routed
gate/up quantize gathered every token row top-k times (0.84 GB per 8K
layer). Quantize the compact rows and scatter the 144-byte Q8 blocks into
the expert-sorted layout the tiles stream (+1 %; see 2.7 for the variant
that lost).
- Hyper-connection rows converted to BF16 once by the norm instead of per
consuming GEMM (+3.3 %), later the norm storing only BF16 rows plus one
scale per lane, with F32 recomputed from the hyper input where needed.
- The Qwen HC mix is one F32 tensor consumed by qkv+z or q+index; emitting
its Q8 once through the existing norm-q8 registry retired the per-GEMM
quantize (bit-identical, +0.8 % / +0.9 %). The D2R preq entry had to
accept the same K % 128 as the launch; a leftover %1024 gate kept the
published buffer unused.
- Counting-sort expert id maps instead of the warp-scan builder
(bit-identical, +7 % with the sanitize drop).
- Trigonometry tables: K2's QK-norm/RoPE kernel computed `pow/fmod/cos/sin`
in double per (head, pair) per launch; on GB10 FP64 runs at 1/64 rate. One
(cos, sin) table per chunk position: 40 ms per 512-token chunk gone,
12 us per decode launch gone.
### 2.4 Redundant memory traffic
At ~240 GB/s every [rows x hidden] F32 round trip at 8K rows costs ~0.35 ms
per read or write; a MoE layer had a dozen of them. Count bytes per layer
before and after; the trace's kernel time for bandwidth-bound kernels is
bytes / 240 GB/s within 10 %.
- **Sanitize passes.** A standalone non-finite pass over every MMQ output
is a full read + write; when every consumer already zeroes non-finite
values at read (`moe_sum`, the weighted SwiGLU), the pass is redundant.
K2 Prefill 9 removed 58 launches per chunk; after D2R retired its own
outputs Qwen still carries 303 dense ones on a 6,144-row chunk (0.8 %).
- **Pack copies.** The expert-down read its 512-column input from a packed
copy of the 640-wide SwiGLU rows although the same entry already read the
128-column tail in place with a row stride. Reading both halves in place
removed 378 MB per layer (+0.8 %). Whenever a kernel takes a stride,
strided reads are free; a pack kernel is a smell.
- **Fused epilogues on elementwise chains** (safe class): `moe_sum` +
shared-expert sigmoid gate + residual add in one kernel (two round trips
per layer, +1 %); residual + next norm in one kernel; SwiGLU producing the
quantized down input directly where the layout allows.
- **Keep F32 intermediates out of memory.** The hyper-connection mix used
to write [rows x 10240] F32 logits from a cuBLAS GEMM and re-read them in
a mix kernel (~3 GB per sub-layer); forming the mix-up logits on the
tensor cores inside the mix kernel and applying the sigmoid in the
epilogue cut it to ~1.7 GB (+7 %). The 1.6 GB per-bank QSA score scratch
went the same way: a fused attention kernel with an online softmax never
writes scores (104 + 64 -> 67 ms per layer).
- **Recurrent state in registers.** The GDN recurrence did two loads and a
store per state element per token; keeping 32 rows per thread in registers
for the whole chunk halved the kernel (33 -> 17 ms), with `fmaf` written
explicitly so the contraction matched the old code bit for bit.
### 2.5 Launch fragmentation
Per-launch cost on this host is ~5-10 us of GPU gap plus host time; a Qwen
layer had ~62 launches at decode and an 8K prefill chunk ~4,000. Pairs
(2.3), persistent worklists (2.2) and fused epilogues (2.4) all cut launches
as a side effect; the rounds that targeted launches directly:
- One launch per layer for the routed vec tier instead of one per token
(Qwen R15, -7.6 % per decode step) — a per-token split introduced for
row-stability had silently split the assignment-major down call too.
- Paired 320-row and 4-row BF16 projections (R12), warp-parallel router
selection replacing a serial kernel (R13).
- Decode attention over the whole context in one launch was the wrong
direction: split-K across the context (section 3.4) adds a combine launch
and still wins 78 % because the single launch used 1 of 48 SMs.
### 2.6 Kernel tuning, last
Only after the graph is clean. The signals that led to each tuning round:
- **Shuffle-bound reductions.** The QSA block scorer reduced every lane's
4-dim partials with five shuffles per head (20 shuffles per 16 FMAs, ~12 %
of FMA peak). Eight lanes per block holding 16-dim query slices in
registers, a transpose tree for the four head sums, and a 132-float tile
stride for conflict-free `LDS.128`: 12.0 -> 4.2 ms.
- **Block barriers in a per-token loop.** The GDN recurrence paid four
`__syncthreads` per token across four warps; one warp owning eight
columns with xor-shuffle cross-group sums, a warp-private shared Q/K copy,
and the next token's operands prepared one iteration ahead: 16.6 ->
7.1 ms (fp32 reorder; fixture 1.5e-8).
- **Long-scoreboard stalls in a K loop.** ncu on the worklist MMQ kernels:
255 registers, one block per SM, 61-74 % issue slots idle, the dominant
stall the global loads issued at the top of each K iteration and consumed
immediately. Prefetching the next block's raw bytes into registers after
expanding the current tile, and staging both activation halves with
`cp.async` one iteration ahead (4 -> 2 barriers per iteration): +5.3 %,
bit-identical, for tiles up to 64 wide (a 128-wide two-stage buffer no
longer fits the 100 KB SM).
- **Fragment loads.** `ldmatrix` / `ldmatrix.trans` for the HMMA attention
fragments plus a BF16 next-tile register prefetch: +2.4 %.
- **Occupancy over reuse for small per-thread state.** GQA decode attention
sharing one KV row across six query heads spilled registers and lost
25 %; the same idea over two heads gained 25 %. Check the register count
after every "reuse across heads/rows" change.
### 2.7 What did not work, and the rule each one taught
| attempt | result | rule |
|---|---|---|
| per-column token map inside the MMQ tile so the y tile reads the token-compact activation directly | gate/up kernel +22 %, prefill -3 %, despite 0.6 GB less DRAM traffic per layer | the tile's contiguous 18 KB y loads are the hot loop; DRAM bytes saved outside the loop do not pay for an indirection inside it |
| weighted SwiGLU applied in the up launch's write back (reads the stored gate at the same index) | up launch 4.49 -> 5.84 ms, more than the 0.095 s pass it removed | an MMQ write back may *store* scattered, it must not *read* scattered; fuse elementwise work where the reads are coalesced |
| Linux AIO page workers, depth 32 | cold-PLE prompt -14 % | deeper queues without consumption-order scheduling delay the rows the consumer waits on |
| QSA reduce over 6-head groups; generic Q8 MMVQ with 8 warps; IQ1_M 4-row tile | no stable gain or slower | occupancy and register pressure; measure before assuming reuse wins |
| balanced halves for prompts between one and two opening chunks | -4 % vs the split, -8 % vs one chunk | a chunk under ~2K rows runs the MoE tiles at low occupancy; below that size the I/O overlap is worth less than the efficiency lost |
| batched IQ1_M MMVQ across tokens (K2 Prefill 1) | isolated parity passed, first token changed at the frontier | an isolated-kernel parity test is not a graph gate; the frontier logits and greedy IDs are |
---
## 3. Decode: a different workload
Decode at width 1-2 is bandwidth- and launch-bound, not compute-bound. The
Qwen budget at the end of the campaign, per ~52 ms MTP step: dense GEMVs at
~240 GB/s (the 248,320-row LM head alone 2.7 ms per call), the two-row
routed-expert traffic, ~62 launches per layer, host gaps ~2 ms per pass.
Prefill kernels are the wrong kernels here and vice versa; dispatch on width.
### 3.1 Width-dependent dispatch
- Decode widths (1..8 rows, including the two-row verify pass and the
two-bank lane) belong on vector kernels (`mmvq`, warp-per-row aligned Q8),
not on MMQ tiles: moving top-10 routing from ten-row MMQ tiles to the vec
tier was -6 % per step (Qwen R9); K2's aligned Q8 dense-vec path serves
every dense projection at width 1.
- The vec tier's assignment cap is a policy enum, not a magic number
(`DS4_ROUTED_VEC_MAX_ROWS`), and the split between "one launch per
assignment" and "one launch per token" must be checked per call site
(R15).
- Tiny GEMVs deserve their own kernels: a warp per output row with 16-byte
BF16x8 loads replaced a naive per-thread kernel on the hyper-connection
path (1,067 launches per step, -8 %).
### 3.2 Row stability is a contract, not an optimization
Speculative decoding verifies a draft by running [committed, draft] as one
two-row pass and compares its row-0 arithmetic with what a one-row pass
would have produced. That only works if every kernel's per-row result is
independent of the batch width:
- widths 2..8 use the same aligned Q8 NC kernel as width 1 (the raw `mmvq`
partitions by column count and differs at 2e-7);
- multi-expert rows on the routed vec tier launch one token at a time;
- the F32 router uses a row-stable GEMV, not cuBLAS (9e-4 apart).
The gate is `tests/test_qwen4exp_verify`: 48 accept/reject steps, logits
and every recurrent state bit-exact against the serial oracle. Every decode
round since has had to keep it, and it has caught a real regression (the
per-token split in R14 that R15 undid was found by measurement, but the
state parity of R14 itself came from this test). For MTP on compressed-KV
families the contract is token-level at width > 1 (`AGENTS.md`, "MTP /
compressed-KV decode rules"): the committed cache values are inherently
fp-noisy across widths, the committed token stream and frontier counts are
not.
### 3.3 Speculative decoding: never verify twice
Before R14 the drafter's token was verified by a *second* one-row target
pass, so MTP never saved a pass and an auto-quenched run was 5 % faster than
the speculative one. The fix is protocol, not kernels: run the committed
token and the draft as one two-row pass, checkpoint the recurrent frontiers
(GDN state, PLE conv state) inside the same kernels after row 0 (a template
instantiation, because a branch in the hot loop cost 3.5 % prefill), and
roll back a rejection by pointer swap plus a hash re-derivation; position-
indexed caches only reset their lengths. 67.5 -> 56.7 ms per step, -16 %.
Measure the speculative path with the quench heuristic in mind: under nsys
the per-launch overhead trips it after 12 cycles.
### 3.4 Attention at decode
- **Split the context across blocks.** One block per head over a 32K
context used 1 of 48 SMs; split-K across context chunks with a combine
kernel: K2 decode +78 % (the largest decode round anywhere in these
campaigns). Qwen's sparse-attention reduce got the same treatment at
rows <= 8 (128-slot chunks + combine, -11 % per step).
- **Share KV across GQA heads only as far as registers allow** (2.6).
- **Run the decode-attention pair kernel per chunk**, not once over the
whole context, when the context is split (K2 Decode 3).
### 3.5 The rest of the step
- Two-bank lanes: run the two decode-ready banks as one two-row operation
wherever the arithmetic is bank-independent (embedding, hyper-connection,
output projection, GDN recurrent grid, PLE gather, Q5 tail, routed main);
each was +1-9 % on a two-request microbenchmark and each has a fixture
that shows the two-row result bit-identical to two one-row calls.
- Host gaps: ~2 ms per pass is launch issue and synchronization; captured
graphs help only if every captured kernel reads live substrate state
(`AGENTS.md`, "CUDA captured-decode rules") — by-value arguments replay
stale.
- The LM head (248K rows) is 2.7 ms per call at bandwidth; the only lever
there is the artifact (aligned Q8) and not calling it twice.
---
## 4. Memory: the constraint that shapes everything on a unified-memory host
- One resident model, one owner, one worker. The owner holds the weights
(80 GiB here) through VMM/IPC; workers import them and add their graphs
(24-30 GiB for a 196K-262K two-bank Qwen worker). Every measurement
worker is a fresh process against the same owner, so the A/B cost is
seconds, not the 5-minute weight load.
- Additive derived artifacts (aligned Q8 repacks) live in the owner and are
cheap (0.94 GiB for 160 tensors); artifacts that *replace* raw residency
(IQ2/Q2K SoA) change the memory plan.
- Score/scratch buffers are sized to the width that uses them: the QSA
score scratch shrank from prefill-width to decode-width once prefill went
fused (graph 9.86 -> 8.33 GiB at 64K), which is what made the 262K
two-bank worker fit.
- Bounded caches for SSD sidecars are sized from the overlap policy (one
in-flight chunk per stream), not from free memory; oversizing hides
eviction bugs instead of fixing them.
---
## 5. Numerics: the contract per class of change
Correctness before speed is only useful if "correct" is defined per change.
Three classes were used; every commit names its class.
### 5.1 Bit-identical (scheduling, fusion of elementwise chains, layout)
Same quantized values, same tiles, same dot-product order, same expression
in the fused kernel: the fixture compares byte for byte (`memcmp`), and the
full-model gate (logits + greedy IDs) is exact. Things that break
bit-identity silently:
- **Contraction.** `a + b * c` in one kernel is an FMA; the same math in
two kernels is a rounded product then an add. Write `__fmul_rn` /
`__fadd_rn` (or `fmaf` explicitly, if the old code was contracted) in the
fused kernel and prove it on the fixture.
- **Different translation units.** `expf` and division under
`--use_fast_math` are the same intrinsic only if both TUs use the same
flags; the Makefile does, but check when the epilogue moves from
`ds4_cuda.cu` into the vendored MMQ TU.
- **Batch width.** `mmvq` at N=1 and N=2 partition by column count and
differ at 2e-7; MMQ tile widths do not change the per-element order.
- **Chunk size / row count.** cuBLAS picks algorithms per shape; the
opening chunk changed the greedy continuation of a near-tie prompt while
every kernel stayed bit-identical (the kill switch reproduced the old
text). This is not drift, but it is a shape change to document.
### 5.2 fp32 reorder (accumulation order changes, fused attention)
Fixture tolerance against the CPU/F64 reference of the order of 1e-7 (max
abs) to 1e-8, self-parity of the chunked vs single-shot path bit-exact, the
two-row verify gate exact. Examples: the fused QSA attention (1.3e-7), the
warp-independent GDN loop (1.5e-8), the block scorer (identical top-512
selection).
### 5.3 Value parity (a different quantization path or tier)
For rounds that change what gets computed (Q8_0 DP4A -> Q8_1 MMQ
activations, D2R vs mmq fold order, chunk-size changes on K2): a kernel
parity row against F32 at the tier's known tolerance (~2.5e-3 rel RMS),
the same-binary kill-switch A/B, the comparator band on the frontier logits
(same argmax, top-10 overlap >= 8/10, KL <= 0.05, rel RMS <= 0.11), a
byte-identical repeat of the new path, and the contract written into the
doc. K2's chunk-size round was rejected under the byte-identical gate and
accepted under this one with its drift measured (rel RMS 0.0636).
### 5.4 Reading "the text changed"
A greedy continuation that differs after N tokens is a *symptom* with three
causes: a bug, a class-5.3 change, or a near-tie token under a class-5.1/5.2
change. Tell them apart with the kill switch on the same binary: if the
old text comes back with the switch off and the fixture is bit-identical,
it is the shape (5.1); if the fixture shows the tier tolerance, it is 5.3
and needs the band; if neither, it is a bug. Do not compare texts across
sessions or artifacts, and never quote tok/s from a run whose text differs
without saying so.
### 5.5 Fixtures: make them look like production
- The Qwen MoE fixture routed 1,200 rows over 8 experts (150 rows each:
128-wide tiles plus a 32-wide tail) and passed a kernel that faulted in
production, where 512 experts see ~5 rows each and the 8- and 16-wide
tiles carry the work. Add the production routing regime (a few rows per
expert) to every MoE fixture.
- Model-free fixtures register weights as host memory; a kernel that
re-reads a weight tile per row block runs 2.5x slower there than in the
served model. Cache the fixture weights on the device
(`ds4_gpu_cache_model_range`) before timing.
- Include a non-finite input in every fixture for a kernel that guards at
read; the guard is part of the contract.
- A fixture that passes is not a full-model gate. Boot the real model
(the 256-token prewarm chunk found the narrow-tile bug in seconds).
---
## 6. Host and tooling pitfalls that cost hours
- `pgrep -f`/`pkill -f` with a pattern that appears in the calling shell's
own command line matches the shell. Keep patterns out of inline heredocs;
check the PID's `ps` line before acting on it.
- The engine's instance lock refuses a second `ds4` process: stop the
production worker before any `ds4-bench` run against the same owner.
- tmux starts commands from the server's environment; per-run engine
switches ride in the command string (`EXTRA_ENV`), and a switch that
silently did not reach the worker re-measures the baseline.
- Unix socket paths are limited to ~108 bytes; a manifest under a long
scratch path makes the weight owner exit at "broker socket path is too
long". Use `/tmp`.
- Never grep a worker log for "failed" as a completion signal (engine
lines contain "failed 0"); wait on a marker you print yourself.
- `nsys` traces of the first chunk include the bench's warm-up; identify
chunks by the kernel that runs once per layer (the GDN recurrence, 36
launches per chunk) rather than by time.
- `ncu` with the resident owner is a documented OOM on this host.
- After `git checkout` of a file, `make` rebuilds its object even if the
bytes are unchanged; a 13-minute MMQ compile can be the price of a
cosmetic revert, and a build started while `make` is still evaluating an
earlier goal picks up edits made in between. Do not edit sources while
a build runs; keep patches as scripts in scratch and apply them to a
clean tree.
- A compile that runs during a timed run perturbs it; run traces (which are
compared by kernel time) during builds, timed A/Bs never.
---
## 7. Checklist per round
1. Trace the production shape; rank gaps and kernels with launch counts.
2. Name the class of the change (5.1 / 5.2 / 5.3) before writing it.
3. Fixture with the production regime; bit-exact or a stated tolerance.
4. Full-model boot (prewarm chunk) and the family's resident gates.
5. Same-hour A/B, fresh workers, medians of three, both shapes (short
prompt for TTFT, long prompt for the depth term), decode unchanged.
6. Kill switch on the same binary; attribute any text change.
7. Commit: bottleneck and its share, why the old path was inefficient, the
contract, the numbers with shape and corpus, `Tests:`.
8. Update the doc with what was rejected and why; the next campaign starts
there.