File size: 12,060 Bytes

e1acc15

# PP1 q transport and scheduling design

Design only, OFF. No v1 wire/serving files changed. Correctness reference:
[reference.py](reference.py); deployment boundary: [AUDIT.md](AUDIT.md).

## Distribution contract and precision

The wire must describe the **same categorical q that sampled each token**,
conditioned on the committed prefix and all earlier tokens of that proposal
chain. Confidence/margin and q(x) alone cannot reconstruct a residual.

The CPU oracle accepts full or sparse normalized weights. Its FP32 helper first
rounds weights, then defines q as their exact rational normalization at both
ends. It proves the semantics; arbitrary FP32 weights would require a carefully
matched GPU normalization/sampling implementation. Do not send independently
rounded probabilities after sampling from unrounded logits. Do not silently
renormalize a different q on the verifier.

Recommended bounded GPU prototype: **dyadic categorical q**, with total mass
M=2^24. Define integer counts c_v≥0 whose exact sum is M; q(v)=c_v/M.
Transmit q(v) as little-endian FP32. Every such value is exactly representable;
all partial sums of nonnegative counts up to M are also exact in FP32 when
scaled by M. Both sides can recover c_v by multiplying by M, require it to be
an integer, and validate the sum using uint64. There is no receiver-only
renormalization. Zero counts have exactly zero probability.

Proposal construction (new implementation still required): obtain globally
normalized draft weights after draft temperature and Markov bias, reject NaN,
+inf, negative probabilities and zero effective mass; −inf masked logits are
allowed before softmax. Allocate integer counts using floor plus largest
remainders, ties by lowest global token ID. Use FP64 allocation arithmetic and
assert 0≤remaining<M and exact final count sum; refuse a numerically inconsistent
allocation rather than clipping or silently changing totals. Tiny entries may
become zero. That changes the proposal only, not the legal target.

For sparse top-k, select the global support first, normalize within it, then
allocate M counts on that support. Drop zero-count entries and declare zero
everywhere else. Sampling uses a uniform 24-bit integer and exact integer
cumulative counts (first cumulative count strictly greater than the draw).
Record the semantic ID `dyadic24-standard-v1`; do not call this unrounded
softmax sampling. q can approximate the draft arbitrarily poorly and still
preserve p under correct verification; approximation can reduce acceptance.

The full target p is the distribution after the **unchanged target** processors,
penalties, grammar, T=1 and top_p=1. Ratio tests and full-vocabulary residual
sampling must use this p and the defined q. GPU finite-precision p/ratio/log
residual calculations still need numerical qualification against the rational
oracle; the design is not a claim of bit-identical real-arithmetic GPU sampling.
The existing local verifier caches pre-temperature logits and reuses the target
temperature; it cannot be pointed at a probability buffer. Supporting independent
draft temperature requires a new q-aware standard-verification interface.
Passing log(q) through the old head dtype/cache would round it and redefine q.

## Versioned v2 message design

Keep v1 greedy production intact; negotiate v2 capabilities before allocating
or accepting sampled requests. No auto-upgrade or environment-only enablement.

| Boundary | Required fields / invariant |
|---|---|
| HELLO | wire version, q semantic ID, standard-verifier ID, vocab 129280, global ID map/hash, draft checkpoint/native-or-adapted identity, target identity, maximum slots/depth/support, byte lengths, RNG ABI, precision |
| STEP request | session ID, slot, generation, round ID, absolute anchor/first-draft position, committed-prefix/state digest, fixed depth, proposal temperature/support policy ID; target processor/grammar/stop context version |
| Per-position reply | sampled global int32 token ID, representation dense/sparse, count of entries, q values and sparse global IDs; probability checksum; q conditioned on that row's preceding sampled tokens |
| Completion | matching session/generation/round/position/context, exact reply size and byte checksum, explicit status; whole row becomes consumable only after validation |
| Commit/reset | accepted draft count, correction/bonus IDs, new committed position; cancel/reset invalidates generation, every probability slot and all speculative Markov/KV/grammar state |

Bounds are negotiated and checked before memory access. Reject duplicate sparse
IDs (even duplicate zeros), unsorted/noncanonical support, out-of-range/padded
IDs, missing q, non-finite values, negative/non-dyadic values, wrong total mass,
zero q at the proposed token, wrong identities, unexpected sizes and stale or
duplicate generations/rounds. Dense padding beyond vocabulary is always zero.
Checksums detect corrupt transport, not proof that the worker sampled correctly;
that requires independent CPU/GPU distribution and RNG tests.

One RNG owner draws each proposal: worker leader, keyed by request generation,
round, absolute predicted position, domain and draw counter. The peer receives
the chosen token for the next Markov step; it does not independently choose one.
The Spark sampler owns separate acceptance and correction/bonus RNG domains.
Never reuse proposal noise for correction. Request seed overrides and reset
semantics are explicit. Reconnection invalidates old in-flight RNG/state keys;
retries either reproduce the identical reply or reset the proposal generation.
Do not redraw under a reused key and consume an earlier q.

Missing/corrupt/timed-out q discards the **entire unverified proposal** on both
Spark ranks. The capacity path may then run target-only from the committed
prefix using its ordinary target sampler, after synchronized invalidation.
It must never return q=None alongside sampled IDs. No partial commit on one
rank and fallback on the other. Exposed failures have counters and errors.
No new resilience/local-drafter allocation is assumed.

## Size and critical path

All sizes below are **ESTIMATED, exact shape arithmetic**, excluding headers,
alignment, checksums, padding and pool bookkeeping. MB=10^6 bytes.

| Layout, depth 5 | Bytes / request | Bytes / four requests |
|---|---:|---:|
| Dense FP32, 129280 tokens | 2,585,600 (2.586 MB) | 10,342,400 (10.342 MB) |
| Sparse k=16, int32 ID + FP32 q | 640 | 2,560 |
| Sparse k=32 | 1,280 | 5,120 |
| Sparse k=64 | 2,560 | 10,240 |
| Sparse k=256 | 10,240 | 40,960 |
| Dense TP2 shard, 64640 tokens | 1,292,800 per rank | 5,171,200 per rank |

At nominal 40 Gb/s, dense serialization alone has an ideal lower bound of
0.517 ms/request or 2.068 ms/four-request batch on one link. These are **not
measured latency**: actual D2H, host staging, RDMA/TCP, GPU H2D, synchronization
and graph boundaries add cost. Sending only q(x) is smaller but incorrect.

First correctness integration: leader returns full q to Spark rank 0; validate
and TP-broadcast to the rank(s) that own sampled requests in the installed
vocabulary/DP sampling layout. The installed runner can shard requests, which
is different from vocabulary sharding. Do not assume that exposing a TP2 head
means its current sampler consumes vocabulary shards. Full replication costs
another 10.342 MB/batch on the Spark TP fabric plus each rank's buffers.

Two in-flight dense batch slots require 20,684,800 bytes (19.73 MiB) per full
copy on each endpoint, in addition to today's buffers. Preflight actual ofl
slot/reply limits: the tiny v1 reply pools are not sized for this. Bound at four
requests and fixed depth five; refuse incompatible allocation/graph shapes.
Use preallocated registered host staging, not assumed GPUDirect support.
Publish q-ready only after all D2H/network/H2D work and validation complete;
release a slot only after every sampler/rank consumer is done. Timeout watchdogs
must invalidate the generation before buffers can be reused. Preserve current
process-proxy isolation and late-reply protections.

## Sparse support and vocabulary-sharded TP2 follow-up

Sparse k=32 is the initial bounded candidate, k=16/64 only a development screen.
Local top-k on each worker shard produces at most 2k candidates; merge these
with global IDs and deterministic ties to obtain exact global top-k. This is
not independent local normalization. Compute one global support/count vector;
leader samples once and broadcasts the sampled ID before computing the next
Markov-conditioned row. A correct fixed-depth chain requires that dependency
even when backbone rows were computed in parallel.

For dense vocabulary sharding, each rank computes its logit max and shifted sum;
all-reduce max/sum yields global draft normalization. Count allocation must
coordinate remaining tickets globally. Store global shard ranges in the header.
For true rank-to-rank links, each worker rank could send only its q shard to the
corresponding Spark rank. Concurrent links give an ideal 0.259 ms/request/shard
serialization bound; shared uplinks and transport ownership must be measured.
This requires a different target sampler, not a switch in today's proxy.

A sharded exact sampler chooses the shard by its global count mass using one
leader-owned integer draw, then the token within it. Each Spark rank constructs
max(p−q,0) for **every** token it owns, including tokens outside sparse q support.
Global residual mass selects the correction shard; local residual CDF chooses
the token; broadcast its global ID. Shard selection and within-shard draws must
form the intended categorical, with zero-mass shards impossible to choose.
Global target normalization and grammar masks must match target-only sampling.
No full-vocab gather is necessary mathematically, but the current serving
sampler must first gain and qualify this ownership model.

Candidate kernels, in dependency order: global logsumexp and counts, global
top-k merge, integer categorical proposal, sparse q lookup by global ID,
full-target residual mass and sampling. Test each against the CPU oracle and
small-vocabulary sharding emulator before using CUDA graphs. Profile them only
after source-faithful native qualification, counting exposed whole-round cost.

## Depth admission must not inspect its own future draws

Initial PP1 always fixes depth **before** sampling (five, or a separately named
fixed-depth arm). No confidence pruning, adaptive acceptance window, margin
masking, global best-depth search, batch-dependent retrospective prefix cut,
or full-block confidence scheduling is enabled for sampled proposals.

[DSpark v1 appendix A](https://arxiv.org/html/2607.05147v1#A1) gives an explicit
counterexample: first-prefix throughput 0.9 loses to target-only 1.0, but a
later confidence depending on the first sampled token can raise the two-token
score to 1.134. Global selection then admits the first token for some values
of that token and excludes it for others. An exact verifier alone cannot undo
that selection. This is a PUBLISHED counterexample, not deployment evidence.

Any later adaptive policy must make the event “position i will be verified”
measurable before x_i is sampled. It may use committed history, load measured
before drawing, prior rounds, and x_1..x_(i−1); it cannot examine x_i or any
later confidence that encodes x_i and then revoke i. A sequential stopping rule
decides admit/stop, samples only if admitted, and never reopens a stopped prefix.
Even if a backbone computes all head features at once, later Markov confidences
are not automatically eligible inputs to earlier admission decisions.

The CPU suite includes a separate toy selection-bias regression: when q=p is
uniform, selectively admitting the first draw only when it is token 0 and
otherwise resampling target-only makes its output mass 3/4 instead of 1/2.
The fixed-depth reference never performs this selection. Adaptive scheduling
requires a separate proof and conditional-frequency suite before hardware.