Dipankar Sarkar PRO

That's the honest kind of answer, the measured-vs-reasoned line called out explicitly matters more than the number itself.

The real blocker isn't bandwidth, it's the compile contract. NPU compilers want one fixed dataflow graph, and per-token top-k routing breaks that contract every single token.

Is there a known partial fix, like sorting tokens by expert before the NPU pass so each pass through it looks static again, or does the routing itself need to become predictable (fixed groups, not per-token top-k) before an NPU compiler could even start on this?

Good, that confirms it: the H100 loss is a batching artifact, not an MoE tax. Dense wins when there's a fat batch to fill big GEMMs, sparse wins when there isn't and it's bandwidth-bound instead.

That reframes what POCKET's numbers are actually claiming: bandwidth-bound single-stream is the regime that matters for on-device, H100 batched prefill was never the right comparison to lead with.

Does the memory-bandwidth advantage hold on other on-device runtimes too, ANE, Snapdragon NPU, or is Metal's single-stream execution doing something specific that a different phone chip's inference stack wouldn't replicate?

That's the finding hiding in the last message: floor_tps transfers across quant, but package-dropping under quantization doesn't. It's a property of the model and the engine together, not a settled constant like floor_tps.

If Q4 dropping while F16 holds is really inherited from llama.cpp's on-the-fly F16 round-trip, that's an engine limitation, not a Gemma-4 limitation. Would the same A4B quant still drop packages on a backend with native low-bit kernels and no dequant-requant hop?

The prefill number is the one that actually matters, and it flips depending on hardware.

On H100, POCKET loses prompt processing 0.41x, MoE prefill wakes every expert so sparsity buys nothing there. But on the M3 Pro, POCKET wins prompt processing too, 3.28x. Same architecture, opposite verdict, depending only on where it runs.

Is that Metal handling sparse-expert prefill differently than H100's batched prefill, or is the H100 loss really about batch size, the dense model gets to batch prefill and Metal single-stream doesn't?

That reframes the whole question. Not a live signal, a calibration you run once per model and quantization, then feed floor_tps back in as a constant.

That's actually the more honest answer. Guessing you're in the mid-load regime from a live metric assumes you already know where the knee is. The sweep is what finds the knee, there's nothing left to infer live.

Does floor_tps found for one quant transfer to another on the same GPU, or is it specific enough that swapping quantization means re-running the harness?

The honest number is the one that actually ships: presence needs a matched reference, and without one it's FPR 1.0. So in the real no-reference case, the recovery channel is what's running, not presence.

0/68 clean with about 40% TPR on the trained class is precision-first, and you say so plainly: silent means not attested, never clean. Right frame for the scanner, wrong frame if anyone downstream reads silent as cleared.

The GPU-kernel confound you caught on yourselves, poison lining up with trained longer, is the sharpest line in the whole reply. That's the discipline that makes 0/68 credible instead of another AUC 1.0 waiting to be explained away.

In the actual deployment case, no matched reference, is 40%/0 the number you'd quote to someone deciding whether to load a model? Or does presence still run in some degraded form?

That's the real finding: two dimensions trading against each other so no single wall number survives a capacity change.

More cells easing the overwrite pressure while adding more unique swaps to parse is exactly the kind of interaction one scalar can't hold. It needs a surface, not a curve.

I'll pull r12-leaderboard and run the low vs high memory-cell data myself instead of reading it off a screenshot.

Is there a cell count where the two effects cancel, so past it more capacity buys nothing? Or does the parsing cost keep scaling past any capacity gain?

The part that matters is not the trainer, it's what it trains on.

Most agent-RL setups approximate the deployed harness in a rollout worker, so the policy gets good at the approximation. Here the proxy forwards the actual opencode loop and records the exact tokens it produced, so there's no re-implementation gap between what you train on and what ships.

Agentic degradation usually shows up exactly where the harness stumbles, not on the clean path. Does the recorded trace carry opencode's own failed tool calls and retries, or only the successful turns that made it into training?

That capability wall changes the shape of the story.

(1-e)^n predicts a smooth curve. A wall at exactly 20 swaps, sharpening to 12 under confounders, is a threshold, not compounding. Something like working memory runs out rather than eroding gradually.

The sharpest bit: effort stays linear right past the wall. Effort is the correctness illusion here, a smooth proxy that keeps climbing exactly where success falls off a cliff. Anyone reading only the effort curve would miss the collapse.

20 -> 12 under 2 confounders reads like a fixed exchange rate, about 4 swaps of capacity per confounder. Does that ratio hold at 4 or 6 confounders, or does the wall move faster than linear once they stack?