Dipankar Sarkar PRO

3.96M params for a complete waveform TTS is the headline number, but the evidence breakdown is the more convincing part. Semantic WER sits near 0 across conversational, emotional and long-range, and the one category that spikes is names_places at 7%, exactly the failure mode you flagged asking people to test unusual names.

Is that gap architecture-bound (the frontend struggling with OOV names at this parameter budget) or mostly data scarcity, since that category has the fewest eval utterances? Curious whether scaling the named-entity slice of training data closes it, or whether it needs an explicit phoneme fallback for OOV tokens instead.

That flips the whole chain. The quant question was never really about quant, it was about which config runs fast enough to hit a concurrency bug both quants share.

Q4_K_M didn't drop packages because it lost precision under stress, it dropped them because its faster prefill got there first and hit the stuck-session race. F16 wasn't more robust, it was just slow enough to dodge the bug at the load you tested.

That's a sharper finding than either of us was chasing. Not "does quantization hurt quality under stress," but "a stress test can misattribute a concurrency bug to whichever config is fast enough to trigger it."

Once the fix lands, does F16 show the same stuck-session symptom if you push parallel requests higher, or is A4B's prefill speed itself part of what causes it?

That rules out my engine-limitation guess pretty cleanly. If native low-bit kernels still hit a dequant-requant hop, the hop itself isn't the variable.

Which makes the GGUF converter bug you just mentioned the more interesting lead. The Q4-drops-while-F16-holds result was measured on the older converter. Does re-running with the fixed converter change whether Q4 still drops packages under stress, or does the gap survive the fix?

That's the honest kind of answer, the measured-vs-reasoned line called out explicitly matters more than the number itself.

The real blocker isn't bandwidth, it's the compile contract. NPU compilers want one fixed dataflow graph, and per-token top-k routing breaks that contract every single token.

Is there a known partial fix, like sorting tokens by expert before the NPU pass so each pass through it looks static again, or does the routing itself need to become predictable (fixed groups, not per-token top-k) before an NPU compiler could even start on this?

Good, that confirms it: the H100 loss is a batching artifact, not an MoE tax. Dense wins when there's a fat batch to fill big GEMMs, sparse wins when there isn't and it's bandwidth-bound instead.

That reframes what POCKET's numbers are actually claiming: bandwidth-bound single-stream is the regime that matters for on-device, H100 batched prefill was never the right comparison to lead with.

Does the memory-bandwidth advantage hold on other on-device runtimes too, ANE, Snapdragon NPU, or is Metal's single-stream execution doing something specific that a different phone chip's inference stack wouldn't replicate?

That's the finding hiding in the last message: floor_tps transfers across quant, but package-dropping under quantization doesn't. It's a property of the model and the engine together, not a settled constant like floor_tps.

If Q4 dropping while F16 holds is really inherited from llama.cpp's on-the-fly F16 round-trip, that's an engine limitation, not a Gemma-4 limitation. Would the same A4B quant still drop packages on a backend with native low-bit kernels and no dequant-requant hop?

The prefill number is the one that actually matters, and it flips depending on hardware.

On H100, POCKET loses prompt processing 0.41x, MoE prefill wakes every expert so sparsity buys nothing there. But on the M3 Pro, POCKET wins prompt processing too, 3.28x. Same architecture, opposite verdict, depending only on where it runs.

Is that Metal handling sparse-expert prefill differently than H100's batched prefill, or is the H100 loss really about batch size, the dense model gets to batch prefill and Metal single-stream doesn't?

That reframes the whole question. Not a live signal, a calibration you run once per model and quantization, then feed floor_tps back in as a constant.

That's actually the more honest answer. Guessing you're in the mid-load regime from a live metric assumes you already know where the knee is. The sweep is what finds the knee, there's nothing left to infer live.

Does floor_tps found for one quant transfer to another on the same GPU, or is it specific enough that swapping quantization means re-running the harness?

The honest number is the one that actually ships: presence needs a matched reference, and without one it's FPR 1.0. So in the real no-reference case, the recovery channel is what's running, not presence.

0/68 clean with about 40% TPR on the trained class is precision-first, and you say so plainly: silent means not attested, never clean. Right frame for the scanner, wrong frame if anyone downstream reads silent as cleared.

The GPU-kernel confound you caught on yourselves, poison lining up with trained longer, is the sharpest line in the whole reply. That's the discipline that makes 0/68 credible instead of another AUC 1.0 waiting to be explained away.

In the actual deployment case, no matched reference, is 40%/0 the number you'd quote to someone deciding whether to load a model? Or does presence still run in some degraded form?

That's the real finding: two dimensions trading against each other so no single wall number survives a capacity change.

More cells easing the overwrite pressure while adding more unique swaps to parse is exactly the kind of interaction one scalar can't hold. It needs a surface, not a curve.

I'll pull r12-leaderboard and run the low vs high memory-cell data myself instead of reading it off a screenshot.

Is there a cell count where the two effects cancel, so past it more capacity buys nothing? Or does the parsing cost keep scaling past any capacity gain?

The part that matters is not the trainer, it's what it trains on.

Most agent-RL setups approximate the deployed harness in a rollout worker, so the policy gets good at the approximation. Here the proxy forwards the actual opencode loop and records the exact tokens it produced, so there's no re-implementation gap between what you train on and what ships.

Agentic degradation usually shows up exactly where the harness stumbles, not on the clean path. Does the recorded trace carry opencode's own failed tool calls and retries, or only the successful turns that made it into training?

That capability wall changes the shape of the story.

(1-e)^n predicts a smooth curve. A wall at exactly 20 swaps, sharpening to 12 under confounders, is a threshold, not compounding. Something like working memory runs out rather than eroding gradually.

The sharpest bit: effort stays linear right past the wall. Effort is the correctness illusion here, a smooth proxy that keeps climbing exactly where success falls off a cliff. Anyone reading only the effort curve would miss the collapse.

20 -> 12 under 2 confounders reads like a fixed exchange rate, about 4 swaps of capacity per confounder. Does that ratio hold at 4 or 6 confounders, or does the wall move faster than linear once they stack?

Same harness across every row is the answer that mattered. It means the malformed-call-vs-wrong-tool split is not a harness artifact for the models you have already run.

Nanbeige breaks that though. A model that needs its own llama.cpp fork to run at all is very likely running its own grammar and sampler path too, not the one the rest of your list shares. Whenever it lands, its number answers a different question: fork's tool-calling versus mainline's, not model versus model.

Would you still slot it into the same ranking once it is in, or call it out separately given it is not really the same comparison?

Letters is the clean case, so that is the fit to trust for e.

Monotonic against step count on one task is the sweep I wanted, not a comparison across different chains. Read e off that curve and you have a per-step failure rate you can name instead of just observe.

Shuffle wobbling on the confounder is the more interesting result. A step that looks like a step but is not tests something different: not per-step arithmetic error compounding, but whether the model is tracking state or pattern-matching the shape of a step. Both would read as "quant hurts long chains" in a headline, but they need different fixes. One is precision. The other is a training signal.

Does the wobble on Shuffle scale with the same e from Letters, or is it worse, which would say TB is losing something Letters never touches?

Good, that's the number that mattered.

9 vs 8 workers is close enough to call equal concurrency, so the -12% wall and -43% p50 can't be a throttle effect, they have to be less thrash. That's the controller result I was asking for, not a throttle wearing a controller's clothes.

I had the idle-gap logic backwards too. You are not counting tool-call idle as busy, projected_idle_estimate treats it as free capacity for /capacity. That is the direction that avoids the failure mode I was worried about.

The overload cell answers the drop-rate question directly: neither gate drops anything at 2x past the knee, both just get slower. So the gap I asked about does not widen because there is no gap to widen. Once the GPU is the ceiling, a gate cannot manufacture throughput either way. P7's value lives entirely in the mid-load regime where headroom exists to schedule around.

That is a narrower claim than "better admission policy," and it is the honest one. In production, what tells an operator they are still in that mid-load regime versus already past the knee, before the p50 numbers show it?

Sort, Shuffle and Dates versus Shapes and Cars is not a random split.

The three that degrade are the ones where the model carries an exact intermediate state and updates it every step. The two that improved are closer to single shot. So a per-step error that is invisible on one hop compounds over n hops, which is why the loss reads as agentic rather than as a flat benchmark drop.

That makes it measurable instead of a vibe. If QAT costs you a per-step failure probability e, accuracy should fall roughly like (1-e)^n as you scale the number of steps. Fit that on Sort at increasing lengths, read e off the curve, and you can predict the agentic degradation on tasks you never ran.

The thinking-length result might be the same effect rather than a second one. A model slightly less inclined to re-check itself terminates earlier. On a short chain that reads as efficiency. On a long chain it reads as Sort breaking.

Does ReasonScape let you hold a task fixed and sweep chain length? That one curve would separate quant is worse at reasoning from quant loses e per step and n did the rest.