Great question, thanks for digging into this 🙏 Your hypothesis ②(batch size) is essentially right.

The H100 prefill loss isn't really an inherent MoE limitation — it's a batching/GEMM-efficiency effect. A dense model packs its large-batch prefill into one big GEMM and saturates the tensor cores, while an MoE scatters tokens across experts into many small GEMMs with lower utilization. So in that regime, dense wins.

On the M3 Pro (Metal, single-stream), that "big-batch advantage" mostly isn't there — it's memory-bandwidth bound. What actually matters there is active params per token, so the sparse POCKET comes out ahead on prefill too.

So it's not that Metal does something special with sparse prefill — it's that the batch advantage dense enjoys on the H100 disappears on-device, exactly as you suspected. And since POCKET is built for that on-device regime (phone/laptop), we consider those the numbers that actually matter for its use case. Really appreciate the sharp observation! 🙌

Yeah, that's the right read. It's not really a Metal thing, it's just the regime. Any single-stream, batch=1 decode is memory-bound, and since the MoE only touches its active experts each token, it moves fewer bytes per token and comes out ahead. CPU, Metal, Adreno, whatever — as long as you're bandwidth-bound instead of filling a fat batch, sparsity wins. Metal's not doing anything special, it just happens to be one of those backends.

The NPUs are where it gets fuzzy though. Two things: first, they're usually not even in the decode path. Our iPhone build runs through MLX on the GPU, not the ANE, and Snapdragon llama.cpp mostly leans on CPU/GPU too, so the NPU is basically sitting idle while the bandwidth-bound units do the work. Second, even if you did route to them, dynamic MoE is a bad fit — ANE and Hexagon want static dense matmul graphs, and per-token, data-dependent expert gather is exactly the thing they're bad at. You'd need a routing-aware compile path and that doesn't really exist yet.

So short version: it carries across bandwidth-bound on-device runtimes, but I wouldn't bet on it transferring to the NPUs as-is. And to be upfront, we've actually measured CPU and Metal — the ANE/Hexagon part is me reasoning from how those chips work, not something we've benchmarked. Efficient dynamic MoE on mobile NPUs is still kind of an open problem, and honestly a fun one.