VIDRAFT_LAB

Yeah, that's the right read. It's not really a Metal thing, it's just the regime. Any single-stream, batch=1 decode is memory-bound, and since the MoE only touches its active experts each token, it moves fewer bytes per token and comes out ahead. CPU, Metal, Adreno, whatever — as long as you're bandwidth-bound instead of filling a fat batch, sparsity wins. Metal's not doing anything special, it just happens to be one of those backends.

The NPUs are where it gets fuzzy though. Two things: first, they're usually not even in the decode path. Our iPhone build runs through MLX on the GPU, not the ANE, and Snapdragon llama.cpp mostly leans on CPU/GPU too, so the NPU is basically sitting idle while the bandwidth-bound units do the work. Second, even if you did route to them, dynamic MoE is a bad fit — ANE and Hexagon want static dense matmul graphs, and per-token, data-dependent expert gather is exactly the thing they're bad at. You'd need a routing-aware compile path and that doesn't really exist yet.

So short version: it carries across bandwidth-bound on-device runtimes, but I wouldn't bet on it transferring to the NPUs as-is. And to be upfront, we've actually measured CPU and Metal — the ANE/Hexagon part is me reasoning from how those chips work, not something we've benchmarked. Efficient dynamic MoE on mobile NPUs is still kind of an open problem, and honestly a fun one.

Great question, thanks for digging into this 🙏 Your hypothesis ②(batch size) is essentially right.

The H100 prefill loss isn't really an inherent MoE limitation — it's a batching/GEMM-efficiency effect. A dense model packs its large-batch prefill into one big GEMM and saturates the tensor cores, while an MoE scatters tokens across experts into many small GEMMs with lower utilization. So in that regime, dense wins.

On the M3 Pro (Metal, single-stream), that "big-batch advantage" mostly isn't there — it's memory-bandwidth bound. What actually matters there is active params per token, so the sparse POCKET comes out ahead on prefill too.

So it's not that Metal does something special with sparse prefill — it's that the batch advantage dense enjoys on the H100 disappears on-device, exactly as you suspected. And since POCKET is built for that on-device regime (phone/laptop), we consider those the numbers that actually matter for its use case. Really appreciate the sharp observation! 🙌

Thanks for checking from your side — good to have it confirmed by someone logged out rather than just by us.

And no need to worry about the checkpoint part. Easy thing to miss.

The part you keep chewing on is the part worth keeping. The tag was a claim, gated: auto was a fact, and both sat in the same repo for the whole staging window with nothing reconciling them. We didn't catch it either — an anonymous pull from a stranger did.

To answer your question directly: yes. We already run a version of it before release — a logged-out pass that checks anonymous access against what the card claims, along with links and wording. It caught a few leftover lines on this release. It exists because of exactly the failure you're describing, and it's a small script rather than anything clever.

Which is sort of your point. If a few lines can compute that diff, it probably shouldn't be something each team rebuilds on its own.

Thanks again for pulling the repo in the first place.

You're right — good catch, and thank you for actually testing it instead of assuming.

It was a leftover switch, not intent. The base was gated during the private staging phase, and when we flipped everything public that one toggle didn't come along with it. Nothing in the license or the release plan ever called for a gate there.

It's fixed now. Aether-7B-5Attn is ungated — and I verified it from a logged-out client rather than trusting the settings page: anonymous file pulls return 200, and /refs resolves again.

One clarification while I'm here: the 110k / 115k / 162k checkpoints aren't branches of the base repo — they're folders in a separate public dataset, FINAL-Bench/Aether-7B-5Attn-checkpoints (~13 GB each, ungated). So enumerating refs was always going to be a dead end for those, quite apart from the 403.

And thank you for the point about the logs — that's the part we most hoped people would open rather than star. The multi-node run is messy in places, which is rather the point: it's the actual run, not a tidied-up excerpt.

The 35B model currently released for testing is being served through Hugging Face Spaces in an on-premise-style, GPU-free CPU-only environment.

The free CPU version provided by Hugging Face is running on a virtualized 2 vCPU + 16GB RAM instance, delivering approximately 5.5 to 6 tokens per second.

URL: https://FINAL-Bench-Ourbox-35B-VKUE-FreeCPU.hf.space

The 35B model currently released for testing is being served on Hugging Face Spaces in a fully on-premise-style environment with no GPU — CPU only.

It is running on an upgraded CPU instance with 8 virtual CPUs and 32GB of RAM, delivering approximately 20 tokens per second.

https://FINAL-Bench-Ourbox-35B-VKUE-CPU.hf.space