Spaces:
Running
NF4 grouped GEMM for MoE experts (Triton), and kernel publishing access
Hi! I’d like to publish kernels on the Hub, starting with this one: a grouped GEMM for MoE experts with weights in bitsandbytes’ NF4 format, so 4-bit MoE models run all experts in one launch on A100/H100, where there is no native 4-bit path. Repo (README, tests, H100 benchmarks): https://huggingface.co/LegumMagister/nf4-grouped-gemm
On an H100 with Qwen3-30B-A3B shapes it’s 3–14× faster than running bitsandbytes expert by expert, with values bit-identical to dequantize_4bit. It’s about 0.5× bf16 at small batches for now; a Marlin/FLUTE-style CUDA version is next.
I write kernels with Claude, and I test and benchmark each one myself before publishing: correctness against the reference implementation, and timings against the strongest existing baseline, on the hardware it targets. I’ve opened PRs to transformers and tokenizers, and I’ve trained LLMs at scale (I worked on the Apertus models on the Alps supercomputer).
I’ve requested kernel publishing access. I’d be glad to publish more kernels as I write them, and to help with kernels-community if that’s useful. Thanks!