Datapoint: fp4 vs fp8 GLM-5.3-Flash endpoint parity on GSM8K + HumanEval

#8
by joelafrite - opened

Small quality datapoint for the fp4-anxiety file. We A/B'd GLM-5.3-Flash via OpenRouter with the provider pinned per arm and fallbacks disabled (fp4 = Relace, fp8 = Z.AI), greedy, provider identity verified in every response:

  • GSM8K (n=400): fp4 98.2% vs fp8 98.8% — within noise, both arms missing mostly the same items.
  • HumanEval pass@1 (n=164, completions executed against the official tests): 157/164 = 95.7% on both arms exactly, 4 shared fails, 3 unique fails each side.

Caveats: we don't know which fp4 build Relace serves (not necessarily this repo's, and "fp4" isn't confirmed NVFP4); math+code only — no agentic/tool-loop testing. Still, for anyone worried after the input_scale episode: no visible fp4 damage on either benchmark.

LibertAI org

Thanks for the careful A/B, this is a useful datapoint. One thing we've measured since that's relevant to the fp4 question: it matters whether the routed experts run as W4A4 (activations also quantized to FP4) or W4A16 (FP4 weights, 16-bit activations). On our own deployment W4A4 cost about 4% perplexity vs a 16-bit-activation reference and noticeably hurt short-context factual recall. Running the same NVFP4 weights as W4A16 (--moe-backend marlin on vLLM) brought the gap down to about 1.3%, and on our sm_120 boxes it was also faster. So if you see fp4 quality issues on a provider, the activation path is the first thing we'd suspect, not the weights.

Thanks, that's a useful distinction. Our arms went through OpenRouter with the provider pinned, so we can't see which activation path Relace runs; treat our result as an endpoint datapoint, not a statement about W4A4 kernels. GSM8K/HumanEval at n=400/164 may also be too coarse to show a ~4% PPL gap, and short-context factual recall is exactly what we didn't test.

Two questions:

  • Could you share the PPL setup (dataset, context length, 16-bit reference arm), so others can reproduce W4A4 vs W4A16?
  • For 2× GB10 at TP=2, where the card notes the marlin repack (~19 GiB/rank) doesn't fit: is there a W4A16-style path that does, or is W4A4 via flashinfer_cutlass the only option there for now?

Sign up or log in to comment