Hy4-flash NVFP4

An NVFP4 build of tencent/Hy4-flash made with NVIDIA ModelOpt 0.48. The MLPs are NVFP4. The Engram tables are FP8 with one scale per row. Everything else stays BF16. The checkpoint is 164.7 GB (153.4 GiB), against 440 GB for BF16 and 287 GB for Tencent's FP8.

The target is two DGX Spark (2 x 128 GB) with vLLM and tensor parallel 2.

Status

It runs on two DGX Spark with vLLM, tensor parallel 2 with expert parallel, and the model's DFly drafter. No released vLLM runs it yet: Hy4-flash support is an open pull request (vllm-project/vllm#68296), and that code takes FP8 only and needs Hopper or data-center Blackwell for attention. I run the official vllm/vllm-openai:hy4-flash image with five small Python changes: Hy4's learnable attention sink through FlashInfer's SM120 sparse MLA kernel (the kernel already takes a sink), ModelOpt NVFP4 expert loading, and the same two for the DFly drafter. The recipe with these files is in atakann/dgx-spark-recipes (llm/hy4-flash-nvfp4-vllm-tp2). I will send the changes to vLLM.

Measured on two DGX Spark

vLLM (hy4-flash image, 77766db) with the changes above, TP=2 over the QSFP link (RoCE), expert parallel, DFly with 5 draft tokens, fp8_ds_mla KV cache, 262,144 tokens of context, one user, thinking off (reasoning_effort: no_think), 2026-11-18.

Prompt (300 tokens out, median of 3) Decode tok/s DFly accepted tokens a step
JSON 38.3 3.99
Code 36.5 3.80
Rewrite 36.1 3.79
Explain 25.0 2.75
Story 17.3 1.85

Without the drafter: about 15 tok/s.

Prompt depth Prefill tok/s Decode tok/s, first turn Decode tok/s, second turn
4K 1,110 20.8 29.6
16K 1,403 22.1 36.8
32K 1,461 26.6 27.0

Memory: the weights load in 104 s, 51.7 GiB on each Spark's GPU side plus half of the Engram tables in host memory. With a 6 GiB KV cache (578K tokens) the lowest free memory was 12.5 GiB on one Spark and 13.0 GiB on the other.

Quality, thinking off (reasoning_effort: no_think), temperature 0, up to 16,384 tokens a reply, on this two-Spark setup:

Benchmark Score 95% interval Replies cut at the limit
GPQA Diamond (198) 72.7% 66.1 to 78.5 1
IFEval (541 prompts) 83.5% 80.2 to 86.4 1

I have no BF16 or FP8 run of Hy4-flash on the same harness to set these against: neither fits two Sparks.

The config.json, tokenizer and chat template in this repository are Tencent's original files; hf_quant_config.json is ModelOpt's.

What is quantized

Part Format Size
Routed experts (512 per layer, 16 active) NVFP4 83.9 GB
Shared expert and the dense MLP of layer 0 NVFP4 in the line above
Engram tables (layers 2, 12, 22, 32; 165M rows x 96 each) FP8 E4M3, one BF16 scale per row 63.4 GB
Attention, sparse indexer, router, latent MoE projections, Engram projections, embeddings, lm_head, vision tower, DFly2 drafter BF16 17.4 GB

Tensor types in the files: U8 holds the NVFP4 weights (two 4-bit E2M1 values per byte, so Hugging Face counts half the real parameters and shows an "8-bit" tag), F8_E4M3 holds the NVFP4 block scales and the Engram rows, BF16 the unquantized parts and the Engram row scales, F32 the per-tensor and input scales, I32 the Engram lookup tables. NVIDIA's own NVFP4 checkpoints show the same mix.

NVFP4 here is the ModelOpt default: E2M1 weights and activations (W4A4), one FP8 E4M3 scale per 16 elements, one FP32 scale per tensor, static input scales from calibration. The routed experts are stored one tensor per expert (experts.N.gate_proj, up_proj, down_proj with weight, weight_scale, weight_scale_2, input_scale), the layout ModelOpt NVFP4 MoE loaders read.

The Engram tables are hash lookup tables, not matrix multiplies, so NVFP4 GEMM kernels do not apply to them. They use the format the Hy4-flash vLLM code already reads: FP8 rows in multi_head_embedding.embedding.weight and a BF16 per-row scale in multi_head_embedding.embedding.weight_scale_inv (value = row x scale, scale = row max / 448). config.json sets engram_fp8_embedding: true.

How it was made

  • Source: tencent/Hy4-flash BF16, weights and config.json of 2026-11-17 (later commits change only the README, assets and finetune files).
  • Tool: NVIDIA ModelOpt 0.48.0, NVFP4_DEFAULT_CFG with BF16 rules added for the parts in the table above. The fused 3-D expert tensors go through ModelOpt's fused-experts path, so each expert gets its own scales.
  • Calibration data: ModelOpt's nemotron-post-training-v3 mix (NVIDIA's open post-training sets for chat, science, code, agentic use, math, SWE and multilingual text), 128 samples of up to 2,048 tokens. NVFP4 weight scales come from the weights alone; the calibration data sets the activation scales.
  • Engram: a separate pass after export converts each table to FP8 with a per-row scale.
  • Hardware: 8 x RTX PRO 6000 Blackwell Max-Q. Calibration took 42 minutes, export 5 minutes.

Checks done so far

Check Result
Expert weights, dequantized and compared with BF16 (gate, up and down of 3 experts in layers 5, 20, 40) 9.5 % relative error each, the normal level for NVFP4
Engram tables, FP8 against BF16 2.6 % relative RMS error on each of the 4 tables
File check after upload 68 files, sizes match the local export

These are weight-level checks.

License

Hy4-flash is released under the Tencent Hy License. This build keeps that license; see LICENSE.

Downloads last month
126
Safetensors
Model size
145B params
Tensor type
F32
·
BF16
·
F8_E4M3
·
U8
·
I32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for atakans/Hy4-flash-NVFP4

Finetuned
(5)
this model
Quantizations
1 model