Instructions to use atakans/Hy4-flash-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use atakans/Hy4-flash-NVFP4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="atakans/Hy4-flash-NVFP4") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForSeq2SeqLM model = AutoModelForSeq2SeqLM.from_pretrained("atakans/Hy4-flash-NVFP4", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use atakans/Hy4-flash-NVFP4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "atakans/Hy4-flash-NVFP4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "atakans/Hy4-flash-NVFP4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/atakans/Hy4-flash-NVFP4
- SGLang
How to use atakans/Hy4-flash-NVFP4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "atakans/Hy4-flash-NVFP4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "atakans/Hy4-flash-NVFP4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "atakans/Hy4-flash-NVFP4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "atakans/Hy4-flash-NVFP4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use atakans/Hy4-flash-NVFP4 with Docker Model Runner:
docker model run hf.co/atakans/Hy4-flash-NVFP4
Hy4-flash NVFP4
An NVFP4 build of tencent/Hy4-flash made with NVIDIA ModelOpt 0.48. The MLPs are NVFP4. The Engram tables are FP8 with one scale per row. Everything else stays BF16. The checkpoint is 164.7 GB (153.4 GiB), against 440 GB for BF16 and 287 GB for Tencent's FP8.
The target is two DGX Spark (2 x 128 GB) with vLLM and tensor parallel 2.
Status
It runs on two DGX Spark with vLLM, tensor parallel 2 with expert parallel, and the model's DFly drafter. No released vLLM runs it yet: Hy4-flash support is an open pull request (vllm-project/vllm#68296), and that code takes FP8 only and needs Hopper or data-center Blackwell for attention. I run the official vllm/vllm-openai:hy4-flash image with five small Python changes: Hy4's learnable attention sink through FlashInfer's SM120 sparse MLA kernel (the kernel already takes a sink), ModelOpt NVFP4 expert loading, and the same two for the DFly drafter. The recipe with these files is in atakann/dgx-spark-recipes (llm/hy4-flash-nvfp4-vllm-tp2). I will send the changes to vLLM.
Measured on two DGX Spark
vLLM (hy4-flash image, 77766db) with the changes above, TP=2 over the QSFP link (RoCE), expert parallel, DFly with 5 draft tokens, fp8_ds_mla KV cache, 262,144 tokens of context, one user, thinking off (reasoning_effort: no_think), 2026-11-18.
| Prompt (300 tokens out, median of 3) | Decode tok/s | DFly accepted tokens a step |
|---|---|---|
| JSON | 38.3 | 3.99 |
| Code | 36.5 | 3.80 |
| Rewrite | 36.1 | 3.79 |
| Explain | 25.0 | 2.75 |
| Story | 17.3 | 1.85 |
Without the drafter: about 15 tok/s.
| Prompt depth | Prefill tok/s | Decode tok/s, first turn | Decode tok/s, second turn |
|---|---|---|---|
| 4K | 1,110 | 20.8 | 29.6 |
| 16K | 1,403 | 22.1 | 36.8 |
| 32K | 1,461 | 26.6 | 27.0 |
Memory: the weights load in 104 s, 51.7 GiB on each Spark's GPU side plus half of the Engram tables in host memory. With a 6 GiB KV cache (578K tokens) the lowest free memory was 12.5 GiB on one Spark and 13.0 GiB on the other.
Quality, thinking off (reasoning_effort: no_think), temperature 0, up to 16,384 tokens a reply, on this two-Spark setup:
| Benchmark | Score | 95% interval | Replies cut at the limit |
|---|---|---|---|
| GPQA Diamond (198) | 72.7% | 66.1 to 78.5 | 1 |
| IFEval (541 prompts) | 83.5% | 80.2 to 86.4 | 1 |
I have no BF16 or FP8 run of Hy4-flash on the same harness to set these against: neither fits two Sparks.
The config.json, tokenizer and chat template in this repository are Tencent's original files; hf_quant_config.json is ModelOpt's.
What is quantized
| Part | Format | Size |
|---|---|---|
| Routed experts (512 per layer, 16 active) | NVFP4 | 83.9 GB |
| Shared expert and the dense MLP of layer 0 | NVFP4 | in the line above |
| Engram tables (layers 2, 12, 22, 32; 165M rows x 96 each) | FP8 E4M3, one BF16 scale per row | 63.4 GB |
| Attention, sparse indexer, router, latent MoE projections, Engram projections, embeddings, lm_head, vision tower, DFly2 drafter | BF16 | 17.4 GB |
Tensor types in the files: U8 holds the NVFP4 weights (two 4-bit E2M1 values per byte, so Hugging Face counts half the real parameters and shows an "8-bit" tag), F8_E4M3 holds the NVFP4 block scales and the Engram rows, BF16 the unquantized parts and the Engram row scales, F32 the per-tensor and input scales, I32 the Engram lookup tables. NVIDIA's own NVFP4 checkpoints show the same mix.
NVFP4 here is the ModelOpt default: E2M1 weights and activations (W4A4), one FP8 E4M3 scale per 16 elements, one FP32 scale per tensor, static input scales from calibration. The routed experts are stored one tensor per expert (experts.N.gate_proj, up_proj, down_proj with weight, weight_scale, weight_scale_2, input_scale), the layout ModelOpt NVFP4 MoE loaders read.
The Engram tables are hash lookup tables, not matrix multiplies, so NVFP4 GEMM kernels do not apply to them. They use the format the Hy4-flash vLLM code already reads: FP8 rows in multi_head_embedding.embedding.weight and a BF16 per-row scale in multi_head_embedding.embedding.weight_scale_inv (value = row x scale, scale = row max / 448). config.json sets engram_fp8_embedding: true.
How it was made
- Source: tencent/Hy4-flash BF16, weights and
config.jsonof 2026-11-17 (later commits change only the README, assets and finetune files). - Tool: NVIDIA ModelOpt 0.48.0,
NVFP4_DEFAULT_CFGwith BF16 rules added for the parts in the table above. The fused 3-D expert tensors go through ModelOpt's fused-experts path, so each expert gets its own scales. - Calibration data: ModelOpt's
nemotron-post-training-v3mix (NVIDIA's open post-training sets for chat, science, code, agentic use, math, SWE and multilingual text), 128 samples of up to 2,048 tokens. NVFP4 weight scales come from the weights alone; the calibration data sets the activation scales. - Engram: a separate pass after export converts each table to FP8 with a per-row scale.
- Hardware: 8 x RTX PRO 6000 Blackwell Max-Q. Calibration took 42 minutes, export 5 minutes.
Checks done so far
| Check | Result |
|---|---|
| Expert weights, dequantized and compared with BF16 (gate, up and down of 3 experts in layers 5, 20, 40) | 9.5 % relative error each, the normal level for NVFP4 |
| Engram tables, FP8 against BF16 | 2.6 % relative RMS error on each of the 4 tables |
| File check after upload | 68 files, sizes match the local export |
These are weight-level checks.
License
Hy4-flash is released under the Tencent Hy License. This build keeps that license; see LICENSE.
- Downloads last month
- 126