Hy3 295B-A21B — NVFP4 for veloGB10 (NVIDIA DGX Spark / GB10)
Original model: tencent/Hy3 by Tencent's Hunyuan team, Apache-2.0. This artifact contains modified weights (NVFP4 quantization); the original model is © its authors.
NVFP4-quantized variant of Hy3 (295B total / ~21B activated per token, MoE). This quantized
artifact exists first and foremost to run with the veloGB10
inference engine (gb10_inference) on NVIDIA DGX Spark / GB10 systems — it is produced,
validated, and tuned for veloGB10's kernels. The weights use the standard HuggingFace
compressed-tensors layout (nvfp4-pack-quantized), so this artifact can be used for any
purpose, with any framework that reads the format. It is in particular fit and proven to work
with veloGB10 — two-node TP=2 serving, native NextN speculative decoding, and the engine's
greedy-lossless correctness gates all pass on this artifact.
What this pack is: sources and credits
- Original model: tencent/Hy3 by Tencent's Hunyuan team, Apache-2.0.
- This artifact: NVFP4 quantization made by us from the original with veloGB10's offline quantizer;
compressed-tensorslayout. Weights are modified relative to the original; tokenizer and chat template are as shipped by the original. - Engine: veloGB10 (Apache-2.0).
Specifications
| Base model | tencent/Hy3 |
| Architecture | MoE transformer, 80 layers, hidden 4096 |
| Attention | GQA — 64 Q / 8 KV heads, head_dim 128, per-head qk-norm |
| MoE | 192 routed experts, top-8 + 1 shared (expert inter 1536); sigmoid router with aux-loss-free bias, ×2.826 scaling |
| Dense layers | first 1 layer dense MLP (inter 13312) |
| MTP head | native NextN prediction layer (used for speculative decoding) |
| Context | 262,144 tokens |
| Vocabulary | 120,832 |
| Quantization | all,-router,-embed,-lmhead — all GEMMs NVFP4 except router, embeddings, and LM head |
| Format | nvfp4-pack-quantized (compressed-tensors compatible) |
| On-disk size |
Running it with veloGB10
Download a release binary of veloGB10 (one self-contained file; see its README), and the pack with hf download doth4580/Tencent-Hy3-295B-A21B-NVFP4 --local-dir /path/to/Tencent-Hy3-295B-A21B-NVFP4. The server is OpenAI-compatible on http://<head-ip>:9000/v1. The flags below are the ones this pack was tested with; --max-batch sets how many requests run at once.
Not part of the November 2026 test run. These are the standard veloGB10 command lines for this pack; it was served at the topologies shown when it was published. TP=2 is required: the model does not fit one GB10.
Two GB10s (TP=2)
On the second machine (no model copy, no configuration):
./gb10_inference --node --port 29500
On the head (the machine with the model folder):
./gb10_inference --server \
--model-dir /path/to/Tencent-Hy3-295B-A21B-NVFP4 \
--tp 2 --nodes <node-ip>:29500 \
--host 0.0.0.0 --port 9000 \
--max-seq-len 32768 \
--prefix-cache on
The nodes need no copy of the model. The head sends each node only its own share of the weights over the network and keeps it in a local cache, so a second start of the same model transfers nothing. On a box with two active RoCE port pairs, name the one to use with --rdma-dev <rail1>,<rail2>; with one pair it is found automatically.
Notes
- Tokenizer, chat template, and generation config are included as shipped by the quantization
pipeline (chat format: Hunyuan
hy_*tags with<think>blocks). - Quantization can degrade output quality relative to the bf16 original; validate on your own workload. Report issues via the veloGB10 issue tracker.
- Intended for research and local/agentic serving use, under the base model's Apache-2.0 terms.
Intellectual property
We do not claim any intellectual-property ownership of the model, its weights, its architecture or its original documentation. All rights stay with the original authors and with every contributor credited above, under their own licences, which apply to this repository unchanged. Our contribution is limited to what the "What this pack is" section says we did (quantization, packaging, testing and the veloGB10 instructions). If you are a rights holder and want something here changed or removed, open a discussion on this repository.
Original model card — tencent/Hy3
Reproduced verbatim from the original repository (metadata header removed).
中文 | English
🖥️ Official Website | 💬 GitHub
Table of Contents
- Model Introduction
- Stronger Agent Capabilities
- More Reliable Product Experiences
- Benchmark Appendix
- News
- Model Links
- Quickstart
- Deployment
- Finetuning
- RL Post-training
- Quantization
- License
- Contact Us
Model Introduction
Hy3 is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters and 3.8B MTP layer parameters, developed by the Tencent Hy Team. Following the Hy3 Preview launch in late April, we gathered feedback from 50+ products and scaled up post-training with higher quality data. Today, we introduce Hy3, which outperforms similar-size models and rivals flagship open-source models with 2-5x parameters. It also shows significant gains in utility across various products and productivity tasks.
| Property | Value |
|---|---|
| Architecture | Mixture-of-Experts (MoE) |
| Total Parameters | 295B |
| Activated Parameters | 21B |
| MTP Layer Parameters | 3.8B |
| Number of Layers (excluding MTP layer) | 80 |
| Number of MTP Layers | 1 |
| Attention Heads | 64 (GQA, 8 KV heads, head dim 128) |
| Hidden Size | 4096 |
| Intermediate Size | 13312 |
| Context Length | 256K |
| Vocabulary Size | 120832 |
| Number of Experts | 192 experts, top-8 activated |
| Supported Precisions | BF16 |
Stronger Agent Capabilities
Building on Hy3 Preview, we further improved the quality and diversity of post-training data while scaling up RL training. Hy3 shows solid gains across reasoning, agentic, and long-context tasks, competitive with much larger flagship models.
In productivity scenarios such as coding, office work, financial modeling, frontend design, and game development, Hy3 has made remarkable progress and can now serve as a reliable, cost-effective model option.
We don't think public benchmark scores tell the full story. So we ran a blind evaluation with 270 experts using tasks from their work, and Hy3 scored 2.67/4, outperforming GLM-5.1 at 2.51/4. The advantage was most substantial in frontend development, data & storage, and CI/CD tasks.
More Reliable Product Experiences
Model usefulness is not fully captured by benchmarks. Based on extensive product feedback, we identified and fixed the following issues, receiving consistently positive feedback from product teams.
Stability of tool calls and output formats: We fixed multiple baseline reliability issues, bringing the model to production-grade standards across tool configurations and output constraints. Tool-call error recovery and overall efficiency improved. Hy3 also generalizes across different agent scaffoldings. On SWE-Bench Verified, accuracy variance across scaffoldings like CodeBuddy, Cline, and KiloCode remains within 4%.
Knowledge and anti-hallucination: Guided by the ideal of "answer when grounded, state when evidence is missing, do not conflate sources or fabricate data," we implemented fine-grained data cleaning and training constraints. In internal evaluations based on real-world scenarios, Hy3's hallucination rate dropped from 12.5% to 5.4%, and commonsense error rates fell from 25.4% to 12.7%. These improvements materially reduce fact conflation, fabrication, and logical contradiction.
Complex context retention and multi-turn intent tracking: Through joint optimization of SFT and RL, Hy3 improved on operational pain points like coreference resolution, ellipsis recovery, and multi-turn constraint inheritance. On internal comprehensive multi-turn tests, the issue rate dropped from 17.4% to 7.9%. Hy3 also improved markedly on long-dialogue evals like MRCR. Its outputs are more concise while ensuring complex intents do not decay or drift over long-horizon interactions.
Benchmark Appendix
News
- 🔥 We open-source Hy3 and Hy3-FP8 model weights on Hugging Face, ModelScope, GitCode, and CNB.
Model Links
| Model Name | Description | Hugging Face | ModelScope | GitCode | CNB |
|---|---|---|---|---|---|
| Hy3 | Instruct model | 🤗 Model | Model | Model | Model |
| Hy3-FP8 | FP8 quantized instruct model | 🤗 Model | Model | Model | Model |
Quickstart
Deploy Hy3 with vLLM or SGLang first, then call the OpenAI-compatible API:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="hy3",
messages=[
{"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
],
temperature=0.9,
top_p=1.0,
# reasoning_effort: "no_think" (default, direct response), "low", "high" (deep chain-of-thought)
extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)
Recommended parameters:
temperature=0.9,top_p=1.0.Reasoning mode: Set
reasoning_effortto"high"for complex tasks (math, coding, reasoning) or"no_think"for direct responses.
See the Deployment section below for how to start the API server.
Deployment
Hy3 has 295B parameters in total. To serve it on 8 GPUs, we recommend using H20-3e or other GPUs with larger memory capacity.
For production serving, we recommend using vLLM or SGLang, both of which provide dedicated recipes for Hy3:
vLLM - see vLLM recipes
SGLang - see SGLang cookbook
vLLM
Build vLLM from source:
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install --editable . --torch-backend=auto
Start the vLLM server with MTP enabled:
# Switch to trtllm backend to work-around mnnvl workspace size issue.
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
vllm serve tencent/Hy3 \
--tensor-parallel-size 8 \
--speculative-config.method mtp \
--speculative-config.num_speculative_tokens 2 \
--tool-call-parser hy_v3 \
--reasoning-parser hy_v3 \
--enable-auto-tool-choice \
--port 8000 \
--served-model-name hy3
SGLang
Build SGLang from source:
git clone https://github.com/sgl-project/sglang
cd sglang
pip3 install pip --upgrade
pip3 install "transformers>=5.6.0"
pip3 install -e "python"
Launch SGLang server with MTP enabled:
python3 -m sglang.launch_server \
--model tencent/Hy3 \
--tp-size 8 \
--tool-call-parser hunyuan \
--reasoning-parser hunyuan \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 3 \
--speculative-algorithm EAGLE \
--port 8000 \
--served-model-name hy3
Finetuning
Hy3 provides a complete model finetuning pipeline. For detailed documentation, please refer to: Finetuning Guide
RL Post-training
Hy3 supports GRPO reinforcement learning training with verl, training on Megatron-LM (model conversion via NVIDIA Megatron-Bridge) with vLLM rollout. For detailed documentation, please refer to: RL Training Guide
Quantization
We provide AngelSlim, a more accessible, comprehensive, and efficient toolkit for large model compression. AngelSlim supports a comprehensive suite of compression tools for large-scale multimodal models, including common quantization algorithms, low-bit quantization, and speculative sampling.
License
Hy3 is released under the Apache License 2.0. See LICENSE for details.
Contact Us
If you would like to leave a message for our R&D and product teams, welcome to contact us. You can also reach us via email:
📧 hunyuan_opensource@tencent.com
Hy3 is developed by the Tencent Hy Team.
- Downloads last month
- 46
Model tree for doth4580/Tencent-Hy3-295B-A21B-NVFP4
Base model
tencent/Hy3