ModaLeap/hy-mt2-1.8b-onnx

Hy-MT2-1.8B INT4 ONNX

This model is used by https://github.com/mercallureAI/local-multimodal-infra .

This repository provides an INT4 ONNX export of tencent/Hy-MT2-1.8B for ONNX Runtime with the CUDA execution provider.

Original model:

https://huggingface.co/tencent/Hy-MT2-1.8B

This is not an official Hy-MT2 release.

Description

Hy-MT2-1.8B is Tencent Hunyuan's translation model: it translates between 33 languages and takes terminology, background information and earlier lines in its prompt. At 1.8B parameters it is fast enough to translate live captions as they are spoken.

The package was built with the onnxruntime-genai model builder:

python -m onnxruntime_genai.models.builder -i <Hy-MT2-1.8B> -o <dir> -p int4 -e cuda \
  --extra_options int4_algo_config=rtn_last prune_lm_head=true

• Weight-only INT4 (MatMulNBits, round-to-nearest), activations FP16.
• GroupQueryAttention over a KV cache shared between past and present, so a runtime can keep the cache of the previous request and only prefill what follows the shared prefix.
• The LM head is computed for the last position only (prune_lm_head).
• The builder's model.onnx is published as model_int4.onnx (its weights as model_int4.onnx.data); genai_config.json names it.
• One fix to make the builder run: it reads rope_theta from the model config, which Hy-MT2 keeps under rope_parameters. A wrapper set rope_theta from there before building. The graph itself is the builder's.

Files

File	Role
model_int4.onnx + model_int4.onnx.data	Decoder (INT4 weights, FP16 activations, GQA KV cache)
genai_config.json	onnxruntime-genai configuration: inputs, outputs, KV cache layout, special token ids
tokenizer.json, tokenizer_config.json	Tokenizer, copied from the original model
chat_template.jinja	Chat template, copied from the original model
manifest.json	Source, license, size and SHA-256 of every file
config.json	Package summary; the Hub counts downloads by requests for this file. Not read at run time
LICENSE	License of the original model (Apache-2.0)

Use

The directory works with onnxruntime-genai and with local-multimodal-infra's qwen3_chat adapter (Rust, ONNX Runtime; reuses the KV cache of the previous request):

hf download ModaLeap/hy-mt2-1.8b-onnx --local-dir workdir/models/hy-mt2-1.8b-int4-onnx

Use the official prompt templates of the model card (no system prompt, one user turn), for example:

将以下文本翻译为英语，注意只需要输出翻译后的结果，不要额外解释：

<text>

The card also gives the terminology (参考下面的翻译：X 翻译成 Y) and background (【背景信息】) templates, and English versions for pairs without Chinese.

Runtime notes:

• Targets NVIDIA GPUs (CUDA execution provider).
• About 2.1 GB of GPU memory with a 4,096-token KV cache (about 64 KB per token), as loaded on an RTX 4090.

Testing

Measured with the official prompt templates on an RTX 4090 (ONNX Runtime 1.31), greedy decoding:

• About 150 generated tokens per second.
• chrF on 100 FLEURS sentences per direction: 36.1 English to Chinese, 55.5 Chinese to English (Qwen3-4B-Instruct-2507 INT4 under the same setup: 34.2 and 54.7).
• All 16 glossary terms of a 10-line terminology test were used as given.

These are quick checks, not a benchmark against the original model. INT4 output differs from the BF16 model.

Source

Item	Revision
tencent/Hy-MT2-1.8B	9a341cd1b679d3efd23b46e847b01745a71ed792

License

Apache-2.0, as the original model. See LICENSE, which keeps Tencent's copyright notice.

All rights to the original model belong to Tencent.

Disclaimer

This is an unofficial ONNX export of Hy-MT2-1.8B. It is not affiliated with, maintained by, or endorsed by Tencent.