Instructions to use ernie-research/HiUni with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ernie-research/HiUni with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="ernie-research/HiUni", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("ernie-research/HiUni", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use ernie-research/HiUni with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ernie-research/HiUni" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ernie-research/HiUni", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/ernie-research/HiUni
- SGLang
How to use ernie-research/HiUni with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ernie-research/HiUni" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ernie-research/HiUni", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ernie-research/HiUni" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ernie-research/HiUni", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use ernie-research/HiUni with Docker Model Runner:
docker model run hf.co/ernie-research/HiUni
HiUni is a unified multimodal model initialized from Qwen3-VL-4B-Instruct. A language-aligned ViT and one dense backbone operate in a shared visual space for four capabilities: image understanding, text-to-image generation, image editing, and text chat. Generation-specific computation stays at the output in a Pixel Diffusion Head, with HiGate preserving fine pixel details.
Checkpoints
This repository provides two checkpoints in both Hugging Face and Megatron formats:
| Name | Hugging Face path | Megatron path | Recommended generation size |
|---|---|---|---|
| HiUni | Repository root | Megatron/HiUni/ |
Up to 2048 x 2048 |
| HiUni Base | HiUni-Base/ |
Megatron/HiUni-Base/ |
1024 x 1024 |
HiUni is the primary release. HiUni Base is trained without synthetic data.
Model details
| Property | Value |
|---|---|
| Architecture | Language-aligned ViT + dense Transformer + Pixel Diffusion Head |
| Initialization | Qwen3-VL-4B-Instruct |
| Shared backbone | 4B parameters |
| Generation-specific modules | 0.9B parameters |
| Image output | Direct RGB pixel-space diffusion |
| Maximum generation resolution | 2048 x 2048 |
| Recommended dtype | bfloat16 |
| Weight formats | Hugging Face safetensors (6 shards per checkpoint) and Megatron distributed checkpoint |
The same checkpoint supports all four capabilities. The shared visual pathway retains the starting model's understanding quality while adding high-resolution generation and editing.
These weights use EMA decay 0.95. The EMA starts from the hot-start checkpoint and is updated every 50 steps through step 8,000 of the final SFT run. This repository contains the main HiUni release.
Quick start
Install the inference dependencies:
pip install "torch>=2.6" "transformers==4.57.1" "safetensors>=0.4.5" \
"accelerate>=1.0" "Pillow>=10.0" "PyYAML>=6.0"
The repository includes custom modeling code, so loading requires
trust_remote_code=True. Review the code or pin a revision in production.
Text-to-image
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "ernie-research/HiUni"
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
dtype=torch.bfloat16,
attn_implementation="sdpa",
).to("cuda").eval()
tokenizer = AutoTokenizer.from_pretrained(model_id)
image = model.generate_image(
"A red fox standing in fresh snow.",
tokenizer=tokenizer,
height=1024,
width=1024,
num_steps=25,
guidance_scale=4.0,
shift=4.0,
seed=42,
)
image.save("hiuni.png")
Complete commands for text chat, image understanding, text-to-image generation, image editing, batch inference, and multi-GPU sharding are available in the HiUni inference guide.
HiUni Base
Load the earlier Stage-3 checkpoint from the HiUni-Base subfolder:
base_model = AutoModelForCausalLM.from_pretrained(
"ernie-research/HiUni",
subfolder="HiUni-Base",
trust_remote_code=True,
dtype=torch.bfloat16,
attn_implementation="sdpa",
).to("cuda").eval()
base_tokenizer = AutoTokenizer.from_pretrained(
"ernie-research/HiUni",
subfolder="HiUni-Base",
)
The recommended generation resolution for HiUni Base is height=1024 and width=1024.
Megatron checkpoints
The corresponding model-only Megatron distributed checkpoints are available under
Megatron/HiUni/ and Megatron/HiUni-Base/. Each directory is a regular Megatron
checkpoint root with latest_checkpointed_iteration.txt, an iter_* directory,
common.pt, metadata.json, and distributed checkpoint shards.
These releases contain model parameters only. They intentionally exclude optimizer, RNG,
dataloader, and other training-runtime state. Load them for inference or fine-tuning with
checkpoint.finetune: true and checkpoint.load_optim: false; they are not full-state
training-resume checkpoints. The included run_config.yaml records the model structure and
uses portable paths.
Showcase
Text-to-image
Text rendering
Image editing
Evaluation highlights
- Understanding: 75.56 average across seven benchmarks, compared with 75.61 for the Qwen3-VL-4B-Instruct initialization.
- Text rendering: Qwen-Image-Bench overall score 47.98; LongText ZH 0.963 and EN 0.954.
- Image editing: ImgEdit 3.66; GEdit CN 6.70 and EN 6.81, evaluated with a GPT-5.2 judge.
See the project page for full benchmark tables, comparisons, and ablations.
Intended use
HiUni is intended for research on unified multimodal understanding, generation, and editing. Generated or edited media may contain inaccuracies, unwanted artifacts, or biased content. Users are responsible for reviewing outputs and complying with applicable laws and policies, especially for public-facing or high-impact use.
Citation
@article{hu2026hiuni,
title = {HiUni: Harmonizing Understanding and Generation in Shared Visual Space},
author = {Hu, Jiahao and Du, Jingdong and Zhang, Yichen and Song, Yi and
Zhang, Zhenyu and Niu, Jiaxin and Li, Ruihang and Hao, Xiangzhao and
Zhang, Yiqian and Huang, Yong and Zhang, Zefeng and Gu, Zheyuan and
Wang, Shuohuan},
journal = {arXiv preprint arXiv:2611.26107},
year = {2026},
eprint = {2611.26107},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2611.26107}
}
HiUni is initialized from Qwen3-VL. Please also cite the Qwen3-VL project when using this model.
Acknowledgements
We thank the Qwen3-VL, Megatron-Bridge, Megatron-LM, NeMo, and FlashMask teams.
- Downloads last month
- -
Model tree for ernie-research/HiUni
Base model
Qwen/Qwen3-VL-4B-Instruct


