HiUni banner

HiUni: Harmonizing Understanding and Generation in Shared Visual Space

Project Page GitHub arXiv

HiUni is a unified multimodal model initialized from Qwen3-VL-4B-Instruct. A language-aligned ViT and one dense backbone operate in a shared visual space for four capabilities: image understanding, text-to-image generation, image editing, and text chat. Generation-specific computation stays at the output in a Pixel Diffusion Head, with HiGate preserving fine pixel details.

Checkpoints

This repository provides two checkpoints in both Hugging Face and Megatron formats:

Name Hugging Face path Megatron path Recommended generation size
HiUni Repository root Megatron/HiUni/ Up to 2048 x 2048
HiUni Base HiUni-Base/ Megatron/HiUni-Base/ 1024 x 1024

HiUni is the primary release. HiUni Base is trained without synthetic data.

HiUni architecture

Model details

Property Value
Architecture Language-aligned ViT + dense Transformer + Pixel Diffusion Head
Initialization Qwen3-VL-4B-Instruct
Shared backbone 4B parameters
Generation-specific modules 0.9B parameters
Image output Direct RGB pixel-space diffusion
Maximum generation resolution 2048 x 2048
Recommended dtype bfloat16
Weight formats Hugging Face safetensors (6 shards per checkpoint) and Megatron distributed checkpoint

The same checkpoint supports all four capabilities. The shared visual pathway retains the starting model's understanding quality while adding high-resolution generation and editing.

These weights use EMA decay 0.95. The EMA starts from the hot-start checkpoint and is updated every 50 steps through step 8,000 of the final SFT run. This repository contains the main HiUni release.

Quick start

Install the inference dependencies:

pip install "torch>=2.6" "transformers==4.57.1" "safetensors>=0.4.5" \
  "accelerate>=1.0" "Pillow>=10.0" "PyYAML>=6.0"

The repository includes custom modeling code, so loading requires trust_remote_code=True. Review the code or pin a revision in production.

Text-to-image

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ernie-research/HiUni"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
).to("cuda").eval()
tokenizer = AutoTokenizer.from_pretrained(model_id)

image = model.generate_image(
    "A red fox standing in fresh snow.",
    tokenizer=tokenizer,
    height=1024,
    width=1024,
    num_steps=25,
    guidance_scale=4.0,
    shift=4.0,
    seed=42,
)
image.save("hiuni.png")

Complete commands for text chat, image understanding, text-to-image generation, image editing, batch inference, and multi-GPU sharding are available in the HiUni inference guide.

HiUni Base

Load the earlier Stage-3 checkpoint from the HiUni-Base subfolder:

base_model = AutoModelForCausalLM.from_pretrained(
    "ernie-research/HiUni",
    subfolder="HiUni-Base",
    trust_remote_code=True,
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
).to("cuda").eval()
base_tokenizer = AutoTokenizer.from_pretrained(
    "ernie-research/HiUni",
    subfolder="HiUni-Base",
)

The recommended generation resolution for HiUni Base is height=1024 and width=1024.

Megatron checkpoints

The corresponding model-only Megatron distributed checkpoints are available under Megatron/HiUni/ and Megatron/HiUni-Base/. Each directory is a regular Megatron checkpoint root with latest_checkpointed_iteration.txt, an iter_* directory, common.pt, metadata.json, and distributed checkpoint shards.

These releases contain model parameters only. They intentionally exclude optimizer, RNG, dataloader, and other training-runtime state. Load them for inference or fine-tuning with checkpoint.finetune: true and checkpoint.load_optim: false; they are not full-state training-resume checkpoints. The included run_config.yaml records the model structure and uses portable paths.

Showcase

Text-to-image

HiUni text-to-image samples

Text rendering

HiUni text rendering samples

Image editing

HiUni image editing samples

Evaluation highlights

  • Understanding: 75.56 average across seven benchmarks, compared with 75.61 for the Qwen3-VL-4B-Instruct initialization.
  • Text rendering: Qwen-Image-Bench overall score 47.98; LongText ZH 0.963 and EN 0.954.
  • Image editing: ImgEdit 3.66; GEdit CN 6.70 and EN 6.81, evaluated with a GPT-5.2 judge.

See the project page for full benchmark tables, comparisons, and ablations.

Intended use

HiUni is intended for research on unified multimodal understanding, generation, and editing. Generated or edited media may contain inaccuracies, unwanted artifacts, or biased content. Users are responsible for reviewing outputs and complying with applicable laws and policies, especially for public-facing or high-impact use.

Citation

@article{hu2026hiuni,
  title = {HiUni: Harmonizing Understanding and Generation in Shared Visual Space},
  author = {Hu, Jiahao and Du, Jingdong and Zhang, Yichen and Song, Yi and
            Zhang, Zhenyu and Niu, Jiaxin and Li, Ruihang and Hao, Xiangzhao and
            Zhang, Yiqian and Huang, Yong and Zhang, Zefeng and Gu, Zheyuan and
            Wang, Shuohuan},
  journal = {arXiv preprint arXiv:2611.26107},
  year = {2026},
  eprint = {2611.26107},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2611.26107}
}

HiUni is initialized from Qwen3-VL. Please also cite the Qwen3-VL project when using this model.

Acknowledgements

We thank the Qwen3-VL, Megatron-Bridge, Megatron-LM, NeMo, and FlashMask teams.

Downloads last month
-
Safetensors
Model size
5B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for ernie-research/HiUni

Finetuned
(573)
this model

Space using ernie-research/HiUni 1

Paper for ernie-research/HiUni