How to use from the
Use from the
Diffusers library
pip install -U diffusers transformers accelerate
import torch
from diffusers import DiffusionPipeline

# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("intel-ai/Qwen-Image-2.1-MXFP8DiT-AutoRound", dtype=torch.bfloat16, device_map="cuda")

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]

Model Details

This model is an MXFP8 (W8A8, group_size 32, symmetric) conversion of Qwen/Qwen-Image-2.1. Only the image transformer is quantized; the Qwen3-VL text encoder and VAE stay in BF16.

  • Quantized (MXFP8): 32 DiT blocks × attn.to_q/to_k/to_v/to_out.0 and img_mlp.proj/gate_layer/out (224 Linear layers).
  • Kept in BF16: Qwen3-VL text encoder, VAE, DiT entry/exit layers (img_in, txt_in, time_text_embed, modulation, norm_out, proj_out) and all norms.
  • Checkpoint scope: transformer 7.0 GB (from 14.2 GB); text encoder/VAE unchanged.

How to Use

Environment

uv pip install "diffusers==0.42.1" "transformers==5.19.0" "auto-round==0.17.0" accelerate

Diffusers Usage

import torch
from diffusers import DiffusionPipeline, PipelineQuantizationConfig, AutoRoundConfig

pipe = DiffusionPipeline.from_pretrained(
    "<org>/Qwen-Image-2.1-MXFP8-RTN-AutoRound",
    quantization_config=PipelineQuantizationConfig(
        quant_mapping={
            "transformer": AutoRoundConfig(
                bits=8, group_size=32, data_type="mx_fp", backend="auto"
            )
        }
    ),
    dtype=torch.bfloat16,
    device_map="cuda",
)

image = pipe(
    prompt='A neon shop sign that reads "QWEN IMAGE 2.1", rainy night, reflections on wet pavement',
    width=2048, height=2048,
    num_inference_steps=40,
    generator=torch.Generator("cuda").manual_seed(42),
).images[0]
image.save("t2i_example.png")

vLLM-Omni Note

vLLM-Omni's MXFP8 has two kernel paths. The Marlin kernel is W8A16 (input_quant_key=None) and silently ignores the checkpoint's activation quantization; with it the RGBA result collapses (CLIP 22.1). The FlashInfer kernel is W8A8 (kMxfp8Dynamic) and recovers RGBA 34.7. Enable FlashInfer with matching flashinfer-python / flashinfer-cubin versions and CUDA_HOME / nvcc / ninja, or use diffusers. Check that layer.quant_method.scheme.kernel is a FlashInfer*Mxfp8LinearKernel, not MarlinMxfp8LinearKernel.

Evaluation

Run note: this directory's gate.json is the raw Marlin W8A16 run (NO-GO, RGBA 22.1). The authoritative diffusers W8A8 evaluation is results/qwen-image-2.1-mxfp8-diffusers/ (GO, T2I/RGBA 45.38/34.37, average 39.88). FlashInfer W8A8 also recovers RGBA to 34.7.

Official model-card examples, generated by diffusers at 2048×2048, 40 steps, seed 42, true_cfg_scale=1.0 (no CFG). CLIP score is image-text cosine ×100 (higher is better).

Example BF16 MXFP8 Recovery
Text-to-Image (neon sign) 46.08 45.39 98.51%
RGBA (transparent dragon sticker) 33.65 34.37 102.14%
Average 39.86 39.88 100.03%

Gate: GO (average drop 0.01).

Pixel/alpha comparison (same seed and scheduler):

  • T2I: PSNR 17.96 dB, SSIM 0.778. Sampling is stochastic, so pixels differ; CLIP is within 0.7 of BF16.
  • RGBA: alpha MAE 0.043, alpha IoU@0.5 0.851; transparency structure is preserved.
BF16 MXFP8
t2i bf16 t2i mxfp8
rgba bf16 rgba mxfp8

Generate the Model

auto-round \
    --model_name Qwen/Qwen-Image-2.1 \
    --scheme MXFP8 --iters 0 --format auto_round \
    --output_dir Qwen-Image-2.1-MXFP8-RTN-AutoRound \
    --disable_opt_rtn --disable_model_free \
    --dataset coco2014 --num_inference_steps 40 --guidance_scale 1.0

Evaluation (examples/run_qwen_image_2.1_mxfp8_diffusers.yaml):

aqa run examples/run_qwen_image_2.1_mxfp8_diffusers.yaml

Ethical Considerations and Limitations

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs. Therefore, before deploying any applications of this model, developers should perform safety testing.

Caveats and Recommendations

  • vLLM-Omni MXFP8 scheme: the default Marlin fallback is W8A16 and makes RGBA much worse for this W8A8 checkpoint. Enable the FlashInfer W8A8 kernel (matching flashinfer-cubin) or use diffusers.
  • RTN (iters=0) was used; no calibration or tuning.
  • MXFP8 inference is experimental in auto-round/diffusers; speed is not necessarily better than BF16.
  • Diffusers AutoRoundConfig must be given explicit bits=8, group_size=32, data_type="mx_fp"; the defaults are W4G128.

Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.

Cite

@article{cheng2023optimize,
  title={Optimize weight rounding via signed gradient descent for the quantization of llms},
  author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
  journal={arXiv preprint arXiv:2309.05516},
  year={2023}
}

arxiv github

Signed-off-by: {member name} <{member email}>

Downloads last month
44
Safetensors
Model size
7B params
Tensor type
BF16
·
F8_E4M3
·
U8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for intel-ai/Qwen-Image-2.1-MXFP8DiT-AutoRound

Finetuned
(100)
this model

Paper for intel-ai/Qwen-Image-2.1-MXFP8DiT-AutoRound