How to use from the
Use from the
Diffusers library
pip install -U diffusers transformers accelerate
import torch
from diffusers import DiffusionPipeline

# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("SUSSYEATING/GLM-Image-ConvRot", dtype=torch.bfloat16, device_map="cuda")

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]

GLM-Image ConvRot (INT8, INT6 w6w8, INT4 w4w8)

High-fidelity ConvRot quantized checkpoints for zai-org/GLM-Image (hybrid 9B Autoregressive VLM + 7B Diffusion Decoder DiT), powered by comfy-kitchen INT8 / W6A8 / W4A8 Tensor Core kernels.

  • 7B DiT (transformer): All 180 attention/FFN block linears quantized with ConvRot-256 (int8_tensorwise, w6a8_int8, asym_w4a8_int8) + 30 AdaLN modulation layers (norm1.linear) kept at INT8 ConvRot-256 across all tiers; prior_token_embedding, glyph_projector, image_projector, time_condition_embed, norm_out, and proj_out remain BF16.
  • 9B AR VLM (vision_language_encoder): All 40 AR LLM layers quantized with SmoothQuant + ConvRot-256 (q_proj, o_proj, mlp.gate_up_proj) and INT8 ConvRot-64 on mlp.down_proj (K = 13696, divisible by 64) for w6w8 / w4w8 compatibility; 40 ViT blocks quantized to INT8 ConvRot-256; k_proj, v_proj (0.17 GB total), embed_tokens, lm_head, and vqmodel remain BF16 to preserve KV-cache precision and discrete visual-token accuracy.
  • Typography & VAE (text_encoder ByT5 + vae): Loaded unquantized directly from zai-org/GLM-Image (1.68 GB combined) for crisp text rendering and pixel fidelity.

Verified Benchmarks (1024x1024, 50 DiT steps, guidance_scale=1.5)

Tier DiT + VLM Size Peak VRAM DiT Warm Time DiT-Only PSNR (cabin / poster / watch) DiT-Only SSIM VLM Teacher-Forced Top-1 / KL
BF16 Baseline 34.08 GB (13.85 + 20.23) 40.55 GB 21.60s Ref 1.0000 100.0% / 0.000
INT8 ConvRot 18.10 GB (7.07 + 11.03, -46.9%) 24.59 GB 17.21s (1.26x) 47.21 / 42.83 / 38.65 dB 0.9982 / 0.9965 / 0.9957 89.71% / 0.081
INT6 (w6w8) ConvRot 15.88 GB (5.94 + 9.94, -53.4%) 22.31 GB 17.95s (1.20x) 41.74 / 39.73 / 36.68 dB 0.9966 / 0.9941 / 0.9933 89.19% / 0.082
INT4 (w4w8) ConvRot 12.91 GB (4.43 + 8.48, -62.1%) 19.40 GB 17.80s (1.21x) 30.44 / 30.55 / 26.37 dB 0.9723 / 0.9650 / 0.9608 79.62% / 0.316

Note: In end-to-end sampling, GLM-Image's AR VLM autoregressively samples 4,096 discrete visual tokens, so small logit shifts produce different valid compositions of the same prompt. Both DiT-isolated reconstruction (identical BF16 prior tokens -> quantized DiT) and VLM teacher-forced visual-token agreement are reported above alongside full end-to-end visual comparisons below.


Visual Comparisons (BF16 | INT8 | INT6 w6w8 | INT4 w4w8)

End-to-End Photorealistic Scene (t2i_cabin_s42)

t2i_cabin_s42

End-to-End Typography Poster (t2i_poster_s1337)

t2i_poster_s1337

End-to-End Macro Detail (t2i_watch_s777)

t2i_watch_s777

DiT-Isolated Reconstruction Fidelity (t2i_cabin_s42, Same BF16 Prior Tokens)

t2i_cabin_s42_dit


Quickstart (diffusers + comfy-kitchen)

pip install comfy-kitchen>=0.2.41 diffusers transformers safetensors accelerate
import torch
from huggingface_hub import hf_hub_download, snapshot_download
from glm_image_convrot import load_glm_image_convrot

base_model = snapshot_download("zai-org/GLM-Image")
tier = "int6"  # "int8", "int6", or "int4"

transformer_path = hf_hub_download(
    "SUSSYEATING/GLM-Image-ConvRot",
    f"{tier}/glm-image-transformer-{tier}-convrot.safetensors",
)
vlm_path = hf_hub_download(
    "SUSSYEATING/GLM-Image-ConvRot",
    f"{tier}/glm-image-vlm-{tier}-convrot.safetensors",
)

pipe, info = load_glm_image_convrot(
    base_model=base_model,
    transformer_path=transformer_path,
    vlm_path=vlm_path,
    device="cuda",
    dtype=torch.bfloat16,
)

image = pipe(
    prompt="A vintage travel poster for 'ALPS EXPRESS' with bold cream typography, a red steam train crossing a stone viaduct over snowy mountains, 'SWITZERLAND 1964'",
    height=1024,
    width=1024,
    num_inference_steps=50,
    guidance_scale=1.5,
    generator=torch.Generator(device="cuda").manual_seed(1337),
).images[0]
image.save("glm_convrot_out.png")

License & Attribution

  • Derived from zai-org/GLM-Image, released under the MIT License.
  • As noted upstream, the incorporated VQ tokenizer and ViT weights from X-Omni/X-Omni-En remain subject to the Apache License, Version 2.0.
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for SUSSYEATING/GLM-Image-ConvRot

Finetuned
(13)
this model