GLM-Image-ConvRot / README.md
SUSSYEATING's picture
Release GLM-Image ConvRot INT8, INT6 & INT4 checkpoints, metrics, and showcases
9ca4e71
|
Raw History Blame Contribute Delete
5.06 kB
---
license: mit
language:
- zh
- en
library_name: diffusers
pipeline_tag: text-to-image
base_model: zai-org/GLM-Image
tags:
- glm-image
- convrot
- quantization
- int8
- int6
- int4
- comfy-kitchen
---
# GLM-Image ConvRot (`INT8`, `INT6 w6w8`, `INT4 w4w8`)
High-fidelity **ConvRot** quantized checkpoints for [**zai-org/GLM-Image**](https://huggingface.co/zai-org/GLM-Image) (hybrid **9B Autoregressive VLM** + **7B Diffusion Decoder DiT**), powered by [`comfy-kitchen`](https://github.com/Comfy-Org/comfy-kitchen) INT8 / W6A8 / W4A8 Tensor Core kernels.
- **7B DiT (`transformer`)**: All `180` attention/FFN block linears quantized with ConvRot-256 (`int8_tensorwise`, `w6a8_int8`, `asym_w4a8_int8`) + `30` AdaLN modulation layers (`norm1.linear`) kept at `INT8 ConvRot-256` across all tiers; `prior_token_embedding`, `glyph_projector`, `image_projector`, `time_condition_embed`, `norm_out`, and `proj_out` remain BF16.
- **9B AR VLM (`vision_language_encoder`)**: All `40` AR LLM layers quantized with SmoothQuant + ConvRot-256 (`q_proj`, `o_proj`, `mlp.gate_up_proj`) and **`INT8 ConvRot-64`** on `mlp.down_proj` (`K = 13696`, divisible by 64) for `w6w8` / `w4w8` compatibility; `40` ViT blocks quantized to `INT8 ConvRot-256`; `k_proj`, `v_proj` (`0.17 GB` total), `embed_tokens`, `lm_head`, and `vqmodel` remain BF16 to preserve KV-cache precision and discrete visual-token accuracy.
- **Typography & VAE (`text_encoder` ByT5 + `vae`)**: Loaded unquantized directly from `zai-org/GLM-Image` (`1.68 GB` combined) for crisp text rendering and pixel fidelity.
---
## Verified Benchmarks (`1024x1024`, 50 DiT steps, `guidance_scale=1.5`)
| Tier | DiT + VLM Size | Peak VRAM | DiT Warm Time | DiT-Only PSNR (`cabin / poster / watch`) | DiT-Only SSIM | VLM Teacher-Forced Top-1 / KL |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **BF16 Baseline** | `34.08 GB` (`13.85 + 20.23`) | `40.55 GB` | `21.60s` | Ref | `1.0000` | `100.0%` / `0.000` |
| **INT8 ConvRot** | **`18.10 GB`** (`7.07 + 11.03`, `-46.9%`) | **`24.59 GB`** | **`17.21s`** (**`1.26x`**) | **`47.21 / 42.83 / 38.65 dB`** | **`0.9982 / 0.9965 / 0.9957`** | **`89.71%`** / `0.081` |
| **INT6 (`w6w8`) ConvRot** | **`15.88 GB`** (`5.94 + 9.94`, `-53.4%`) | **`22.31 GB`** | **`17.95s`** (**`1.20x`**) | **`41.74 / 39.73 / 36.68 dB`** | **`0.9966 / 0.9941 / 0.9933`** | **`89.19%`** / `0.082` |
| **INT4 (`w4w8`) ConvRot** | **`12.91 GB`** (`4.43 + 8.48`, `-62.1%`) | **`19.40 GB`** | **`17.80s`** (**`1.21x`**) | **`30.44 / 30.55 / 26.37 dB`** | **`0.9723 / 0.9650 / 0.9608`** | **`79.62%`** / `0.316` |
> **Note:** In end-to-end sampling, GLM-Image's AR VLM autoregressively samples 4,096 discrete visual tokens, so small logit shifts produce different valid compositions of the same prompt. Both **DiT-isolated** reconstruction (identical BF16 prior tokens -> quantized DiT) and **VLM teacher-forced** visual-token agreement are reported above alongside full end-to-end visual comparisons below.
---
## Visual Comparisons (`BF16 | INT8 | INT6 w6w8 | INT4 w4w8`)
### End-to-End Photorealistic Scene (`t2i_cabin_s42`)
![t2i_cabin_s42](comparisons/t2i_cabin_s42_4way.png)
### End-to-End Typography Poster (`t2i_poster_s1337`)
![t2i_poster_s1337](comparisons/t2i_poster_s1337_4way.png)
### End-to-End Macro Detail (`t2i_watch_s777`)
![t2i_watch_s777](comparisons/t2i_watch_s777_4way.png)
### DiT-Isolated Reconstruction Fidelity (`t2i_cabin_s42`, Same BF16 Prior Tokens)
![t2i_cabin_s42_dit](comparisons/t2i_cabin_s42_dit_4way.png)
---
## Quickstart (`diffusers` + `comfy-kitchen`)
```bash
pip install comfy-kitchen>=0.2.41 diffusers transformers safetensors accelerate
```
```python
import torch
from huggingface_hub import hf_hub_download, snapshot_download
from glm_image_convrot import load_glm_image_convrot
base_model = snapshot_download("zai-org/GLM-Image")
tier = "int6" # "int8", "int6", or "int4"
transformer_path = hf_hub_download(
"SUSSYEATING/GLM-Image-ConvRot",
f"{tier}/glm-image-transformer-{tier}-convrot.safetensors",
)
vlm_path = hf_hub_download(
"SUSSYEATING/GLM-Image-ConvRot",
f"{tier}/glm-image-vlm-{tier}-convrot.safetensors",
)
pipe, info = load_glm_image_convrot(
base_model=base_model,
transformer_path=transformer_path,
vlm_path=vlm_path,
device="cuda",
dtype=torch.bfloat16,
)
image = pipe(
prompt="A vintage travel poster for 'ALPS EXPRESS' with bold cream typography, a red steam train crossing a stone viaduct over snowy mountains, 'SWITZERLAND 1964'",
height=1024,
width=1024,
num_inference_steps=50,
guidance_scale=1.5,
generator=torch.Generator(device="cuda").manual_seed(1337),
).images[0]
image.save("glm_convrot_out.png")
```
---
## License & Attribution
- Derived from [**zai-org/GLM-Image**](https://huggingface.co/zai-org/GLM-Image), released under the **MIT License**.
- As noted upstream, the incorporated VQ tokenizer and ViT weights from [**X-Omni/X-Omni-En**](https://huggingface.co/X-Omni/X-Omni-En) remain subject to the **Apache License, Version 2.0**.