File size: 4,280 Bytes
22a9e53 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 | ---
license: mit
base_model: zai-org/GLM-Image
tags:
- core-ai
- coreai
- text-to-image
- autoregressive
- diffusion
- glm
- on-device
- apple-silicon
pipeline_tag: text-to-image
library_name: coreai
---
# GLM-Image — Core AI
[ZhipuAI's **GLM-Image**](https://huggingface.co/zai-org/GLM-Image) (16B, MIT) converted to
**Core AI** for on-device image generation on Apple Silicon (macOS 27+) — the zoo's first
**autoregressive + diffusion hybrid**.
GLM-Image generates in two stages: a **9B GLM-4 autoregressive model** writes the image as a
grid of discrete *visual prior tokens* (sampled left-to-right like an LLM, ~36 tok/s on
M4 Max), and a **7B flow-matching DiT** denoises the actual pixels conditioned on those
tokens, followed by a 16-channel VAE decode. Both native **1024×1024** and a faster
**512×512** variant are included.
> **macOS only.** The AR (9.6 GB) + DiT (9.2 GB) weights are resolution-independent, so the
> 512 variant is no smaller — an iPhone port needs int4 + sequential component loading.
## Components
| Component | Description | Size |
| --- | --- | --- |
| `glm_image_ar.aimodelc` | GLM-4-9B visual-token AR decoder (int8, S=1 decode, AOT h16c GPU) | 9.6 GB |
| `glm_image_dit_1024.aimodelc` | 30-block flow-matching DiT, 1024² (int8, AOT h16c GPU) | 9.2 GB |
| `glm_image_dit_512.aimodelc` | Same DiT exported at 512² | 9.2 GB |
| `glm_image_vae_1024.aimodel` | 16ch AutoencoderKL decoder, 1024² (fp32, CPU) | 0.9 GB |
| `glm_image_vae_512.aimodel` | Same VAE exported at 512² | 0.9 GB |
| `tokenizer/` | GLM tokenizer (vocab 168 064; visual tokens < 16 512) | — |
| `ehs.f32` | Empty-glyph text embedding [1×1×1472] — the DiT text input for glyph-free prompts (no T5 needed at runtime) | 6 KB |
Weights are int8 (per-block-32) for AR/DiT; the VAE stays fp32 (fp16 overflows its
activations). Compute precision float16 on GPU. One resolution needs ~19.7 GB on disk
(AR + one DiT + one VAE).
## Usage
### Sample app (easiest)
[**CoreAIImageGen** (macOS)](https://github.com/john-rocky/coreai-model-zoo/tree/main/apps/CoreAIImageGen)
— run the `CoreAIImageGenMac` scheme, pick **GLM-Image 1024 (AR+diffusion)** from the model
menu, tap **Download & Load**, type a prompt, **Generate**.
- **Steps 20** (default) ≈ 4.5 min/image on M4 Max — reference quality
- **Steps 12** ≈ 2.5 min — nearly indistinguishable
- The 512 variant generates in ~1 min
The AR stage is *sampled* (temperature 0.9 / top-p 0.75, the upstream `generation_config`),
so re-rolling the seed meaningfully changes composition.
### Pipeline contract (build your own)
The hybrid does not fit Apple's high-level `CoreAIDiffusionPipeline`; the host loop is
~300 lines of Swift (see the app's `GlmImagePipeline`):
1. **AR**: `tokenize(prompt) + fixed grid suffix` → S=1 prefill/decode with host-computed
3D-mRoPE cos/sin → sample visual tokens → 2× upsample the large grid → `prior[1, N]`
(N = 1024 @512, 4096 @1024).
2. **DiT**: 20-step CFG (guidance 1.5, `prior_scale` 1/0 for cond/uncond). Condition the
DiT on the **raw integer schedule** `trunc(linspace(1000,1,steps+1)) − 1`; step the
latent with the **mu-shifted sigmas** `σ' = μ/(μ + (1/σ − 1))`, μ = 0.75·side/256 + 0.25.
3. **VAE** (fp32, CPU): latent → RGB.
⚠️ Feeding the DiT the *shifted* timesteps (sigma×1000) instead of the raw schedule skews
the adaLN time conditioning every step and drifts colors — resolution-dependently (mild
at 512, strong at 1024). This is the one non-obvious contract in the port.
## Limitations
- **Glyph rendering (text in images) is not wired**: prompts containing quoted text
(`'…'`, `"…"`, `「…」`) route through a T5 glyph encoder that is not part of this bundle.
Keep prompts glyph-free.
- Negative prompts are a no-op (GLM's CFG drops the prior tokens instead).
- Image-to-image / editing (`GlmImageKVCache`) not yet exported.
## Quality
Verified against the bf16 diffusers reference: DiT single-forward cosine 0.9999, VAE
byte-exact, AR prior-injection reproduces the reference composition; end-to-end output is
visually on par with `GlmImagePipeline` at the same step count.
## License
MIT (inherited from [zai-org/GLM-Image](https://huggingface.co/zai-org/GLM-Image)).
|