File size: 2,010 Bytes
8ebf922 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | # Model Architecture
```
prompt ─► Qwen3-VL-8B-Instruct (extract hidden states from layers (0,3,…,33,35) → concat)
│
▼
┌─────────────────────────────────────────────────┐
│ Ideogram4Transformer │
│ • 34 × Ideogram4TransformerBlock │
│ – Ideogram4Attention (QK-RMSNorm, MRoPE) │
│ – Ideogram4MLP (SwiGLU) │
│ – adaln scale/gate from t-embedding │
│ • Ideogram4FinalLayer │
└─────────────────────────────────────────────────┘
│
│ velocity prediction
│
▼
Euler flow-matching sampler with asymmetric CFG
│
│ denoised image latents
│
▼
VAE decode
│
▼
PIL.Image
```
The transformer is a single-stream DiT: text tokens (Qwen3-VL hidden states from
the activation layers) and image latent tokens are concatenated into one
sequence, modulated per-block by an AdaLN computed from the flow-matching
timestep embedding. Attention uses QK-RMSNorm and 3D MRoPE so that text and
image tokens share a unified positional space.
Model spec:
| field | value |
|-------------------|---------------|
| `emb_dim` | 4608 |
| `num_layers` | 34 |
| `num_heads` | 18 |
| `intermediate` | 12288 |
| `adanln_dim` | 512 |
| `rope_theta` | 5_000_000 |
| `mrope_section` | (24, 20, 20) |
| latent channels | 32 × 2² = 128 |
| max text tokens | 2048 |
| sampler | Euler flow-matching, logit-normal schedule, asymmetric CFG |
|