ZYI-Opus (256×256 text-to-image, trained from scratch for $15)
ZYI-Opus is a 282M-parameter text-conditioned Diffusion Transformer. The backbone was initialised randomly and trained with a rectified-flow objective, on a total budget of US$ 15 of Modal compute (GPU + CPU + RAM, every stage included).
The 20 fixed evaluation prompts: one image per prompt, seed 42, no cherry-picking
(CFG 5, Euler 50 steps).
Components
| part | model | trained here? |
|---|---|---|
| VAE | mit-han-lab/dc-ae-f32c32-sana-1.1-diffusers (256² → 32×8×8 latent) |
no, frozen |
| text encoder | openai/clip-vit-large-patch14 (77 hidden states + pooled projection) |
no, frozen |
| backbone | ZYIDiT-medium: 16 blocks, width 1024, 16 heads, SwiGLU, QK-RMSNorm, adaLN-single, cross-attention to CLIP | yes, from random init |
Training
Data: 889,296 image-caption pairs from 24 of the 47
data_512_2Mshards of jackyhate/text-to-image-2M (MIT):- real photos re-captioned by Qwen2-VL;
- DALL·E 3 images;
- FLUX-dev images.
Text-, document- and pixel-art-heavy captions were filtered out. Duplicates were removed by caption and by image hash. Images are centre-cropped to 256².
Objective: rectified flow, v-prediction (
v = noise − x0), timesteps from a logit-normal(0, 1). 10% caption dropout enables CFG; 30% of captions are cut to their first sentence.Optimisation:
- AdamW, lr 3e-4, betas (0.9, 0.95), weight decay 0.01;
- global batch 1024; bf16 autocast;
torch.compile; EMA 0.9995; - warmup-stable-decay LR schedule, linear decay over the last 30% of the time budget.
Hardware: 1× NVIDIA H100 80GB on Modal, 81 min of training, 11,688 optimizer steps (≈ 12.0 M images seen, ≈ 13.5 epochs).
Cost: ≈ US$ 13.06 (billed by Modal: $4.87 preprocessing, $7.92 training incl. benchmark/smoke test, $0.07 sampling, rest tests/publishing) in total. The per-stage cost log is in the GitHub repo.
Usage
import torch
from safetensors.torch import load_file
from diffusers import AutoencoderDC
import sys; sys.path.insert(0, "code") # model.py / sampler.py from the code/ folder of this repo
from model import ZYIDiT
from sampler import TextEncoder, sample, decode
cfg = {"dim": 1024, "depth": 16, "heads": 16}
model = ZYIDiT(**cfg).cuda().eval()
model.load_state_dict(load_file("zyi_opus_ema.safetensors"))
stats = load_file("latent_stats.safetensors")
m, s = stats["lat_mean"].cuda(), stats["lat_std"].cuda()
te = TextEncoder("openai/clip-vit-large-patch14")
vae = AutoencoderDC.from_pretrained("mit-han-lab/dc-ae-f32c32-sana-1.1-diffusers", torch_dtype=torch.bfloat16).cuda()
z = sample(model, te, ["a red apple on a wooden table"], m, s, steps=50, cfg=5.0, shift=1.0, seed=42)
img = decode(vae, z, m, s)[0] # uint8 HxWx3
Limitations
- Trained with roughly 1/1000 of the compute of production models. Objects and scenes are recognisable and colours and composition follow the prompt, but fine detail (faces, hands, text, small objects) is often malformed.
- Only 256×256, square. The 77-token CLIP text encoder limits compositional understanding: counting, spatial relations and attribute binding are weak.
- The training data mixes real photos with synthetic DALL·E 3 / FLUX images, and inherits their biases and styles. The data and the model are for research and non-commercial use.
Code: GitHub Zylun-AI/zyi-opus.
- Downloads last month
- 44

