ZYI-Opus (256×256 text-to-image, trained from scratch for $15)

ZYI-Opus is a 282M-parameter text-conditioned Diffusion Transformer. The backbone was initialised randomly and trained with a rectified-flow objective, on a total budget of US$ 15 of Modal compute (GPU + CPU + RAM, every stage included).

samples

The 20 fixed evaluation prompts: one image per prompt, seed 42, no cherry-picking (CFG 5, Euler 50 steps).

Components

part model trained here?
VAE mit-han-lab/dc-ae-f32c32-sana-1.1-diffusers (256² → 32×8×8 latent) no, frozen
text encoder openai/clip-vit-large-patch14 (77 hidden states + pooled projection) no, frozen
backbone ZYIDiT-medium: 16 blocks, width 1024, 16 heads, SwiGLU, QK-RMSNorm, adaLN-single, cross-attention to CLIP yes, from random init

Training

  • Data: 889,296 image-caption pairs from 24 of the 47 data_512_2M shards of jackyhate/text-to-image-2M (MIT):

    • real photos re-captioned by Qwen2-VL;
    • DALL·E 3 images;
    • FLUX-dev images.

    Text-, document- and pixel-art-heavy captions were filtered out. Duplicates were removed by caption and by image hash. Images are centre-cropped to 256².

  • Objective: rectified flow, v-prediction (v = noise − x0), timesteps from a logit-normal(0, 1). 10% caption dropout enables CFG; 30% of captions are cut to their first sentence.

  • Optimisation:

    • AdamW, lr 3e-4, betas (0.9, 0.95), weight decay 0.01;
    • global batch 1024; bf16 autocast; torch.compile; EMA 0.9995;
    • warmup-stable-decay LR schedule, linear decay over the last 30% of the time budget.
  • Hardware: 1× NVIDIA H100 80GB on Modal, 81 min of training, 11,688 optimizer steps (≈ 12.0 M images seen, ≈ 13.5 epochs).

  • Cost: ≈ US$ 13.06 (billed by Modal: $4.87 preprocessing, $7.92 training incl. benchmark/smoke test, $0.07 sampling, rest tests/publishing) in total. The per-stage cost log is in the GitHub repo.

loss

Usage

import torch
from safetensors.torch import load_file
from diffusers import AutoencoderDC
import sys; sys.path.insert(0, "code")   # model.py / sampler.py from the code/ folder of this repo
from model import ZYIDiT
from sampler import TextEncoder, sample, decode

cfg = {"dim": 1024, "depth": 16, "heads": 16}
model = ZYIDiT(**cfg).cuda().eval()
model.load_state_dict(load_file("zyi_opus_ema.safetensors"))
stats = load_file("latent_stats.safetensors")
m, s = stats["lat_mean"].cuda(), stats["lat_std"].cuda()
te = TextEncoder("openai/clip-vit-large-patch14")
vae = AutoencoderDC.from_pretrained("mit-han-lab/dc-ae-f32c32-sana-1.1-diffusers", torch_dtype=torch.bfloat16).cuda()
z = sample(model, te, ["a red apple on a wooden table"], m, s, steps=50, cfg=5.0, shift=1.0, seed=42)
img = decode(vae, z, m, s)[0]   # uint8 HxWx3

Limitations

  • Trained with roughly 1/1000 of the compute of production models. Objects and scenes are recognisable and colours and composition follow the prompt, but fine detail (faces, hands, text, small objects) is often malformed.
  • Only 256×256, square. The 77-token CLIP text encoder limits compositional understanding: counting, spatial relations and attribute binding are weak.
  • The training data mixes real photos with synthetic DALL·E 3 / FLUX images, and inherits their biases and styles. The data and the model are for research and non-commercial use.

Code: GitHub Zylun-AI/zyi-opus.

Downloads last month
44
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train caikybaldo999/zyi-opus