suchiragarwal/lilie

A Language-Informed Loss for
Visual Generation

Suchir Agarwal,  Keshigeyan Chandrasegaran,  Ziyu Chen,  Kyle Sargent, 
Juan Carlos Niebles,  Li Fei-Fei,  Jiajun Wu
Stanford University
📄 arXiv | 🌎 Website | 💻 Code | 🤗 Models

Checkpoints

lilie/                       LILIE encoders (+ the SmolVLM LoRA adapters they were trained with)
├── small/                   32M parameters, used for every generator below
├── tiny/                    9M
└── micro/                   1.5M
jit-t2i/                     JiT-T2I generators
├── xl_pixel-390600/         XL (700M), pixel loss, 390,600 steps
├── xl_lilie-390600/         XL, LILIE loss
├── xl_pixel+lilie-390600/   XL, pixel + LILIE loss
├── g_pixel-390600/          g (1.1B), pixel loss
├── g_lilie-390600/          g, LILIE loss
├── g_pixel+lilie-390600/    g, pixel + LILIE loss
├── g_pixel-781200/          g, pixel loss, continued to 781,200 steps
└── g_pixel+lilie-781200/    g, pixel + LILIE loss, continued to 781,200 steps

Each encoder folder contains encoder.safetensors, vlm_lora.safetensors and meta.json. Each generator folder contains train_weights.safetensors, ema_weights.safetensors and meta.json.

hf download suchiragarwal/lilie --local-dir checkpoints

Training and sampling instructions are in the code repository.