Instructions to use AiArtLab/ra with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use AiArtLab/ra with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("AiArtLab/ra", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Ra: card
Browse files
README.md
ADDED
|
@@ -0,0 +1,203 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
pipeline_tag: text-to-image
|
| 4 |
+
tags:
|
| 5 |
+
- text-to-image
|
| 6 |
+
- flow-matching
|
| 7 |
+
- diffusion-transformer
|
| 8 |
+
- pyramidal-unet
|
| 9 |
+
- 256x256
|
| 10 |
+
- ettin-150m
|
| 11 |
+
- from-scratch
|
| 12 |
+
- ra
|
| 13 |
+
- udit
|
| 14 |
+
datasets:
|
| 15 |
+
- LucasFang/FLUX-Reason-6M
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
# Ra
|
| 19 |
+
|
| 20 |
+
**Ra** is a **256×256 text-to-image** model trained from scratch: **177.3M parameters** in three
|
| 21 |
+
parts — a pyramidal U-transformer (143.3M) that flows in the latent of an f8/32-channel AE, a frozen
|
| 22 |
+
Ettin-150M text encoder with a learnable layer-slice mixer (10.0M), and the AE decoder retrained as a
|
| 23 |
+
pixel-space flow head (23.9M). Named after the Egyptian sun god: the body is a pyramid of widths —
|
| 24 |
+
192 channels where the grid is dense, 1536 where only 16 tokens remain.
|
| 25 |
+
|
| 26 |
+
| | |
|
| 27 |
+
|---|---|
|
| 28 |
+
| output | 256×256 native (sides must be multiples of 64; other sizes work on the same weights) |
|
| 29 |
+
| sampler | body: 8–16 Euler steps, x0 parametrisation, cfg 1.5–2.5; head: 4–8 steps |
|
| 30 |
+
| speed | 0.2 s (8 steps) – 0.4 s (16 steps) per image on one RTX 4080, measured while a training run shared the card |
|
| 31 |
+
| generator | 143.3M, widths 192 → 512 → 1024 → 1536 at ÷1, ÷2, ÷4, ÷8 of the latent grid |
|
| 32 |
+
| text encoder | Ettin-150M (ModernBERT, 768-dim), **frozen**, 192 tokens padded and masked; its unused MLM head is stripped from the bundle (149M) |
|
| 33 |
+
| text adapter | 7 layer slices (2, 5, 8, 11, 14, 17, 20) → LayerNorm → 2 attention blocks along the slice axis → `Linear(7→1)`, 10.04M trainable |
|
| 34 |
+
| latent | 8× / 32 channels from [`Muinez/f8c32-ae`](https://huggingface.co/Muinez/f8c32-ae) (encoder 68.4M, decoder 23.9M) — 32×32 at 256 px |
|
| 35 |
+
| training | from scratch, 256², MSE, AdamW 8-bit, lr 1e-4 cosine, one RTX 4080 |
|
| 36 |
+
| weights | 177.3M stored as **fp16** (338 MiB); the fp32 master stays for training, the whole bundle is 0.8 GiB |
|
| 37 |
+
| licence | MIT (our code and weights), see *Licences* |
|
| 38 |
+
|
| 39 |
+
**Pyramid UDiT — разбор на русском:** [docs/presentation.pdf](docs/presentation.pdf) (12 слайдов:
|
| 40 |
+
почему ширина растёт к низу, как считается стоимость уровней, зачем текст на всех уровнях,
|
| 41 |
+
условие хвоста, замеры на бабочках и на FLUX-Reason).
|
| 42 |
+
|
| 43 |
+
## How it works
|
| 44 |
+
|
| 45 |
+
```
|
| 46 |
+
text: Ettin-150M (768, 22 layers, frozen)
|
| 47 |
+
└─ SliceMixer: 7 slices → attention along the slice axis → Linear(7→1) 10.04M
|
| 48 |
+
|
| 49 |
+
noise (32, h, w) ─► BODY, pyramid 192 → 512 → 1024 → 1536 143.3M
|
| 50 |
+
L0 ÷1 conv + text 6 enc / 4 dec 1024 tokens at 256²
|
| 51 |
+
L1 ÷2 transformer 4 / 3 256
|
| 52 |
+
L2 ÷4 transformer 3 / 2 64
|
| 53 |
+
L3 ÷8 transformer 2 16 ← wide bottleneck, full attention
|
| 54 |
+
additive skips, down = mean-pool 2× + Linear, up = Linear + nearest 2×
|
| 55 |
+
─► latent z (32, h, w) ─► HEAD: AE decoder (6 blocks @512, PixelShuffle ×3) 23.87M
|
| 56 |
+
pixel flow conditioned on z0 + (body error).detach()
|
| 57 |
+
─► image
|
| 58 |
+
```
|
| 59 |
+
|
| 60 |
+
- **Width grows where tokens get scarce.** A layer costs ~`tokens × C²`, so the 256² grid stays narrow
|
| 61 |
+
conv while the 16-token bottleneck carries 1536 channels and full attention — global constraints
|
| 62 |
+
(symmetry, composition) are decided there.
|
| 63 |
+
- **Text reaches every level** through cross-attention; a pooled text vector joins `t` in the
|
| 64 |
+
shift/scale/gate modulation of each level.
|
| 65 |
+
- **The head is a separate flow, not a decoder pass.** It is conditioned on the body's own error
|
| 66 |
+
(`z0 + (body error).detach()`), so it renders the real mistake instead of learning to fix garbage.
|
| 67 |
+
- Body predicts **x0**, RoPE phase is in latent pixels on every level, block outputs are zero-init.
|
| 68 |
+
|
| 69 |
+
## Training
|
| 70 |
+
|
| 71 |
+
| stage | data | steps | batch | loss |
|
| 72 |
+
|---|---|---|---|---|
|
| 73 |
+
| 1 | 970 butterflies, 256² (`~/dataset/butterflies_256x64`) | 20 003 | 8 | `mse_lat` 0.107 |
|
| 74 |
+
| 2 | `LucasFang/FLUX-Reason-6M` Aesthetics-Part01 (~1.5M images) | 187 500 | 16 | `mse_lat` 0.236 (ema), low t 0.025, high t 0.56, `mse_pix` 0.018 |
|
| 75 |
+
| 3 | Parts 01 + 02 + Imaginative + Text (5.9M images) | 212 000, still running | 16 | `mse_lat` 0.245 |
|
| 76 |
+
|
| 77 |
+
Stage 2 ran 13.7 h at 241 ms/step. Stage 1 ended with high-frequency energy at **0.54** of the ground
|
| 78 |
+
truth against the AE ceiling of 0.859 (0.913 for GT) — that is, 63% of what the VAE itself preserves
|
| 79 |
+
at 256², and 3× the value of the flat 1024-wide stack this architecture replaced. Stage 1 samples are
|
| 80 |
+
kept in `docs/img/samples_butterflies_20k.png`.
|
| 81 |
+
|
| 82 |
+
The bundled `transformer/` is the **stage-3 snapshot at step 212 000**. Stage 3 restarted the lr
|
| 83 |
+
schedule at 1e-4 on step 192 000 (after stage 2 had cosine-decayed to 1e-5), so this snapshot is
|
| 84 |
+
mid-recovery: the stage-2 endpoint (step 187 500, still on disk as `tmp/flux/ckpt/step_0187500.pt`)
|
| 85 |
+
renders noticeably sharper. Expect the samples below to improve as the schedule comes down again.
|
| 86 |
+
|
| 87 |
+
## Examples
|
| 88 |
+
|
| 89 |
+
`transformer/` (step 212 000), 256 px, 16 Euler steps, 8 head steps, cfg 2.5, seed 0 for
|
| 90 |
+
every tile — drawn from the fp16 bundle that ships in this repository. Top row general prompts,
|
| 91 |
+
bottom row the butterfly domain it started from; prompts 1–8 in reading order:
|
| 92 |
+
|
| 93 |
+
1. A photograph of a red fox standing in fresh snow among bare birch trees, soft overcast light.
|
| 94 |
+
2. An oil painting of a wooden sailboat on a calm turquoise sea at sunset, warm golden light.
|
| 95 |
+
3. A portrait of an elderly fisherman with a weathered face and grey beard, window light, shallow depth of field.
|
| 96 |
+
4. A wide landscape of sunflowers in a field under a dramatic stormy sky, distant hills on the horizon.
|
| 97 |
+
5. A macro photograph of a blue morpho butterfly resting on a purple orchid, dew on the petals.
|
| 98 |
+
6. A close-up photograph of a white butterfly with orange wing tips on a green leaf, dorsal view.
|
| 99 |
+
7. A dark brown swallowtail butterfly with pale yellow spots, wings spread, on a white background.
|
| 100 |
+
8. A photograph of an orange and black monarch butterfly on a pink flower, soft natural light.
|
| 101 |
+
|
| 102 |
+
<img src="docs/img/samples.png" width="100%"/>
|
| 103 |
+
|
| 104 |
+
## Limitations
|
| 105 |
+
|
| 106 |
+
- **Anatomy is the weak spot.** Animals come out as plausible blobs — see tile 1: composition, snow and
|
| 107 |
+
light are right, the fox is not. Faces hold up at this size, hands do not.- **256² only for now.** Detail finer than 8 px is invented by the AE decoder, not decided by the body
|
| 108 |
+
(the latent is 8× downsampled); 512² is untested on this architecture.
|
| 109 |
+
- **Caption adherence is loose.** On a 5.9M-image stream MSE makes the caption worth a few percent of
|
| 110 |
+
the loss: subject and style arrive, counts and small details do not.
|
| 111 |
+
- **No glyphs.** Letters appear in signs and are nonsense; the objective never rewards glyph identity.
|
| 112 |
+
- Sides must be multiples of **64** (three halvings), else the down/up path is not integral.
|
| 113 |
+
- **Research artifact, no safety tuning** — no NSFW filter, no watermark, no provenance metadata.
|
| 114 |
+
|
| 115 |
+
## Files
|
| 116 |
+
|
| 117 |
+
```
|
| 118 |
+
src/model.py UDiT: pyramid body + pixel head + sample()
|
| 119 |
+
src/text.py Ettin + SliceMixer
|
| 120 |
+
src/train.py trainer (folder of pairs or a HF parquet stream)
|
| 121 |
+
src/export.py export to transformer/ (safetensors + config.json)
|
| 122 |
+
src/make_dataset*.py dataset builders (buckets multiple of 64)
|
| 123 |
+
src/sample.py CLI: prompt → PNG from transformer/ or from a training .pt
|
| 124 |
+
src/publish.py build the fp16 bundle (tmp/publish) and push it to the Hub
|
| 125 |
+
transformer/ checkpoint (step 212 000) + config.json; the checkpoint describes its own architecture
|
| 126 |
+
vae/ f8c32-ae: encoder 68.4M + decoder 23.9M
|
| 127 |
+
text_encoder/ Ettin-150M
|
| 128 |
+
docs/ARCH.md architecture, decisions and the measurements behind them
|
| 129 |
+
docs/STATUS.md what works, what is next, known limits
|
| 130 |
+
docs/README.ru.md the same repo notes in Russian (development log)
|
| 131 |
+
docs/img/ sample grids
|
| 132 |
+
docs/presentation.pdf slide-by-slide walkthrough of the pyramid (Russian)
|
| 133 |
+
tmp/ runs (in .gitignore)
|
| 134 |
+
```
|
| 135 |
+
|
| 136 |
+
## Usage
|
| 137 |
+
|
| 138 |
+
Inference — one image from the bundled checkpoint (all flags below are defaults):
|
| 139 |
+
|
| 140 |
+
```bash
|
| 141 |
+
python3 src/sample.py --prompt "A photograph of a red fox in fresh snow." --out fox.png
|
| 142 |
+
# 256 px, 16 body steps, 8 head steps, cfg 2.5, seed 0; --n 4 gives four seeds,
|
| 143 |
+
# --model tmp/run/ckpt/step_0212000.pt samples a training checkpoint directly
|
| 144 |
+
```
|
| 145 |
+
|
| 146 |
+
The same from Python:
|
| 147 |
+
|
| 148 |
+
```python
|
| 149 |
+
import sys; sys.path[:0] = ["src", "vae"]
|
| 150 |
+
from model import DEV, UDiT
|
| 151 |
+
from text import EttinText
|
| 152 |
+
from ae.model import AutoEncoder
|
| 153 |
+
from export import load_exported
|
| 154 |
+
|
| 155 |
+
ae = AutoEncoder.from_pretrained("vae", device=DEV)
|
| 156 |
+
enc = EttinText("text_encoder", DEV, max_len=192)
|
| 157 |
+
m = UDiT(ae, text_dim=enc.dim).to(DEV)
|
| 158 |
+
load_exported(m, enc, ae, "transformer", DEV); m.eval()
|
| 159 |
+
|
| 160 |
+
null_hs, null_mk = enc.slices([""], pad_to_len=192)
|
| 161 |
+
null_t = enc.mix(null_hs.float())
|
| 162 |
+
hs, mk = enc.slices(["A photograph of a red fox in fresh snow."], pad_to_len=192)
|
| 163 |
+
y = m.sample(enc.mix(hs.float()), mk, 32, 32, steps=16, tail_steps=8, cfg=2.5,
|
| 164 |
+
null_text=null_t, null_mask=null_mk) # (1, 3, 256, 256) in [-1, 1]
|
| 165 |
+
```
|
| 166 |
+
|
| 167 |
+
## Reproduce
|
| 168 |
+
|
| 169 |
+
The run behind the bundled checkpoint — one RTX 4080, batch 16 at 256², ~240 ms/step, 13 h per
|
| 170 |
+
187 500-step epoch:
|
| 171 |
+
|
| 172 |
+
```bash
|
| 173 |
+
python3 src/train.py --batch 16 --init tmp/flux2/ckpt/step_0192000.pt \
|
| 174 |
+
--hf-dataset LucasFang/FLUX-Reason-6M --epochs 1
|
| 175 |
+
# --parts Aesthetics-Part01,Aesthetics-Part02,Imaginative,Text (default)
|
| 176 |
+
# --out tmp/run every 1000 steps a panel to tmp/run/samples/latest.png,
|
| 177 |
+
# every 2000 steps a checkpoint to tmp/run/ckpt/, resume with --resume
|
| 178 |
+
```
|
| 179 |
+
|
| 180 |
+
From scratch on a folder of `<name>.jpg` + `<name>.txt` pairs (bucket sides multiple of 64, centre
|
| 181 |
+
crop):
|
| 182 |
+
|
| 183 |
+
```bash
|
| 184 |
+
python3 src/make_dataset.py --src ~/dataset/butterflies/butterfly \
|
| 185 |
+
--out ~/dataset/butterflies_256x64 --long-side 256 --eval-n 30
|
| 186 |
+
python3 src/train.py --data ~/dataset/butterflies_256x64/train \
|
| 187 |
+
--eval ~/dataset/butterflies_256x64/eval --batch 8 --steps 20000
|
| 188 |
+
```
|
| 189 |
+
|
| 190 |
+
Export to `transformer/` after training: `python3 src/export.py --ckpt tmp/run/last.pt --out transformer`.
|
| 191 |
+
The checkpoint carries its own `args`, so `config.json` describes the run it came from.
|
| 192 |
+
|
| 193 |
+
Publish: `python3 src/publish.py --out tmp/publish --push` — copies the code and docs, converts the
|
| 194 |
+
three weight files to fp16, drops the unused MLM head of the text encoder and uploads 0.8 GiB to this
|
| 195 |
+
repository.
|
| 196 |
+
|
| 197 |
+
## Licences
|
| 198 |
+
|
| 199 |
+
**MIT** for our code and weights. Bundled upstream components keep their own terms:
|
| 200 |
+
[`jhu-clsp/ettin-encoder-150m`](https://huggingface.co/jhu-clsp/ettin-encoder-150m) (MIT) and
|
| 201 |
+
[`Muinez/f8c32-ae`](https://huggingface.co/Muinez/f8c32-ae) (Apache-2.0). Training data:
|
| 202 |
+
`LucasFang/FLUX-Reason-6M` (Apache-2.0 per its dataset card; images generated with FLUX.1-dev);
|
| 203 |
+
the butterfly set is a local 970-image collection.
|