recoilme commited on
Commit
3dd752f
·
verified ·
1 Parent(s): aa72fdd
Files changed (1) hide show
  1. README.md +203 -0
README.md ADDED
@@ -0,0 +1,203 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ pipeline_tag: text-to-image
4
+ tags:
5
+ - text-to-image
6
+ - flow-matching
7
+ - diffusion-transformer
8
+ - pyramidal-unet
9
+ - 256x256
10
+ - ettin-150m
11
+ - from-scratch
12
+ - ra
13
+ - udit
14
+ datasets:
15
+ - LucasFang/FLUX-Reason-6M
16
+ ---
17
+
18
+ # Ra
19
+
20
+ **Ra** is a **256×256 text-to-image** model trained from scratch: **177.3M parameters** in three
21
+ parts — a pyramidal U-transformer (143.3M) that flows in the latent of an f8/32-channel AE, a frozen
22
+ Ettin-150M text encoder with a learnable layer-slice mixer (10.0M), and the AE decoder retrained as a
23
+ pixel-space flow head (23.9M). Named after the Egyptian sun god: the body is a pyramid of widths —
24
+ 192 channels where the grid is dense, 1536 where only 16 tokens remain.
25
+
26
+ | | |
27
+ |---|---|
28
+ | output | 256×256 native (sides must be multiples of 64; other sizes work on the same weights) |
29
+ | sampler | body: 8–16 Euler steps, x0 parametrisation, cfg 1.5–2.5; head: 4–8 steps |
30
+ | speed | 0.2 s (8 steps) – 0.4 s (16 steps) per image on one RTX 4080, measured while a training run shared the card |
31
+ | generator | 143.3M, widths 192 → 512 → 1024 → 1536 at ÷1, ÷2, ÷4, ÷8 of the latent grid |
32
+ | text encoder | Ettin-150M (ModernBERT, 768-dim), **frozen**, 192 tokens padded and masked; its unused MLM head is stripped from the bundle (149M) |
33
+ | text adapter | 7 layer slices (2, 5, 8, 11, 14, 17, 20) → LayerNorm → 2 attention blocks along the slice axis → `Linear(7→1)`, 10.04M trainable |
34
+ | latent | 8× / 32 channels from [`Muinez/f8c32-ae`](https://huggingface.co/Muinez/f8c32-ae) (encoder 68.4M, decoder 23.9M) — 32×32 at 256 px |
35
+ | training | from scratch, 256², MSE, AdamW 8-bit, lr 1e-4 cosine, one RTX 4080 |
36
+ | weights | 177.3M stored as **fp16** (338 MiB); the fp32 master stays for training, the whole bundle is 0.8 GiB |
37
+ | licence | MIT (our code and weights), see *Licences* |
38
+
39
+ **Pyramid UDiT — разбор на русском:** [docs/presentation.pdf](docs/presentation.pdf) (12 слайдов:
40
+ почему ширина растёт к низу, как считается стоимость уровней, зачем текст на всех уровнях,
41
+ условие хвоста, замеры на бабочках и на FLUX-Reason).
42
+
43
+ ## How it works
44
+
45
+ ```
46
+ text: Ettin-150M (768, 22 layers, frozen)
47
+ └─ SliceMixer: 7 slices → attention along the slice axis → Linear(7→1) 10.04M
48
+
49
+ noise (32, h, w) ─► BODY, pyramid 192 → 512 → 1024 → 1536 143.3M
50
+ L0 ÷1 conv + text 6 enc / 4 dec 1024 tokens at 256²
51
+ L1 ÷2 transformer 4 / 3 256
52
+ L2 ÷4 transformer 3 / 2 64
53
+ L3 ÷8 transformer 2 16 ← wide bottleneck, full attention
54
+ additive skips, down = mean-pool 2× + Linear, up = Linear + nearest 2×
55
+ ─► latent z (32, h, w) ─► HEAD: AE decoder (6 blocks @512, PixelShuffle ×3) 23.87M
56
+ pixel flow conditioned on z0 + (body error).detach()
57
+ ─► image
58
+ ```
59
+
60
+ - **Width grows where tokens get scarce.** A layer costs ~`tokens × C²`, so the 256² grid stays narrow
61
+ conv while the 16-token bottleneck carries 1536 channels and full attention — global constraints
62
+ (symmetry, composition) are decided there.
63
+ - **Text reaches every level** through cross-attention; a pooled text vector joins `t` in the
64
+ shift/scale/gate modulation of each level.
65
+ - **The head is a separate flow, not a decoder pass.** It is conditioned on the body's own error
66
+ (`z0 + (body error).detach()`), so it renders the real mistake instead of learning to fix garbage.
67
+ - Body predicts **x0**, RoPE phase is in latent pixels on every level, block outputs are zero-init.
68
+
69
+ ## Training
70
+
71
+ | stage | data | steps | batch | loss |
72
+ |---|---|---|---|---|
73
+ | 1 | 970 butterflies, 256² (`~/dataset/butterflies_256x64`) | 20 003 | 8 | `mse_lat` 0.107 |
74
+ | 2 | `LucasFang/FLUX-Reason-6M` Aesthetics-Part01 (~1.5M images) | 187 500 | 16 | `mse_lat` 0.236 (ema), low t 0.025, high t 0.56, `mse_pix` 0.018 |
75
+ | 3 | Parts 01 + 02 + Imaginative + Text (5.9M images) | 212 000, still running | 16 | `mse_lat` 0.245 |
76
+
77
+ Stage 2 ran 13.7 h at 241 ms/step. Stage 1 ended with high-frequency energy at **0.54** of the ground
78
+ truth against the AE ceiling of 0.859 (0.913 for GT) — that is, 63% of what the VAE itself preserves
79
+ at 256², and 3× the value of the flat 1024-wide stack this architecture replaced. Stage 1 samples are
80
+ kept in `docs/img/samples_butterflies_20k.png`.
81
+
82
+ The bundled `transformer/` is the **stage-3 snapshot at step 212 000**. Stage 3 restarted the lr
83
+ schedule at 1e-4 on step 192 000 (after stage 2 had cosine-decayed to 1e-5), so this snapshot is
84
+ mid-recovery: the stage-2 endpoint (step 187 500, still on disk as `tmp/flux/ckpt/step_0187500.pt`)
85
+ renders noticeably sharper. Expect the samples below to improve as the schedule comes down again.
86
+
87
+ ## Examples
88
+
89
+ `transformer/` (step 212 000), 256 px, 16 Euler steps, 8 head steps, cfg 2.5, seed 0 for
90
+ every tile — drawn from the fp16 bundle that ships in this repository. Top row general prompts,
91
+ bottom row the butterfly domain it started from; prompts 1–8 in reading order:
92
+
93
+ 1. A photograph of a red fox standing in fresh snow among bare birch trees, soft overcast light.
94
+ 2. An oil painting of a wooden sailboat on a calm turquoise sea at sunset, warm golden light.
95
+ 3. A portrait of an elderly fisherman with a weathered face and grey beard, window light, shallow depth of field.
96
+ 4. A wide landscape of sunflowers in a field under a dramatic stormy sky, distant hills on the horizon.
97
+ 5. A macro photograph of a blue morpho butterfly resting on a purple orchid, dew on the petals.
98
+ 6. A close-up photograph of a white butterfly with orange wing tips on a green leaf, dorsal view.
99
+ 7. A dark brown swallowtail butterfly with pale yellow spots, wings spread, on a white background.
100
+ 8. A photograph of an orange and black monarch butterfly on a pink flower, soft natural light.
101
+
102
+ <img src="docs/img/samples.png" width="100%"/>
103
+
104
+ ## Limitations
105
+
106
+ - **Anatomy is the weak spot.** Animals come out as plausible blobs — see tile 1: composition, snow and
107
+ light are right, the fox is not. Faces hold up at this size, hands do not.- **256² only for now.** Detail finer than 8 px is invented by the AE decoder, not decided by the body
108
+ (the latent is 8× downsampled); 512² is untested on this architecture.
109
+ - **Caption adherence is loose.** On a 5.9M-image stream MSE makes the caption worth a few percent of
110
+ the loss: subject and style arrive, counts and small details do not.
111
+ - **No glyphs.** Letters appear in signs and are nonsense; the objective never rewards glyph identity.
112
+ - Sides must be multiples of **64** (three halvings), else the down/up path is not integral.
113
+ - **Research artifact, no safety tuning** — no NSFW filter, no watermark, no provenance metadata.
114
+
115
+ ## Files
116
+
117
+ ```
118
+ src/model.py UDiT: pyramid body + pixel head + sample()
119
+ src/text.py Ettin + SliceMixer
120
+ src/train.py trainer (folder of pairs or a HF parquet stream)
121
+ src/export.py export to transformer/ (safetensors + config.json)
122
+ src/make_dataset*.py dataset builders (buckets multiple of 64)
123
+ src/sample.py CLI: prompt → PNG from transformer/ or from a training .pt
124
+ src/publish.py build the fp16 bundle (tmp/publish) and push it to the Hub
125
+ transformer/ checkpoint (step 212 000) + config.json; the checkpoint describes its own architecture
126
+ vae/ f8c32-ae: encoder 68.4M + decoder 23.9M
127
+ text_encoder/ Ettin-150M
128
+ docs/ARCH.md architecture, decisions and the measurements behind them
129
+ docs/STATUS.md what works, what is next, known limits
130
+ docs/README.ru.md the same repo notes in Russian (development log)
131
+ docs/img/ sample grids
132
+ docs/presentation.pdf slide-by-slide walkthrough of the pyramid (Russian)
133
+ tmp/ runs (in .gitignore)
134
+ ```
135
+
136
+ ## Usage
137
+
138
+ Inference — one image from the bundled checkpoint (all flags below are defaults):
139
+
140
+ ```bash
141
+ python3 src/sample.py --prompt "A photograph of a red fox in fresh snow." --out fox.png
142
+ # 256 px, 16 body steps, 8 head steps, cfg 2.5, seed 0; --n 4 gives four seeds,
143
+ # --model tmp/run/ckpt/step_0212000.pt samples a training checkpoint directly
144
+ ```
145
+
146
+ The same from Python:
147
+
148
+ ```python
149
+ import sys; sys.path[:0] = ["src", "vae"]
150
+ from model import DEV, UDiT
151
+ from text import EttinText
152
+ from ae.model import AutoEncoder
153
+ from export import load_exported
154
+
155
+ ae = AutoEncoder.from_pretrained("vae", device=DEV)
156
+ enc = EttinText("text_encoder", DEV, max_len=192)
157
+ m = UDiT(ae, text_dim=enc.dim).to(DEV)
158
+ load_exported(m, enc, ae, "transformer", DEV); m.eval()
159
+
160
+ null_hs, null_mk = enc.slices([""], pad_to_len=192)
161
+ null_t = enc.mix(null_hs.float())
162
+ hs, mk = enc.slices(["A photograph of a red fox in fresh snow."], pad_to_len=192)
163
+ y = m.sample(enc.mix(hs.float()), mk, 32, 32, steps=16, tail_steps=8, cfg=2.5,
164
+ null_text=null_t, null_mask=null_mk) # (1, 3, 256, 256) in [-1, 1]
165
+ ```
166
+
167
+ ## Reproduce
168
+
169
+ The run behind the bundled checkpoint — one RTX 4080, batch 16 at 256², ~240 ms/step, 13 h per
170
+ 187 500-step epoch:
171
+
172
+ ```bash
173
+ python3 src/train.py --batch 16 --init tmp/flux2/ckpt/step_0192000.pt \
174
+ --hf-dataset LucasFang/FLUX-Reason-6M --epochs 1
175
+ # --parts Aesthetics-Part01,Aesthetics-Part02,Imaginative,Text (default)
176
+ # --out tmp/run every 1000 steps a panel to tmp/run/samples/latest.png,
177
+ # every 2000 steps a checkpoint to tmp/run/ckpt/, resume with --resume
178
+ ```
179
+
180
+ From scratch on a folder of `<name>.jpg` + `<name>.txt` pairs (bucket sides multiple of 64, centre
181
+ crop):
182
+
183
+ ```bash
184
+ python3 src/make_dataset.py --src ~/dataset/butterflies/butterfly \
185
+ --out ~/dataset/butterflies_256x64 --long-side 256 --eval-n 30
186
+ python3 src/train.py --data ~/dataset/butterflies_256x64/train \
187
+ --eval ~/dataset/butterflies_256x64/eval --batch 8 --steps 20000
188
+ ```
189
+
190
+ Export to `transformer/` after training: `python3 src/export.py --ckpt tmp/run/last.pt --out transformer`.
191
+ The checkpoint carries its own `args`, so `config.json` describes the run it came from.
192
+
193
+ Publish: `python3 src/publish.py --out tmp/publish --push` — copies the code and docs, converts the
194
+ three weight files to fp16, drops the unused MLM head of the text encoder and uploads 0.8 GiB to this
195
+ repository.
196
+
197
+ ## Licences
198
+
199
+ **MIT** for our code and weights. Bundled upstream components keep their own terms:
200
+ [`jhu-clsp/ettin-encoder-150m`](https://huggingface.co/jhu-clsp/ettin-encoder-150m) (MIT) and
201
+ [`Muinez/f8c32-ae`](https://huggingface.co/Muinez/f8c32-ae) (Apache-2.0). Training data:
202
+ `LucasFang/FLUX-Reason-6M` (Apache-2.0 per its dataset card; images generated with FLUX.1-dev);
203
+ the butterfly set is a local 970-image collection.