recoilme commited on
Commit
0f8003c
·
verified ·
1 Parent(s): 8be5ec9

Card: why the pyramid (11x less compute)

Browse files
Files changed (1) hide show
  1. README.md +18 -2
README.md CHANGED
@@ -20,8 +20,24 @@ learnable layer-slice mixer (10.0M), and the AE decoder retrained as a pixel-spa
20
  | training | FLUX-Reason-6M, one RTX 4080 |
21
  | licence | MIT |
22
 
23
- **Training is still running** — the bundled `transformer/` is a snapshot of step **212 000** (lr was
24
- restarted at 1e-4 on step 192 000, so it is mid-recovery and will be replaced).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
25
 
26
  ## Examples
27
 
 
20
  | training | FLUX-Reason-6M, one RTX 4080 |
21
  | licence | MIT |
22
 
23
+ ## Why the pyramid
24
+
25
+ At the same parameter count, a flat classic DiT (12 blocks at width 1024, one resolution) costs
26
+ **232 GFLOP** per forward pass of the body; the pyramid costs **20.4** — **11× less compute**. The
27
+ step itself runs 97 ms vs 196 ms because it is memory-bound, and the detail metric is 0.54 vs 0.18
28
+ (ground truth 0.913, AE ceiling 0.859).
29
+
30
+ - Width grows as tokens shrink: expensive layers run on 16–64 tokens, while the 1024-token grid keeps
31
+ cheap 3×3 convolutions — convolutions are local, so anatomy and texture survive.
32
+ - Additive skip connections at every level: fine structure is built at its own level instead of being
33
+ pulled through one single-resolution attention stack.
34
+ - Text reaches all four levels; the 16-token bottleneck (width 1536, full attention) holds composition
35
+ and symmetry.
36
+
37
+ ## Training in progress
38
+
39
+ The bundled `transformer/` is a snapshot of step **212 000**; lr was restarted at 1e-4 on step 192 000,
40
+ so it is mid-recovery and will be replaced.
41
 
42
  ## Examples
43