| # AdaLN: 13B timestep-only parameters → a rank-12 basis | |
|  | |
| <!-- BEGIN:params --> | |
| | Part | Parameters | | |
| |---|---| | |
| | AdaLN projections | 13.01B | | |
| | Attention | 7.71B | | |
| | Feed-forward | 11.56B | | |
| | Other | 0.84B | | |
| | **Total** | **33.12B** | | |
| <!-- END:params --> | |
| The AdaLN projections map `silu(temb(t))` — a function of the scalar timestep only — to every block's | |
| shift/scale/gate table. The timestep enters a 256-channel sinusoid unscaled in [0, 1], so the curve it traces | |
| is extremely smooth: its singular values fall to 1e-7 of the first by rank 12. | |
|  | |
| <!-- BEGIN:adaln --> | |
| | Rank | Max rel. error | Mean rel. error | | |
| |---|---|---| | |
| | 4 | 3.1e-01 | 1.2e-03 | | |
| | 6 | 6.0e-02 | 1.5e-04 | | |
| | 8 | 4.4e-03 | 1.3e-05 | | |
| | 10 | 1.2e-03 | 3.5e-06 | | |
| | 12 | 1.2e-04 | 4.4e-07 | | |
| | 16 | 1.2e-04 | 4.3e-07 | | |
| | 24 | 1.2e-04 | 4.3e-07 | | |
| | 32 | 1.1e-04 | 4.4e-07 | | |
| | shipped bf16 path vs fp32 | 1.7e-01 | 1.5e-03 | | |
| <!-- END:adaln --> | |
| Both ComfyUI's rank-8 curve form and this rank-12 basis are orders of magnitude below the rounding error the | |
| shipped bf16 checkpoint already has. The rank-12 basis is kept for margin and because it allows a LoRA's AdaLN | |
| deltas to be folded in exactly (ComfyUI's pruned files store no basis, so such LoRA tensors are skipped). | |