AdaLN: 13B timestep-only parameters → a rank-12 basis
| Part | Parameters |
|---|---|
| AdaLN projections | 13.01B |
| Attention | 7.71B |
| Feed-forward | 11.56B |
| Other | 0.84B |
| Total | 33.12B |
The AdaLN projections map silu(temb(t)) — a function of the scalar timestep only — to every block's
shift/scale/gate table. The timestep enters a 256-channel sinusoid unscaled in [0, 1], so the curve it traces
is extremely smooth: its singular values fall to 1e-7 of the first by rank 12.
| Rank | Max rel. error | Mean rel. error |
|---|---|---|
| 4 | 3.1e-01 | 1.2e-03 |
| 6 | 6.0e-02 | 1.5e-04 |
| 8 | 4.4e-03 | 1.3e-05 |
| 10 | 1.2e-03 | 3.5e-06 |
| 12 | 1.2e-04 | 4.4e-07 |
| 16 | 1.2e-04 | 4.3e-07 |
| 24 | 1.2e-04 | 4.3e-07 |
| 32 | 1.1e-04 | 4.4e-07 |
| shipped bf16 path vs fp32 | 1.7e-01 | 1.5e-03 |
Both ComfyUI's rank-8 curve form and this rank-12 basis are orders of magnitude below the rounding error the shipped bf16 checkpoint already has. The rank-12 basis is kept for margin and because it allows a LoRA's AdaLN deltas to be folded in exactly (ComfyUI's pruned files store no basis, so such LoRA tensors are skipped).