| --- |
| library_name: diffusers |
| tags: |
| - minimax-h3 |
| - vae |
| - video |
| --- |
| |
| # Sol-H3-VAE |
|
|
| A fast video decoder for [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3). It replaces the |
| 36-layer H3 VAE decoder with a 24-layer noncausal decoder (1.62B parameters) distilled from it: |
| whole frames with windowed attention instead of spatial tiles, and each native 5-latent block |
| decoded bidirectionally. It takes the same normalized latents and returns the same frames as |
| the H3 VAE decoder. The audio decoder is unchanged. |
|
|
| It is the video decoder of one stage in |
| [Sol-H3 in ComfyUI](https://github.com/NVlabs/Sana/tree/sol-engine/models/minimax_h3/Sol-H3-ComfyUI). |
|
|
| | One 5-second clip | H3 VAE | Sol-H3-VAE | |
| |---|---:|---:| |
| | 768p, RTX 5090 | 12.5 s | 5.0 s | |
| | 720p, GB200 | 6.16 s | 0.34 s | |
| | 2K, GB200 | 14.5 s | 0.89 s | |
|
|
| Against the H3 VAE decoder, PSNR is 45.4 dB on held-out latents (720p to 2K) and 38.2 dB on a |
| generated 768p clip. Timings on GB200 use a captured CUDA graph; the RTX 5090 timing is eager, |
| as Sol-H3 runs it. |
|
|
| ## Usage |
|
|
| Requirements: PyTorch with CUDA and [Diffusers](https://github.com/huggingface/diffusers) with |
| MiniMax-H3 support. The fast engine (default) also needs Triton, |
| [FlashAttention-4](https://github.com/Dao-AILab/flash-attention) (`flash_attn.cute`) and |
| [QuACK](https://github.com/Dao-AILab/quack); `engine="ref"` runs in plain PyTorch without them. |
| Tested on RTX 5090 with PyTorch 2.12 (CUDA 13), Triton 3.7, Diffusers 0.40, FlashAttention-4 |
| 4.0.0b19 and QuACK 0.6.3. |
|
|
| ```python |
| import sys |
| from huggingface_hub import hf_hub_download, snapshot_download |
| |
| repo = snapshot_download("Efficient-Large-Model/Sol-H3-VAE") |
| sys.path.insert(0, f"{repo}/code") |
| from sol_h3_vae import SolH3VAEDecoder |
| |
| vae_config = hf_hub_download("MiniMaxAI/MiniMax-H3", "vae/config.json") |
| decoder = SolH3VAEDecoder(vae_config, f"{repo}/sol_h3_vae.safetensors") |
| |
| # z: normalized H3 video latents, (1, T, 24, H/16, W/16); T = 0 starts a native 5-latent block |
| pixels = decoder.decode(z) # (1, 3, F, H, W), the H3 VAE decoder's normalized pixel space |
| rgb = decoder.decode_rgb_uint8(z) # (F, H, W, 3) uint8 on the GPU |
| ``` |
|
|
| The decoder needs only the H3 VAE `config.json`, not the H3 VAE weights. The first call for each |
| latent shape compiles kernels and captures a CUDA graph (about 25 s on RTX 5090); later calls |
| reuse it. |
|
|
| On SM120 GPUs (RTX 5090, DGX Spark), FlashAttention-4 4.0.0b19 to 4.0.0b22 has a varlen bug that |
| makes the fast engine hit an illegal memory access. Apply the guard from |
| [Sol-H3 in ComfyUI](https://github.com/NVlabs/Sana/blob/sol-engine/models/minimax_h3/Sol-H3-ComfyUI/sol_h3_engine/fa4_sm120.py) |
| before the first decode, or use `engine="ref"`. |
|
|
| ## Limitations |
|
|
| - Noncausal: it decodes a whole clip at once, so it cannot stream latent by latent. |
| - Memory grows with clip length: about 24 GiB at 720p and 48 GiB at 2K for a 5-second clip |
| with the CUDA graph. Sol-H3 runs the eager engine and moves the weights to host RAM between |
| requests to fit a 32 GB card. |
| - It is distilled, so it is close to the H3 VAE decoder but not identical. |
|
|
| ## License |
|
|
| The code is released under Apache-2.0. The weights are distilled from the MiniMax-H3 VAE and are |
| subject to the |
| [MiniMax-H3 license](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE). |
|
|
