Add model card

README.md

@@ -0,0 +1,186 @@
+---
+license: other
+license_name: minimax-h3-community-license-agreement
+base_model: MiniMaxAI/MiniMax-H3
+tags:
+  - video
+  - vae
+  - decoder
+  - nvfp4
+  - fp4
+  - quantization
+  - blackwell
+  - fastvideo
+  - minimax-h3
+---
+
+# FastH3 NVFP4 VAE Decoder
+
+NVFP4 (4-bit floating point) replacements for the video decoder of the
+[MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) VAE, built for the FastH3
+stacks in [FastVideo](https://github.com/hao-ai-lab/FastVideo).
+
+The H3 video decoder is a 36-block ViT (hidden size 2048, SwiGLU FFN 8192). These
+decoders quantize every transformer-block linear (Q, K, V, attention output, FFN in,
+FFN out) to NVFP4 W4A4: weights and activations both in E2M1 with E4M3 scales per
+16 values. They were trained with quantization-aware distillation (QAD) against the
+original full-precision decoder. `proj_in`, `proj_out`, norms, register tokens and
+`post_quant_conv` stay in high precision. They decode the same latents as the
+original VAE, so no DiT or encoder changes are needed.
+
+## Two tiers
+
+| Tier | Folder | Decoder blocks | Resident weights after packing | Use it when |
+|---|---|---:|---:|---|
+| Quality, **C-v4** | `quality-36L/` | 36 (full depth) | ~1.36 GB | you want output that looks the same as the original decoder |
+| Fast, **F8-v2** | `fast-8L/` | 8 | ~0.30 GB | decode time or memory matters most and you can accept some loss |
+
+For comparison, the original decoder's weights are 9.7 GB in fp32 or 4.8 GB in bf16.
+
+## Quality
+
+LPIPS (AlexNet) against the original fp32 decoder on the same latents. Lower is better.
+
+| Latents | Quality tier (C-v4) LPIPS / PSNR | Fast tier (F8-v2) LPIPS / PSNR |
+|---|---:|---:|
+| FastH3 V2 generations (9 clips, 480p) | 0.0104 / 37.9 dB | 0.0684 / 28.5 dB |
+| FastH3 OmniRef generations (20 clips) | 0.0039 / 46.0 dB | 0.0276 / 36.1 dB |
+| Real videos, VAE-encoded, held out (6 clips) | 0.0038 / 45.0 dB | 0.0330 / 34.3 dB |
+
+These numbers were reproduced from the released files through FastVideo's loader path,
+with weights packed to FP4 and the fused kernels.
+
+### Blind side-by-side votes
+
+Raters watched decoded clips side by side without knowing which decoder made which
+clip. The clips came from FastH3 V2 and OmniRef generations.
+
+| Pairing | Result (wins–losses, ties) |
+|---|---|
+| Quality tier vs original decoder | 2–2, 7 ties: indistinguishable |
+| Quality tier vs TAEH3 | 11–0 |
+| Fast tier vs TAEH3 | 7–1 |
+| Fast tier vs original decoder | 0–7, 3 ties |
+
+## Speed
+
+Decode time for an 832x480, 124-frame clip on one GB200, warm, using FastVideo's fused
+NVFP4 kernels with every spatial tile batched in one decoder call:
+
+| Decoder | Seconds per clip |
+|---|---:|
+| Original decoder, fp16, `torch.compile` | 0.98 |
+| Quality tier (C-v4) | 0.53 |
+| Fast tier (F8-v2) | 0.13 |
+
+## Hardware
+
+NVFP4 matrix multiplies need an NVIDIA Blackwell GPU:
+
+- sm_100: B200, GB200
+- sm_120: RTX 50-series, RTX PRO 6000 Blackwell
+- sm_121: DGX Spark (GB10)
+
+On GB10, FastVideo uses the cuDNN FP4 GEMM backend; elsewhere it uses CUTLASS
+through FlashInfer. The two produce bit-identical output. The fused decoder kernels
+were benchmarked on GB200 and DGX Spark.
+
+On GPUs without FP4 tensor cores (Hopper, Ada, Ampere and older) or on CPU/MPS, the
+FP4 kernels are unavailable, so loading or decoding with these files fails. There is
+no automatic fallback. Use the original H3 VAE decoder on that hardware.
+
+## How to use with FastVideo
+
+> **Not merged upstream yet.** The loader lives on a FastVideo fork branch,
+> [`aryan5v/FastVideo@aryan/h3-nvfp4-vae-spark`](https://github.com/aryan5v/FastVideo/tree/aryan/h3-nvfp4-vae-spark).
+> The training and benchmark code is on
+> [`aryan/h3-nvfp4-decoder`](https://github.com/aryan5v/FastVideo/tree/aryan/h3-nvfp4-decoder).
+> Names and formats may change before the code is merged.
+
+1. Install FastVideo from the fork branch, with FlashInfer, on a Blackwell GPU.
+2. Download a tier and convert it to the `.pt` format the loader reads. The
+   conversion only regroups the tensors; values are unchanged.
+
+   ```bash
+   hf download FastVideo/FastH3-NVFP4-VAE-Decoder --local-dir fasth3-nvfp4-vae
+   python fasth3-nvfp4-vae/convert_to_fastvideo_pt.py \
+       fasth3-nvfp4-vae/quality-36L/decoder.safetensors quality-36L.pt
+   ```
+
+3. Point FastVideo at it and run any MiniMax-H3 / FastH3 pipeline as usual:
+
+   ```bash
+   export FASTVIDEO_H3_VAE_NVFP4_DECODER=$PWD/quality-36L.pt
+   ```
+
+When the VAE loads, FastVideo:
+
+- replaces the decoder with the NVFP4 one;
+- skips the INT8 ConvRot decoder overlay;
+- packs the weights to FP4, so only the packed weights stay on the GPU;
+- runs the decode in bf16 autocast, the precision the decoders were trained in.
+
+Which host VAE each tier needs:
+
+- **Quality tier (36 blocks):** the stack's `vae/` folder must hold the full
+  36-block VAE from `MiniMaxAI/MiniMax-H3`. The 26-block light VAE in the FastH3
+  V2/Trim repos is too shallow, and the loader will refuse it.
+- **Fast tier (8 blocks):** any H3 video VAE works, including the 26-block light
+  VAE that ships with FastH3 V2/Trim. The loader cuts the VAE down to 8 blocks.
+
+Useful environment variables:
+
+- `FASTVIDEO_H3_VAE_NVFP4_FUSED=1`: use the fused kernels. This is the default.
+- `FASTVIDEO_H3_VAE_NVFP4_CUDA_GRAPH=1`: replay small decoder calls as CUDA graphs.
+  This is the default.
+- `FASTVIDEO_H3_VAE_TILE_BATCH`: how many spatial tiles go into one decoder call.
+  `12` covers all tiles at 832x480.
+
+## Files
+
+```
+quality-36L/decoder.safetensors   # C-v4 weights
+quality-36L/config.json           # architecture + NVFP4 settings
+fast-8L/decoder.safetensors       # F8-v2 weights
+fast-8L/config.json
+convert_to_fastvideo_pt.py        # safetensors -> FastVideo loader .pt
+```
+
+Each `decoder.safetensors` holds `decoder.*` and `post_quant_conv.*` tensors. The
+safetensors header carries the NVFP4 settings (`format`, `nvfp4_metadata`). The
+quantized linear weights are stored as bf16 master weights. FastVideo turns them
+into NVFP4 at load with the same deterministic quantizer used in training, so the
+files on disk are larger than the resident sizes above:
+
+- quality tier: 4.86 GB on disk;
+- fast tier: 1.10 GB on disk.
+
+## Training
+
+- **Teacher:** the original 36-block MiniMax-H3 video decoder.
+- **Students:**
+  - Quality tier: the full 36-block decoder.
+  - Fast tier: 8 evenly spaced blocks of the H3 decoder.
+  - Both were trained over several QAD rounds, each starting from the previous one.
+- **Method:** quantization-aware distillation. The forward pass uses exactly the
+  NVFP4 numerics used at inference; gradients pass straight through the quantizer.
+  The loss is pixel L1 plus LPIPS. The activation global scale is fixed at 1.0 and
+  no Hadamard rotation is used.
+- **Data:** decoder tiles cut from latents of FastH3 generations plus real video:
+  - FastH3 V1, V2 (NVFP4 and bf16), Trim (NVFP4, FP8, bf16) and OmniRef DiT
+    outputs, i.e. the latents the decoder sees in production;
+  - VAE-encoded real videos.
+- **Checkpoint selection:** mean LPIPS on held-out FastH3 V2 and OmniRef latents.
+
+## Limitations
+
+- These are lossy decoders. Blind raters could not tell the quality tier from the
+  original. The fast tier lost every decisive vote against the original, so check
+  your own content before using it.
+- They require Blackwell FP4 hardware, as described above.
+- They only decode video. The encoder and the audio VAE are unchanged.
+
+## License
+
+These weights are derived from the MiniMax-H3 VAE and are subject to the
+[MiniMax-H3 Community License Agreement](https://huggingface.co/MiniMaxAI/MiniMax-H3).