Finalize FastH3 OmniRef release model card

README.md

@@ -1,237 +1,81 @@
 ---
 license: other
 license_name: minimax-h3-community-license-agreement
+license_link: https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE
 base_model: MiniMaxAI/MiniMax-H3
 tags:
-  - video
-  - vae
-  - decoder
-  - nvfp4
-  - fp4
-  - quantization
-  - blackwell
-  - fastvideo
-  - minimax-h3
+- video
+- vae
+- decoder
+- nvfp4
+- fp4
+- blackwell
+- fastvideo
+- minimax-h3
 ---
 
 # FastH3 NVFP4 VAE Decoder
 
-NVFP4 (4-bit floating point) replacements for the video decoder of the
-[MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) VAE, built for the FastH3
-stacks in [FastVideo](https://github.com/hao-ai-lab/FastVideo).
+Blackwell NVFP4 video decoders for MiniMax-H3 and FastH3, including OmniRef. They replace the video decoder while retaining the reference encoder, audio VAE and video latent space.
 
-The H3 video decoder is a 36-block ViT (hidden size 2048, SwiGLU FFN 8192). These
-decoders quantize every transformer-block linear (Q, K, V, attention output, FFN in,
-FFN out) to NVFP4 W4A4: weights and activations both in E2M1 with E4M3 scales per
-16 values. They were trained with quantization-aware distillation (QAD) against the
-original full-precision decoder. `proj_in`, `proj_out`, norms, register tokens and
-`post_quant_conv` stay in high precision. They decode the same latents as the
-original VAE, so no DiT or encoder changes are needed.
+[FastVideo](https://github.com/hao-ai-lab/FastVideo) · [OmniRef decoder setup](https://github.com/hao-ai-lab/FastVideo/blob/main/docs/inference/fasth3_omniref.md) · [OmniRef release collection](https://huggingface.co/collections/FastVideo/fasth3-omniref)
 
-## Two tiers
+## Choose a decoder
 
-| Tier | Folder | Decoder blocks | Decoder on the GPU | Use it when |
-|---|---|---:|---:|---|
-| Quality, **C-v4** | `quality-36L/` | 36 (full depth) | 1.29 GiB | you want output that looks the same as the original decoder |
-| Fast, **F8-v2** | `fast-8L/` | 8 | 0.31 GiB | decode time or memory matters most and you can accept some loss |
+| Tier | Folder | Decoder layers | Resident decoder weights |
+| --- | --- | ---: | ---: |
+| Quality | `quality-36L/` | 36 | 1.29 GiB |
+| Fast | `fast-8L/` | 8 | 0.31 GiB |
 
-Each tier ships in two formats that decode **bit-identically**; see [Files](#files).
+Choose the quality tier for closer reconstruction or the fast tier for lower decode cost with a quality tradeoff. Both tiers are lossy.
 
-For comparison, the original decoder's weights are 9.7 GB in fp32 or 4.8 GB in bf16.
+## Use with FastVideo
 
-## Quality
+Download a packed decoder:
 
-LPIPS (AlexNet) against the original fp32 decoder on the same latents. Lower is better.
-
-| Latents | Quality tier (C-v4) LPIPS / PSNR | Fast tier (F8-v2) LPIPS / PSNR |
-|---|---:|---:|
-| FastH3 V2 generations (9 clips, 480p) | 0.0104 / 37.9 dB | 0.0684 / 28.5 dB |
-| FastH3 OmniRef generations (20 clips) | 0.0039 / 46.0 dB | 0.0276 / 36.1 dB |
-| Real videos, VAE-encoded, held out (6 clips) | 0.0038 / 45.0 dB | 0.0330 / 34.3 dB |
-
-These numbers were reproduced from both released formats through FastVideo's loader,
-with the fused kernels. The two formats produced bit-identical frames on all 35 clips.
-
-### Blind side-by-side votes
-
-Raters watched decoded clips side by side without knowing which decoder made which
-clip. The clips came from FastH3 V2 and OmniRef generations.
-
-| Pairing | Result (wins–losses, ties) |
-|---|---|
-| Quality tier vs original decoder | 2–2, 7 ties: indistinguishable |
-| Quality tier vs TAEH3 | 11–0 |
-| Fast tier vs TAEH3 | 7–1 |
-| Fast tier vs original decoder | 0–7, 3 ties |
+```bash
+hf download FastVideo/FastH3-NVFP4-VAE-Decoder \
+  quality-36L/decoder-nvfp4.safetensors --local-dir ./fasth3-vae
+```
 
-## Speed
+Follow the [FastVideo OmniRef guide](https://github.com/hao-ai-lab/FastVideo/blob/main/docs/inference/fasth3_omniref.md) and pass the downloaded file with `--decoder` to `examples/inference/basic/basic_fasth3_omniref_pdd.py`. Use `fast-8L/decoder-nvfp4.safetensors` to select the fast tier instead.
 
-Decode time for an 832x480, 124-frame clip on one GB200, warm, using FastVideo's fused
-NVFP4 kernels with every spatial tile batched in one decoder call:
+The 36-layer decoder needs a full 36-layer MiniMax-H3 host VAE. The 8-layer tier also works with the light H3 VAE. The OmniRef guide explains model composition and device profiles.
 
-| Decoder | Seconds per clip |
-|---|---:|
-| Original decoder, fp16, `torch.compile` | 0.98 |
-| Quality tier (C-v4) | 0.53 |
-| Fast tier (F8-v2) | 0.13 |
+For UniServe deployment of the complete OmniRef model on GB200/B200, RTX PRO 6000 Blackwell or Hopper, follow the [UniServe guide](https://github.com/hao-ai-lab/uniserve/blob/main/docs/minimax_h3/minimax_h3.md) with the [BF16 model](https://huggingface.co/FastVideo/FastVideo-FastH3-Omni-8-Step-V1). These files are FastVideo decoder components, not complete diffusion checkpoints.
 
 ## Hardware
 
-NVFP4 matrix multiplies need an NVIDIA Blackwell GPU:
-
-- sm_100: B200, GB200
-- sm_120: RTX 50-series, RTX PRO 6000 Blackwell
-- sm_121: DGX Spark (GB10)
-
-On GB10, FastVideo uses the cuDNN FP4 GEMM backend; elsewhere it uses CUTLASS
-through FlashInfer. The two produce bit-identical output. The fused decoder kernels
-were benchmarked on GB200 and DGX Spark.
-
-On GPUs without FP4 tensor cores (Hopper, Ada, Ampere and older) or on CPU/MPS, the
-FP4 kernels are unavailable, so decoding with either format fails. There is no
-automatic fallback. Use the original H3 VAE decoder on that hardware. The bf16 file
-is still useful there as a source for other quantization schemes.
-
-## How to use with FastVideo
-
-> **Not merged upstream yet.** The loader lives on a FastVideo fork branch,
-> [`aryan5v/FastVideo@aryan/h3-nvfp4-vae-spark`](https://github.com/aryan5v/FastVideo/tree/aryan/h3-nvfp4-vae-spark).
-> The training and benchmark code is on
-> [`aryan/h3-nvfp4-decoder`](https://github.com/aryan5v/FastVideo/tree/aryan/h3-nvfp4-decoder).
-> Names and formats may change before the code is merged.
+The FP4 decoder kernels require NVIDIA Blackwell: GB200/B200, RTX PRO 6000 Blackwell, RTX 50-series or DGX Spark (GB10). Follow the [local OmniRef guide](https://github.com/hao-ai-lab/FastVideo/blob/main/docs/inference/fasth3_omniref.md#device-profiles) for Spark and RTX 5090 setup. Hopper and older GPUs should use the original H3 video decoder.
 
-1. Install FastVideo from the fork branch, with FlashInfer, on a Blackwell GPU.
-2. Download one file. The packed file is the usual choice:
-
-   ```bash
-   hf download FastVideo/FastH3-NVFP4-VAE-Decoder quality-36L/decoder-nvfp4.safetensors \
-       --local-dir fasth3-nvfp4-vae
-   ```
-
-3. Point FastVideo at the file and run any MiniMax-H3 / FastH3 pipeline as usual:
-
-   ```bash
-   export FASTVIDEO_H3_VAE_NVFP4_DECODER=$PWD/fasth3-nvfp4-vae/quality-36L/decoder-nvfp4.safetensors
-   ```
-
-   The variable accepts either format, `decoder-nvfp4.safetensors` or
-   `decoder.safetensors`, directly.
-
-When the VAE loads, FastVideo:
-
-- swaps the decoder on the CPU before the VAE moves to the GPU, so the dense
-  decoder it replaces never reaches the GPU;
-- uses the packed FP4 weights as they are, or packs the bf16 weights at load;
-- skips the INT8 ConvRot decoder overlay;
-- runs the decode in bf16 autocast, the precision the decoders were trained in.
+## Files
 
-`convert_to_fastvideo_pt.py` is only needed with older checkouts of the branch,
-whose loader read `.pt` files only. It converts `decoder.safetensors`; the packed
-format needs the current loader.
+Each tier includes:
 
-Which host VAE each tier needs:
+- `decoder-nvfp4.safetensors`: packed NVFP4 weights for inference.
+- `decoder.safetensors`: BF16 master weights.
+- `config.json`: decoder architecture and quantization configuration.
 
-- **Quality tier (36 blocks):** the stack's `vae/` folder must hold the full
-  36-block VAE from `MiniMaxAI/MiniMax-H3`. The 26-block light VAE in the FastH3
-  V2/Trim repos is too shallow, and the loader will refuse it.
-- **Fast tier (8 blocks):** any H3 video VAE works, including the 26-block light
-  VAE that ships with FastH3 V2/Trim. The loader cuts the VAE down to 8 blocks.
+## Quality and speed
 
-Useful environment variables:
+Reconstruction against the original FP32 decoder on the same latents; lower LPIPS and higher PSNR are better:
 
-- `FASTVIDEO_H3_VAE_NVFP4_FUSED=1`: use the fused kernels. This is the default.
-- `FASTVIDEO_H3_VAE_NVFP4_CUDA_GRAPH=1`: replay small decoder calls as CUDA graphs.
-  This is the default.
-- `FASTVIDEO_H3_VAE_TILE_BATCH`: how many spatial tiles go into one decoder call.
-  `12` covers all tiles at 832x480.
+| Evaluation clips | Quality tier LPIPS / PSNR | Fast tier LPIPS / PSNR |
+| --- | ---: | ---: |
+| FastH3 V2 generations, 9 clips at 480p | 0.0104 / 37.9 dB | 0.0684 / 28.5 dB |
+| FastH3 OmniRef generations, 20 clips | 0.0039 / 46.0 dB | 0.0276 / 36.1 dB |
+| Real videos, VAE-encoded, 6 held-out clips | 0.0038 / 45.0 dB | 0.0330 / 34.3 dB |
 
-## Files
+Warm video-decode time for one 832×480, 124-frame clip on one GB200, using FastVideo's fused NVFP4 kernels and batching all spatial tiles in one decoder call:
 
-```
-quality-36L/decoder-nvfp4.safetensors   # C-v4, pre-packed NVFP4 (1.39 GB)
-quality-36L/decoder.safetensors         # C-v4, bf16 master weights (4.86 GB)
-quality-36L/config.json                 # architecture, NVFP4 settings, both formats
-fast-8L/decoder-nvfp4.safetensors       # F8-v2, pre-packed NVFP4 (0.33 GB)
-fast-8L/decoder.safetensors             # F8-v2, bf16 master weights (1.10 GB)
-fast-8L/config.json
-convert_to_fastvideo_pt.py              # decoder.safetensors -> .pt, for older loaders
-```
+| Decoder | Seconds per clip |
+| --- | ---: |
+| Original FP16 decoder with `torch.compile` | 0.98 |
+| Quality, 36 layers | 0.53 |
+| Fast, 8 layers | 0.13 |
 
-### Which format to use
-
-- **`decoder-nvfp4.safetensors` (packed):** pick this to run the model. It is the
-  smallest download and uses the least GPU memory while loading. It contains
-  Blackwell FP4 data, so it is Blackwell-only.
-- **`decoder.safetensors` (bf16):** this is the reference copy. Use it to
-  re-quantize, fine-tune, run on a different FP4 kernel stack, or check the packed
-  file. FastVideo packs it at load with the same deterministic quantizer used in
-  training.
-
-Peak memory while FastVideo loads the VAE (GB200; the encoder and the rest of the
-VAE are included):
-
-| | GPU, packed | GPU, bf16 | Host RAM (both formats) |
-|---|---:|---:|---:|
-| Quality tier, on the full 36-block VAE | 1.97 GiB | 9.78 GiB | about 23 GiB |
-| Fast tier, on the 26-block light VAE | 0.99 GiB | 2.78 GiB | about 14 GiB |
-
-The GPU memory after loading is the same for both formats. Host RAM peaks the same
-way for both because the host VAE's own fp32 checkpoint is read in full.
-
-### Layout
-
-Both formats store tensors under `decoder.*` and `post_quant_conv.*`. The
-safetensors header carries `format` and `nvfp4_metadata`, a JSON string with the
-block size, activation scale, number of NVFP4 linears and decoder depth.
-
-- **`decoder.safetensors`**
-  - `format`: `fastvideo_h3_decoder_nvfp4_deploy_v1`.
-  - The quantized linears' weights and biases are bf16.
-  - Every other tensor is fp32, the dtype it has in the trained decoder.
-- **`decoder-nvfp4.safetensors`**
-  - `format`: `fastvideo_h3_decoder_nvfp4_packed_v1`.
-  - `nvfp4_metadata` also describes the layout (`packed_layout`) and the GPU and
-    FlashInfer version used to pack.
-  - Each quantized linear `<name>` has these tensors:
-
-    | Tensor | Contents |
-    |---|---|
-    | `<name>.packed_weight` | uint8 `[out, in/2]`, two E2M1 codes per byte (FlashInfer `fp4_quantize` order) |
-    | `<name>.weight_inv_scale` | per-16-element E4M3 block scales, stored as uint8, in FlashInfer's swizzled `SfLayout.layout_128x4` |
-    | `<name>.weight_global_sf` | fp32 scalar |
-    | `<name>.bias` | bf16 |
-    | `<name>.input_amax` | fp32 static activation range, unused with the unit activation scale these decoders use |
-
-  - Norms, `proj_in`, `proj_out`, register tokens and `post_quant_conv` stay
-    fp32. That is the dtype the decoder was trained and validated with, so the
-    output stays bit-identical.
-
-## Training
-
-- **Teacher:** the original 36-block MiniMax-H3 video decoder.
-- **Students:**
-  - Quality tier: the full 36-block decoder.
-  - Fast tier: 8 evenly spaced blocks of the H3 decoder.
-  - Both were trained over several QAD rounds, each starting from the previous one.
-- **Method:** quantization-aware distillation. The forward pass uses exactly the
-  NVFP4 numerics used at inference; gradients pass straight through the quantizer.
-  The loss is pixel L1 plus LPIPS. The activation global scale is fixed at 1.0 and
-  no Hadamard rotation is used.
-- **Data:** decoder tiles cut from latents of FastH3 generations plus real video:
-  - FastH3 V1, V2 (NVFP4 and bf16), Trim (NVFP4, FP8, bf16) and OmniRef DiT
-    outputs, i.e. the latents the decoder sees in production;
-  - VAE-encoded real videos.
-- **Checkpoint selection:** mean LPIPS on held-out FastH3 V2 and OmniRef latents.
-
-## Limitations
-
-- These are lossy decoders. Blind raters could not tell the quality tier from the
-  original. The fast tier lost every decisive vote against the original, so check
-  your own content before using it.
-- They require Blackwell FP4 hardware, as described above.
-- They only decode video. The encoder and the audio VAE are unchanged.
+These are decoder-only measurements. Full video/audio generation also includes encoding, denoising and media output.
 
 ## License
 
-These weights are derived from the MiniMax-H3 VAE and are subject to the
-[MiniMax-H3 Community License Agreement](https://huggingface.co/MiniMaxAI/MiniMax-H3).
+Use of these weights is governed by the [MiniMax-H3 Community License Agreement](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE).