README.md

---
library_name: diffusers
license: apache-2.0
base_model:
- MiniMaxAI/MiniMax-H3
- lightx2v/LightVAE
pipeline_tag: image-to-video
tags:
- minimax-h3
- video
- modular-diffusers
- lightvae
---

# MiniMax-H3 + LightVAE (modular diffusers)

A ready-to-load **modular diffusers** repo for [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) in which the
video VAE **decoder** is replaced by [lightx2v/LightVAE](https://huggingface.co/lightx2v/LightVAE)'s block-pruned Lite
decoder (ViT decoder pruned from 36 to 12 transformer blocks). LightVAE preserves H3's latent interface
(24-channel latents, same spatial/temporal compression, same `latents_mean/std`), so the transformer, the audio VAE,
and the scheduler are untouched and the decoder is a drop-in replacement.

Only the `vae` component points here; every other component (`transformer`, `text_encoder`, `audio_vae`, ...) is
loaded from the H3 base and pruned repos through this repo's `modular_model_index.json`. The included VAE also keeps
the **stock H3 encoder** (encode is cheap and leaves the latent interface untouched); only the heavy decode path is
swapped.

## Contents

```
modular_model_index.json     # vae -> this repo/vae ; everything else -> MiniMax-H3 / MiniMax-H3-Pruned
vae/                         # AutoencoderKLMiniMaxH3, decoder_num_layers=12 (stock encoder + LightVAE Lite decoder)
```

## Usage

Requires `diffusers` with the MiniMax-H3 integration.

Two things to know up front, both inherent to a stand-alone modular index (verified on a live Space):
1. The LightVAE `vae` loads automatically from this repo. The **pruned transformer is remote code in a different
   repo**, and `trust_remote_code` is only forwarded to components that live in this index's own repo, so
   `transformer_ref` has to be loaded explicitly and injected with `update_components`.
2. Components load on CPU; move the model components to the GPU before calling the pipeline.

```python
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from diffusers import ComponentsManager, AutoModel
from diffusers.modular_pipelines.minimax_h3 import MiniMaxH3Blocks
from diffusers.modular_pipelines.minimax_h3.references import MiniMaxH3VideoReference

# ref2va workflow: loads only the reference transformer partition (not both)
blocks = MiniMaxH3Blocks(workflow="ref2va")
pipe = blocks.init_pipeline(
    "diffusers-modular/MiniMax-H3-LightVAE",
    components_manager=ComponentsManager(),
    collection="h3",
)
pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)  # LightVAE `vae` loads here

# the remote-code pruned transformer lives in a different repo, so load + inject it explicitly
if getattr(pipe, "transformer_ref", None) is None:
    transformer_ref = AutoModel.from_pretrained(
        "multimodalart/MiniMax-H3-Pruned", subfolder="transformer_ref",
        trust_remote_code=True, torch_dtype=torch.bfloat16,
    )
    pipe.update_components(transformer_ref=transformer_ref)

# both autoencoders stay fp32 on the native attention backend
pipe.vae.set_attention_backend("native")
pipe.audio_vae.set_attention_backend("native")

# lightx2v 4-step Ref2V Turbo LoRA. Its keys target the transformer submodules with no `transformer.` prefix,
# so load it onto the component with prefix=None (and drop the PEFT `.default.` adapter segment from the keys).
turbo = load_file(hf_hub_download("lightx2v/Minimax-h3-Turbo", "minimax_h3_ref2v_turbo_4step_v0.1_bf16.safetensors"))
turbo = {k.replace(".lora_A.default.weight", ".lora_A.weight").replace(".lora_B.default.weight", ".lora_B.weight"): v
         for k, v in turbo.items()}
pipe.transformer_ref.load_lora_adapter(turbo, prefix=None, adapter_name="turbo")
pipe.transformer_ref.set_adapters(["turbo"], [1.0])

# components load on CPU; move the models to the GPU (VAEs keep fp32)
for name in ("transformer_ref", "vae", "audio_vae"):
    getattr(pipe, name).to("cuda")

state = pipe(
    prompt="a red fox walking through snow, cinematic",
    references=[MiniMaxH3VideoReference.from_file("reference.mp4")],
    height=768,
    width=768,
    num_frames=124,          # must be 17*n + 5
    num_inference_steps=4,   # 4 with the Turbo LoRA; ~20 without
    generator=torch.Generator("cpu").manual_seed(0),
)

frames = state.get("videos")[0]          # list of PIL frames
audio = state.get("audio")               # waveform (or None)
sampling_rate = state.get("sampling_rate")
```

Swapping back to the stock decoder is just pointing `vae` at `MiniMaxAI/MiniMax-H3` / `vae` in the index.

## How it was built

`vae/` is an `AutoencoderKLMiniMaxH3` instantiated with `decoder_num_layers=12`, loaded with:
- `encoder.*` + `quant_conv.*` from `MiniMaxAI/MiniMax-H3/vae` (stock H3 encoder),
- `decoder.*` + `post_quant_conv.*` from `lightx2v/LightVAE/lightvae-lite-h3-decoder.safetensors`.

`latents_mean/std` are bit-identical between the stock VAE config and the LightVAE decoder checkpoint, confirming the
shared latent interface. State dict loads strict (0 missing, 0 unexpected).

## Roadmap

- **Lite encoder.** LightVAE also ships a Lite *encoder* (`lightvae-lite-h3-encoder.safetensors`, non-uniform resnet
  depth `[1,1,1,1,1,2]`). Loading it needs `AutoencoderKLMiniMaxH3` to accept a per-level `layers_per_block`; tracked
  as a follow-up.
- **int8 decoder.** `lightvae-lite-h3-decoder-int8-convrot.safetensors` for a cheaper decode once its kernel path is wired.

## Credits

LightVAE by [lightx2v](https://huggingface.co/lightx2v/LightVAE). MiniMax-H3 by [MiniMaxAI](https://huggingface.co/MiniMaxAI/MiniMax-H3).