Other
Wan2.2
MiniMax H3
video-vae
video-generation
distillation
image-to-video
text-to-video
comfyui
wan2.1
Instructions to use lightx2v/LightVAE with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Wan2.2
How to use lightx2v/LightVAE with Wan2.2:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Update README.md
#2
by gushiqiao - opened
README.md
CHANGED
|
@@ -13,92 +13,89 @@ tags:
|
|
| 13 |
- minimax-h3
|
| 14 |
---
|
| 15 |
|
| 16 |
-
#
|
| 17 |
|
| 18 |
-
|
| 19 |
|
| 20 |
-
|
| 21 |
|
| 22 |
-
**
|
| 23 |
-
**Code and examples:** [ModelTC/LightVAE](https://github.com/ModelTC/LightVAE)
|
| 24 |
|
| 25 |
## 📦 Checkpoints
|
| 26 |
|
| 27 |
-
| Backbone |
|
| 28 |
-
| :-- | :-- | :-- |
|
| 29 |
-
| Wan2.1 | Pro decoder | [
|
| 30 |
-
| Wan2.1 | Lite decoder | [
|
| 31 |
-
| Wan2.2 | Pro decoder | [
|
| 32 |
-
| Wan2.2 | Lite decoder | [
|
| 33 |
-
| MiniMax-H3 | Lite decoder | [
|
| 34 |
-
| MiniMax-H3 | Lite encoder | [
|
| 35 |
|
| 36 |
-
**Pro**
|
| 37 |
|
| 38 |
-
Wan models
|
| 39 |
|
| 40 |
-
> **
|
| 41 |
|
| 42 |
-
##
|
| 43 |
|
| 44 |
-
|
| 45 |
|
| 46 |
-
###
|
| 47 |
|
| 48 |
-
|
| 49 |
|
| 50 |
```bash
|
| 51 |
cd /path/to/ComfyUI
|
| 52 |
|
| 53 |
git clone https://github.com/ModelTC/LightVAE.git LightVAE
|
| 54 |
-
python -m pip install ./LightVAE huggingface_hub
|
| 55 |
python LightVAE/comfyui/install.py --comfy-root .
|
| 56 |
|
|
|
|
| 57 |
hf download lightx2v/LightVAE \
|
| 58 |
-
lightvae-lite-h3-decoder
|
|
|
|
| 59 |
--local-dir models/vae/LightVAE
|
| 60 |
```
|
| 61 |
|
| 62 |
-
|
| 63 |
|
| 64 |
-
|
| 65 |
|
| 66 |
-
|
| 67 |
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
|
| 74 |
-
|
| 75 |
|
| 76 |
-
|
| 77 |
|
| 78 |
-
|
| 79 |
|
| 80 |
-
|
|
|
|
|
|
|
| 81 |
|
| 82 |
-
|
| 83 |
-
hf download lightx2v/LightVAE \
|
| 84 |
-
lightvae-lite-h3-encoder-comfy.safetensors \
|
| 85 |
-
--local-dir models/vae/LightVAE
|
| 86 |
-
```
|
| 87 |
|
| 88 |
-
|
| 89 |
|
| 90 |
-
|
| 91 |
-
- `encoder`: `lightvae-lite-h3-encoder-comfy.safetensors`
|
| 92 |
|
| 93 |
-
|
| 94 |
|
| 95 |
-
|
| 96 |
|
| 97 |
-
See the [
|
| 98 |
|
| 99 |
-
## 🐍
|
| 100 |
|
| 101 |
-
|
| 102 |
|
| 103 |
```bash
|
| 104 |
git clone https://github.com/ModelTC/LightVAE.git
|
|
@@ -108,14 +105,9 @@ python -m pip install -r requirements.txt
|
|
| 108 |
hf download lightx2v/LightVAE \
|
| 109 |
lightvae-lite-wan21-decoder.safetensors \
|
| 110 |
--local-dir weights/LightVAE
|
| 111 |
-
```
|
| 112 |
|
| 113 |
-
For example, reconstruct a video with the Wan2.1 Lite decoder and original encoder:
|
| 114 |
-
|
| 115 |
-
```bash
|
| 116 |
hf download Wan-AI/Wan2.1-T2V-1.3B \
|
| 117 |
-
Wan2.1_VAE.pth
|
| 118 |
-
--local-dir weights/Wan2.1
|
| 119 |
|
| 120 |
python infer.py \
|
| 121 |
--model wan21 --variant lite \
|
|
@@ -125,8 +117,8 @@ python infer.py \
|
|
| 125 |
--output outputs/wan21_lite_recon.mp4
|
| 126 |
```
|
| 127 |
|
| 128 |
-
|
| 129 |
|
| 130 |
-
## ⭐ Support
|
| 131 |
|
| 132 |
-
If you find
|
|
|
|
| 13 |
- minimax-h3
|
| 14 |
---
|
| 15 |
|
| 16 |
+
# ⚡ LightVAE
|
| 17 |
|
| 18 |
+
**Towards Compact and Efficient Video Autoencoders**
|
| 19 |
|
| 20 |
+
[💻 Code](https://github.com/ModelTC/LightVAE) · [🧩 ComfyUI](https://github.com/ModelTC/LightVAE/tree/main/comfyui) · [⚡ LightX2V](https://github.com/ModelTC/LightX2V)
|
| 21 |
|
| 22 |
+
LightVAE accelerates video autoencoders while preserving their original latent interfaces. We provide compressed components for **Wan2.1, Wan2.2 and MiniMax-H3**, with native ComfyUI integration for H3.
|
|
|
|
| 23 |
|
| 24 |
## 📦 Checkpoints
|
| 25 |
|
| 26 |
+
| Backbone | Component | Download |
|
| 27 |
+
| :-- | :-- | :-- |
|
| 28 |
+
| Wan2.1 | Pro decoder | [safetensors](./lightvae-pro-wan21-decoder.safetensors) |
|
| 29 |
+
| Wan2.1 | Lite decoder | [safetensors](./lightvae-lite-wan21-decoder.safetensors) |
|
| 30 |
+
| Wan2.2 | Pro decoder | [safetensors](./lightvae-pro-wan22-decoder.safetensors) |
|
| 31 |
+
| Wan2.2 | Lite decoder | [safetensors](./lightvae-lite-wan22-decoder.safetensors) |
|
| 32 |
+
| MiniMax-H3 | Lite decoder | [safetensors](./lightvae-lite-h3-decoder.safetensors) |
|
| 33 |
+
| MiniMax-H3 | Lite encoder | [safetensors](./lightvae-lite-h3-encoder.safetensors) |
|
| 34 |
|
| 35 |
+
**Pro** uses temporal low-rank compression while retaining the original decoder depth. **Lite** additionally prunes blocks for Wan; H3 Lite uses block pruning.
|
| 36 |
|
| 37 |
+
Wan models reuse their original encoders. H3 supports independently replacing the encoder and decoder.
|
| 38 |
|
| 39 |
+
> **Choose the correct Wan VAE:** Wan2.2 **14B** uses the **Wan2.1 VAE**; Wan2.2 **5B TI2V** uses the **Wan2.2 VAE**.
|
| 40 |
|
| 41 |
+
## 🧩 ComfyUI · MiniMax-H3
|
| 42 |
|
| 43 |
+
The **Load H3 LightVAE** node reuses ComfyUI’s native H3 VAE. No LightVAE package installation, TLR or custom Triton kernels are needed. The current node is **H3-only**.
|
| 44 |
|
| 45 |
+
### Install
|
| 46 |
|
| 47 |
+
Update ComfyUI to a version with native MiniMax-H3 support. Run with the Python environment used by ComfyUI:
|
| 48 |
|
| 49 |
```bash
|
| 50 |
cd /path/to/ComfyUI
|
| 51 |
|
| 52 |
git clone https://github.com/ModelTC/LightVAE.git LightVAE
|
|
|
|
| 53 |
python LightVAE/comfyui/install.py --comfy-root .
|
| 54 |
|
| 55 |
+
python -m pip install huggingface_hub
|
| 56 |
hf download lightx2v/LightVAE \
|
| 57 |
+
lightvae-lite-h3-decoder.safetensors \
|
| 58 |
+
lightvae-lite-h3-encoder.safetensors \
|
| 59 |
--local-dir models/vae/LightVAE
|
| 60 |
```
|
| 61 |
|
| 62 |
+
Already cloned? Run `git -C LightVAE pull`, then rerun the installer.
|
| 63 |
|
| 64 |
+
The installer creates `custom_nodes/ComfyUI-LightVAE/` automatically. **Do not clone the entire repository directly into `custom_nodes`.**
|
| 65 |
|
| 66 |
+
The original H3 video VAE is required. If missing:
|
| 67 |
|
| 68 |
+
```bash
|
| 69 |
+
hf download Comfy-Org/MiniMax-H3 \
|
| 70 |
+
vae/minimax_h3_video_vae_fp16.safetensors \
|
| 71 |
+
--local-dir models
|
| 72 |
+
```
|
| 73 |
|
| 74 |
+
### Connect
|
| 75 |
|
| 76 |
+
Restart ComfyUI and search for **Load H3 LightVAE**.
|
| 77 |
|
| 78 |
+
Load the **original H3 video VAE** with the standard **Load VAE** node, then connect it to LightVAE’s **`base_vae`** input.
|
| 79 |
|
| 80 |
+
- **FL2V:** select the Lite decoder and leave `encoder = none`. Keep original first/last-frame encoding unchanged; connect LightVAE to **VAE Decode**.
|
| 81 |
+
- **REF2V:** connect LightVAE to both reference encoding and **VAE Decode**. Select either or both Lite components.
|
| 82 |
+
- **`none`** uses the corresponding original component. Encoder-only replacement is supported with `decoder = none`.
|
| 83 |
|
| 84 |
+
**Keep the audio VAE unchanged.** Load Lite components through **Load H3 LightVAE**, not the standard Load VAE node.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
|
| 86 |
+
The standard H3 files above and previous `*-h3-*-comfy.safetensors` exports are both accepted.
|
| 87 |
|
| 88 |
+
### Workflows
|
|
|
|
| 89 |
|
| 90 |
+
Download a JSON and drag it into ComfyUI:
|
| 91 |
|
| 92 |
+
[FL2V](https://github.com/ModelTC/LightVAE/blob/main/comfyui/workflows/h3_fl2v_lightvae.json) · [Image references](https://github.com/ModelTC/LightVAE/blob/main/comfyui/workflows/h3_ref_image_lightvae.json) · [Video reference](https://github.com/ModelTC/LightVAE/blob/main/comfyui/workflows/h3_ref_video_lightvae.json)
|
| 93 |
|
| 94 |
+
See the [workflow guide](https://github.com/ModelTC/LightVAE/tree/main/comfyui/workflows) for required base models and input preparation.
|
| 95 |
|
| 96 |
+
## 🐍 Python · Video Reconstruction
|
| 97 |
|
| 98 |
+
Example: **original Wan2.1 encoder + LightVAE-Lite decoder**.
|
| 99 |
|
| 100 |
```bash
|
| 101 |
git clone https://github.com/ModelTC/LightVAE.git
|
|
|
|
| 105 |
hf download lightx2v/LightVAE \
|
| 106 |
lightvae-lite-wan21-decoder.safetensors \
|
| 107 |
--local-dir weights/LightVAE
|
|
|
|
| 108 |
|
|
|
|
|
|
|
|
|
|
| 109 |
hf download Wan-AI/Wan2.1-T2V-1.3B \
|
| 110 |
+
Wan2.1_VAE.pth --local-dir weights/Wan2.1
|
|
|
|
| 111 |
|
| 112 |
python infer.py \
|
| 113 |
--model wan21 --variant lite \
|
|
|
|
| 117 |
--output outputs/wan21_lite_recon.mp4
|
| 118 |
```
|
| 119 |
|
| 120 |
+
More examples, benchmarks and visual comparisons are available in [LightVAE](https://github.com/ModelTC/LightVAE). For video generation with LightVAE, see [LightX2V](https://github.com/ModelTC/LightX2V).
|
| 121 |
|
| 122 |
+
## ⭐ Support
|
| 123 |
|
| 124 |
+
If you find this useful, please star [LightVAE](https://github.com/ModelTC/LightVAE) and [LightX2V](https://github.com/ModelTC/LightX2V)!
|