Files changed (1) hide show
  1. README.md +48 -56
README.md CHANGED
@@ -13,92 +13,89 @@ tags:
13
  - minimax-h3
14
  ---
15
 
16
- # LightVAE: Towards Compact and Efficient Video Autoencoders
17
 
18
- LightVAE accelerates pretrained video autoencoders through temporal low-rank compression and block pruning while preserving their latent interfaces.
19
 
20
- This repository provides checkpoints for **Wan2.1, Wan2.2, and MiniMax-H3**, including exports for our ComfyUI custom node.
21
 
22
- **Paper:** *LightVAE: Towards Compact and Efficient Video Autoencoders*
23
- **Code and examples:** [ModelTC/LightVAE](https://github.com/ModelTC/LightVAE)
24
 
25
  ## 📦 Checkpoints
26
 
27
- | Backbone | Variant | Python checkpoint | ComfyUI checkpoint |
28
- | :-- | :-- | :-- | :-- |
29
- | Wan2.1 | Pro decoder | [Download](./lightvae-pro-wan21-decoder.safetensors) | [Download](./lightvae-pro-wan21-decoder-comfy.safetensors) |
30
- | Wan2.1 | Lite decoder | [Download](./lightvae-lite-wan21-decoder.safetensors) | [Download](./lightvae-lite-wan21-decoder-comfy.safetensors) |
31
- | Wan2.2 | Pro decoder | [Download](./lightvae-pro-wan22-decoder.safetensors) | [Download](./lightvae-pro-wan22-decoder-comfy.safetensors) |
32
- | Wan2.2 | Lite decoder | [Download](./lightvae-lite-wan22-decoder.safetensors) | [Download](./lightvae-lite-wan22-decoder-comfy.safetensors) |
33
- | MiniMax-H3 | Lite decoder | [Download](./lightvae-lite-h3-decoder.safetensors) | [Download](./lightvae-lite-h3-decoder-comfy.safetensors) |
34
- | MiniMax-H3 | Lite encoder | [Download](./lightvae-lite-h3-encoder.safetensors) | [Download](./lightvae-lite-h3-encoder-comfy.safetensors) |
35
 
36
- **Pro** retains the original decoder depth for higher reconstruction fidelity. **Lite** applies additional pruning for faster inference.
37
 
38
- Wan models retain their original encoders. MiniMax-H3 can use either its original encoder or the provided Lite encoder.
39
 
40
- > **Wan2.2 model selection:** Wan2.2 **14B** uses the **Wan2.1 VAE**; Wan2.2 **5B TI2V** uses the **Wan2.2 VAE**. Choose the decoder accordingly.
41
 
42
- ## ⚡ Use with ComfyUI
43
 
44
- > **Important:** The `-comfy.safetensors` checkpoints require our **Load LightVAE (TLR)** custom node. Do **not** load them with ComfyUI's standard **Load VAE / VAELoader** node.
45
 
46
- ### 1. Install the node and download weights
47
 
48
- Taking **MiniMax-H3** as an example, run these commands in the **same Python environment used by ComfyUI**:
49
 
50
  ```bash
51
  cd /path/to/ComfyUI
52
 
53
  git clone https://github.com/ModelTC/LightVAE.git LightVAE
54
- python -m pip install ./LightVAE huggingface_hub
55
  python LightVAE/comfyui/install.py --comfy-root .
56
 
 
57
  hf download lightx2v/LightVAE \
58
- lightvae-lite-h3-decoder-comfy.safetensors \
 
59
  --local-dir models/vae/LightVAE
60
  ```
61
 
62
- Restart ComfyUI, then add a **Load LightVAE (TLR)** node.
63
 
64
- ### 2. H3 FL2V: replace the decoder
65
 
66
- In **Load LightVAE (TLR)**, select:
67
 
68
- - `decoder`: `lightvae-lite-h3-decoder-comfy.safetensors`
69
- - `encoder`: `none`
70
- - Leave `precision`, `backend`, and `device` set to `auto`.
71
-
72
- Connect its **VAE output** to your existing **VAE Decode** node.
73
 
74
- Keep the original VAE connections for first/last-frame encoding unchanged. For this decoder-only connection, `base_vae` can remain disconnected.
75
 
76
- The same decoder-only replacement also works for **REF2V**.
77
 
78
- ### 3. H3 REF2V: optionally replace the encoder too
79
 
80
- To also accelerate reference-image/video encoding, download the Lite encoder:
 
 
81
 
82
- ```bash
83
- hf download lightx2v/LightVAE \
84
- lightvae-lite-h3-encoder-comfy.safetensors \
85
- --local-dir models/vae/LightVAE
86
- ```
87
 
88
- Restart ComfyUI and select both H3 Lite files in **Load LightVAE (TLR)**:
89
 
90
- - `decoder`: `lightvae-lite-h3-decoder-comfy.safetensors`
91
- - `encoder`: `lightvae-lite-h3-encoder-comfy.safetensors`
92
 
93
- Connect its **VAE output** to both the **reference-encoding node** and **VAE Decode**. No original video VAE is needed for these connections.
94
 
95
- **Keep the H3 audio VAE and its connections unchanged in all cases.**
96
 
97
- See the [ComfyUI guide and example workflows](https://github.com/ModelTC/LightVAE/tree/main/comfyui) for more details and Wan examples.
98
 
99
- ## 🐍 Quick start with Python
100
 
101
- Use the standard checkpoints **without** the `-comfy` suffix:
102
 
103
  ```bash
104
  git clone https://github.com/ModelTC/LightVAE.git
@@ -108,14 +105,9 @@ python -m pip install -r requirements.txt
108
  hf download lightx2v/LightVAE \
109
  lightvae-lite-wan21-decoder.safetensors \
110
  --local-dir weights/LightVAE
111
- ```
112
 
113
- For example, reconstruct a video with the Wan2.1 Lite decoder and original encoder:
114
-
115
- ```bash
116
  hf download Wan-AI/Wan2.1-T2V-1.3B \
117
- Wan2.1_VAE.pth \
118
- --local-dir weights/Wan2.1
119
 
120
  python infer.py \
121
  --model wan21 --variant lite \
@@ -125,8 +117,8 @@ python infer.py \
125
  --output outputs/wan21_lite_recon.mp4
126
  ```
127
 
128
- See the [GitHub repository](https://github.com/ModelTC/LightVAE) for Wan2.2 and MiniMax-H3 examples, benchmarks, and visual comparisons.
129
 
130
- ## ⭐ Support Us
131
 
132
- If you find LightVAE useful, please consider starring [LightVAE](https://github.com/ModelTC/LightVAE) and [LightX2V](https://github.com/ModelTC/LightX2V). Your support helps us keep improving efficient video generation!
 
13
  - minimax-h3
14
  ---
15
 
16
+ # ⚡ LightVAE
17
 
18
+ **Towards Compact and Efficient Video Autoencoders**
19
 
20
+ [💻 Code](https://github.com/ModelTC/LightVAE) · [🧩 ComfyUI](https://github.com/ModelTC/LightVAE/tree/main/comfyui) · [⚡ LightX2V](https://github.com/ModelTC/LightX2V)
21
 
22
+ LightVAE accelerates video autoencoders while preserving their original latent interfaces. We provide compressed components for **Wan2.1, Wan2.2 and MiniMax-H3**, with native ComfyUI integration for H3.
 
23
 
24
  ## 📦 Checkpoints
25
 
26
+ | Backbone | Component | Download |
27
+ | :-- | :-- | :-- |
28
+ | Wan2.1 | Pro decoder | [safetensors](./lightvae-pro-wan21-decoder.safetensors) |
29
+ | Wan2.1 | Lite decoder | [safetensors](./lightvae-lite-wan21-decoder.safetensors) |
30
+ | Wan2.2 | Pro decoder | [safetensors](./lightvae-pro-wan22-decoder.safetensors) |
31
+ | Wan2.2 | Lite decoder | [safetensors](./lightvae-lite-wan22-decoder.safetensors) |
32
+ | MiniMax-H3 | Lite decoder | [safetensors](./lightvae-lite-h3-decoder.safetensors) |
33
+ | MiniMax-H3 | Lite encoder | [safetensors](./lightvae-lite-h3-encoder.safetensors) |
34
 
35
+ **Pro** uses temporal low-rank compression while retaining the original decoder depth. **Lite** additionally prunes blocks for Wan; H3 Lite uses block pruning.
36
 
37
+ Wan models reuse their original encoders. H3 supports independently replacing the encoder and decoder.
38
 
39
+ > **Choose the correct Wan VAE:** Wan2.2 **14B** uses the **Wan2.1 VAE**; Wan2.2 **5B TI2V** uses the **Wan2.2 VAE**.
40
 
41
+ ## 🧩 ComfyUI · MiniMax-H3
42
 
43
+ The **Load H3 LightVAE** node reuses ComfyUI’s native H3 VAE. No LightVAE package installation, TLR or custom Triton kernels are needed. The current node is **H3-only**.
44
 
45
+ ### Install
46
 
47
+ Update ComfyUI to a version with native MiniMax-H3 support. Run with the Python environment used by ComfyUI:
48
 
49
  ```bash
50
  cd /path/to/ComfyUI
51
 
52
  git clone https://github.com/ModelTC/LightVAE.git LightVAE
 
53
  python LightVAE/comfyui/install.py --comfy-root .
54
 
55
+ python -m pip install huggingface_hub
56
  hf download lightx2v/LightVAE \
57
+ lightvae-lite-h3-decoder.safetensors \
58
+ lightvae-lite-h3-encoder.safetensors \
59
  --local-dir models/vae/LightVAE
60
  ```
61
 
62
+ Already cloned? Run `git -C LightVAE pull`, then rerun the installer.
63
 
64
+ The installer creates `custom_nodes/ComfyUI-LightVAE/` automatically. **Do not clone the entire repository directly into `custom_nodes`.**
65
 
66
+ The original H3 video VAE is required. If missing:
67
 
68
+ ```bash
69
+ hf download Comfy-Org/MiniMax-H3 \
70
+ vae/minimax_h3_video_vae_fp16.safetensors \
71
+ --local-dir models
72
+ ```
73
 
74
+ ### Connect
75
 
76
+ Restart ComfyUI and search for **Load H3 LightVAE**.
77
 
78
+ Load the **original H3 video VAE** with the standard **Load VAE** node, then connect it to LightVAE’s **`base_vae`** input.
79
 
80
+ - **FL2V:** select the Lite decoder and leave `encoder = none`. Keep original first/last-frame encoding unchanged; connect LightVAE to **VAE Decode**.
81
+ - **REF2V:** connect LightVAE to both reference encoding and **VAE Decode**. Select either or both Lite components.
82
+ - **`none`** uses the corresponding original component. Encoder-only replacement is supported with `decoder = none`.
83
 
84
+ **Keep the audio VAE unchanged.** Load Lite components through **Load H3 LightVAE**, not the standard Load VAE node.
 
 
 
 
85
 
86
+ The standard H3 files above and previous `*-h3-*-comfy.safetensors` exports are both accepted.
87
 
88
+ ### Workflows
 
89
 
90
+ Download a JSON and drag it into ComfyUI:
91
 
92
+ [FL2V](https://github.com/ModelTC/LightVAE/blob/main/comfyui/workflows/h3_fl2v_lightvae.json) · [Image references](https://github.com/ModelTC/LightVAE/blob/main/comfyui/workflows/h3_ref_image_lightvae.json) · [Video reference](https://github.com/ModelTC/LightVAE/blob/main/comfyui/workflows/h3_ref_video_lightvae.json)
93
 
94
+ See the [workflow guide](https://github.com/ModelTC/LightVAE/tree/main/comfyui/workflows) for required base models and input preparation.
95
 
96
+ ## 🐍 Python · Video Reconstruction
97
 
98
+ Example: **original Wan2.1 encoder + LightVAE-Lite decoder**.
99
 
100
  ```bash
101
  git clone https://github.com/ModelTC/LightVAE.git
 
105
  hf download lightx2v/LightVAE \
106
  lightvae-lite-wan21-decoder.safetensors \
107
  --local-dir weights/LightVAE
 
108
 
 
 
 
109
  hf download Wan-AI/Wan2.1-T2V-1.3B \
110
+ Wan2.1_VAE.pth --local-dir weights/Wan2.1
 
111
 
112
  python infer.py \
113
  --model wan21 --variant lite \
 
117
  --output outputs/wan21_lite_recon.mp4
118
  ```
119
 
120
+ More examples, benchmarks and visual comparisons are available in [LightVAE](https://github.com/ModelTC/LightVAE). For video generation with LightVAE, see [LightX2V](https://github.com/ModelTC/LightX2V).
121
 
122
+ ## ⭐ Support
123
 
124
+ If you find this useful, please star [LightVAE](https://github.com/ModelTC/LightVAE) and [LightX2V](https://github.com/ModelTC/LightX2V)!