Update README.md

README.md

@@ -6,46 +6,26 @@ frameworks:
 - ""
 base_model_relation: quantized
 ---
-# MiniMax-H3-NF4
-
-The **NF4 quantized version** of the MiniMax-H3 multimodal audio-video generation model (4-bit quantized via `bitsandbytes`), designed for use with [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio). Enables joint "text/image/video/audio → video + audio" generation on machines with limited GPU memory or system RAM.
-
-## File Overview
-
-| File | Size | Function | Shared? |
-|---|---|---|---|
-| `minimax-h3-fl2va-nf4.safetensors` | ~16 GB | DiT backbone for **FL2VA** task (text / start-end keyframes → video+audio) | FL2VA only |
-| `minimax-h3-ref2va-nf4.safetensors` | ~16 GB | DiT backbone for **Ref2VA** task (reference image/video/audio → video+audio) | Ref2VA only |
-| `minimax-h3-text-encoder-nf4.safetensors` | ~15 GB | Qwen3-VL text/visual encoder | Shared across tasks |
-| `video_vae_nf4.safetensors` | ~1.6 GB | Video VAE decoder | Shared across tasks |
-| `audio_vae_nf4.safetensors` | ~271 MB | Audio VAE decoder | Shared across tasks |
-
-> Note: Choose one DiT model depending on the task; the other three components (text_encoder / video_vae / audio_vae) are shared between both tasks. The framework automatically identifies component types and applies appropriate quantization configurations (including bf16 fallback for a few quantization-sensitive layers) based on file hashes—no manual configuration required.
-
-## System Requirements
 
-- CUDA GPU (NF4 dequantization relies on `bitsandbytes` CUDA kernels)
-- Processor and tokenizer must be obtained from the original repository `MiniMax/MiniMax-H3` (see `processor_config` below)
+# MiniMax-H3-NF4
 
-### Install DiffSynth-Studio
+This model is the **NF4 quantized version** of the video generation model [MiniMax-H3](https://modelscope.cn/models/MiniMax/MiniMax-H3). It utilizes the `bitsandbytes` 4-bit quantization scheme and is designed to be used with [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), enabling model inference on devices with limited VRAM and RAM.
 
-Install from source (recommended, ensures latest MiniMax-H3 support), including NF4 quantization dependencies:
+## Environment Setup
 
-```bash
+```shell
 git clone https://github.com/modelscope/DiffSynth-Studio.git
 cd DiffSynth-Studio
-pip install -e ".[quant]"
+pip install -e ".[all]"
 ```
 
-## Usage (Disk Offload, Low VRAM & RAM)
-
-Weights remain on disk and are streamed into GPU layer-by-layer during inference, minimizing VRAM usage. **Text-to-video+audio (t2v) can run with as little as ~6 GB VRAM.**
+## Inference Code
 
-> `vram_limit` (in GB) sets the VRAM threshold—lower values reduce memory usage at the cost of speed.
+### Enable VRAM Management
 
-> If you have sufficient CPU RAM, set `offload_device` / `offload_dtype` to `"cpu"` / `torch.bfloat16` (i.e., CPU offload). This keeps weights in main memory instead of reading from disk, resulting in faster performance; all other code remains unchanged.
+Run the following code to perform inference using DiffSynth-Studio. VRAM management will be automatically enabled. The actual VRAM usage depends on the available VRAM on your GPU; a minimum of 10GB VRAM is required to run.
 
-### FL2VA — Text / Start-End Keyframes → Video + Audio
+#### FL2VA (Text-to-Video/Audio):
 
 ```python
 import torch
@@ -54,7 +34,6 @@ from diffsynth.utils.data.audio_video import write_video_audio
 from modelscope import dataset_snapshot_download
 from PIL import Image
 
-
 vram_config = {
     "offload_dtype": "disk",
     "offload_device": "disk",
@@ -78,7 +57,6 @@ pipe = MiniMaxH3Pipeline.from_pretrained(
     vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
 )
 
-# Text -> Video + Audio
 prompt = "A girl is very happy, she is speaking in english: “I enjoy working with Diffsynth-Studio, it's a perfect framework.”"
 video, audio = pipe(
     prompt=prompt,
@@ -88,33 +66,13 @@ write_video_audio(
     video=video, audio=audio,
     output_path="t2va.mp4", fps=24, audio_sample_rate=32000,
 )
-
-# Text + First Frame + Last Frame -> Video + Audio
-dataset_snapshot_download(dataset_id="DiffSynth-Studio/diffsynth_example_dataset", local_dir="data/diffsynth_example_dataset", allow_file_pattern="minimax_h3/MiniMax-H3-FL2VA/*")
-first_frame = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/first.png")
-last_frame = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/last.png")
-prompt = "A short indoor drama scene of a family argument, vertical video format with short-form video aesthetics, realistic live-action performance, Chinese household or small restaurant interior setting, warm lighting, red decorations and calligraphy scrolls in the background, shallow depth of field, intense emotions, fast-paced editing. Performance requirements: authentic short-video acting style, no exaggerated theatrical tone. The man speaks with anger, grievance, and urgent rebuttal, saying 'What exactly do you want?' The middle-aged woman speaks sharply, assertively, and aggressively demanding, saying 'You must pay up!' There should be strong confrontation between them, escalating in intensity. Visual style: vertical 9:16 aspect ratio, smartphone short-video look, realistic live-action footage, shallow depth of field, warm indoor lighting, mostly medium and close-up shots, frequent shot-reverse-shot editing, background should remain everyday and realistic—no sci-fi, no historical costumes, no animation-like visuals. No subtitles, text, platform watermarks, or overlays should appear in the画面."
-video, audio = pipe(
-    prompt=prompt,
-    height=832, width=480, num_frames=124, num_inference_steps=50, seed=0,
-    keyframes=[first_frame, last_frame], keyframe_indices=[0, -1],
-)
-write_video_audio(
-    video=video, audio=audio,
-    output_path="fl2va.mp4", fps=24, audio_sample_rate=32000,
-)
 ```
 
-### Ref2VA — Reference Image/Video/Audio → Video + Audio
+#### Ref2VA (Reference-to-Video/Audio):
 
-Four types of references are supported, which can be combined within a single list (`video` is silent; for videos with sound, use `video_audio`):
+<details>
 
-```python
-{"type": "image",       "image": PIL.Image}
-{"type": "video",       "video": list[PIL.Image]}                            # silent
-{"type": "audio",       "audio": Tensor[C, L], "sample_rate": int}
-{"type": "video_audio", "video": list[PIL.Image], "audio": Tensor[C, L], "sample_rate": int}
-```
+<summary>Expand Code</summary>
 
 ```python
 import torch
@@ -169,7 +127,7 @@ pipe = MiniMaxH3Pipeline.from_pretrained(
 # Text + Reference Image -> Video + Audio
 dataset_snapshot_download(dataset_id="DiffSynth-Studio/diffsynth_example_dataset", local_dir="data/diffsynth_example_dataset", allow_file_pattern="minimax_h3/MiniMax-H3-Ref2VA/*")
 ref_image = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/0.png").convert("RGB")
-prompt = "A website page, UI design of a web page, web animation. The video demonstrates a smooth scrolling effect downward. A highly dynamic and energetic product official website-style landing page UI/UX demonstration video, with the main focus being product image 1. The layout features bold, slanted, oversized sans-serif typography. The background includes fast-paced dynamic lighting effects intertwined with dark carbon fiber or sporty breathable mesh textures in motion. The video showcases a tightly paced, powerful downward scroll effect, along with strong visual interactions such as significant zoom-in and color inversion when hovering over UI elements."
+prompt = "A website page, website UI design, website animation, video showing smooth webpage scrolling effect. A highly explosive and dynamic product official website style product landing page UI/UX demo video, the core display subject is product image 1. The page uses bold, powerful, tilted oversized sans-serif fonts for flamboyant typography. The background features dynamic light and shadow with extreme speed sense, dark carbon fiber or sports breathable mesh textures interweaving and changing. The video shows a tight-paced, powerful webpage downward scrolling effect, as well as strong visual zoom and color inversion UI interaction actions when hovering the mouse."
 video, audio = pipe(
     prompt=prompt,
     height=480, width=832, num_frames=124, num_inference_steps=50, seed=42,
@@ -198,20 +156,137 @@ write_video_audio(
 )
 ```
 
-## Common Parameters
+</details>
+
+### Extreme Hardware Optimization
 
-- `height` / `width`: Resolution, e.g., `480x832` (landscape) or `832x480` (portrait).
-- `num_frames`: Number of frames, must satisfy `num_frames % 17 == 5` (e.g., 124).
-- `num_inference_steps`: Denoising steps, example uses 50.
-- `keyframes` / `keyframe_indices`: FL2VA control for start and end frames (`[0, -1]` means first and last frame).
-- `references`: Ref2VA reference list, elements are `{"type": "image|video|audio|video_audio", ...}`.
-- Output: `write_video_audio(video, audio, output_path, fps=24, audio_sample_rate=32000)`.
+If your computing device has extremely limited performance, we support enabling direct disk-to-VRAM loading. With this configuration, tensors in the model are loaded from disk to VRAM one by one according to the computation order. This allows the model to run with only 10GB of RAM:
 
-## Example Scripts
+```diff
+vram_config = {
++   "offload_dtype": "disk",
++   "offload_device": "disk",
++   "onload_dtype": "disk",
++   "onload_device": "disk",
++   "preparing_dtype": "disk",
++   "preparing_device": "disk",
++   "computation_dtype": torch.bfloat16,
++   "computation_device": "cuda",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=...,
+    processor_config=...,
++   vram_limit=0,
+)
+```
+
+We also support running model inference on Mac M-series chips, although this is not recommended:
+
+```diff
+vram_config = {
++   "offload_dtype": "disk",
++   "offload_device": "disk",
++   "onload_dtype": "disk",
++   "onload_device": "disk",
++   "preparing_dtype": "disk",
++   "preparing_device": "disk",
++   "computation_dtype": torch.bfloat16,
++   "computation_device": "mps",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="mps",
+    model_configs=...,
+    processor_config=...,
++   vram_limit=0,
+)
+```
+
+## Training Code
+
+This quantized model supports LoRA training. Please follow the steps below to start the training program.
+
+Download the sample dataset:
+
+```shell
+modelscope download --dataset DiffSynth-Studio/diffsynth_example_dataset --include "minimax_h3/MiniMax-H3-FL2VA/*" --local_dir ./data/diffsynth_example_dataset
+```
+
+**Training configuration suitable for Data Center GPUs (e.g., Nvidia H20):** Run the following script to start the LoRA training program. Requires 48GB VRAM.
+
+```shell
+accelerate launch examples/minimax_h3/model_training/train.py \
+  --dataset_base_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA \
+  --dataset_metadata_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/metadata.csv \
+  --data_file_keys "video,input_audio" \
+  --extra_inputs "input_audio" \
+  --height 480 \
+  --width 832 \
+  --num_frames 124 \
+  --dataset_repeat 100 \
+  --model_id_with_origin_paths "DiffSynth-Studio/MiniMax-H3-NF4:minimax-h3-text-encoder-nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:minimax-h3-fl2va-nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:video_vae_nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:audio_vae_nf4.safetensors" \
+  --learning_rate 1e-4 \
+  --num_epochs 5 \
+  --remove_prefix_in_ckpt "pipe.dit." \
+  --output_path "./models/train/MiniMax-H3-T2VA-nf4" \
+  --lora_base_model "dit" \
+  --lora_target_modules "qkv_proj,out_proj" \
+  --lora_rank 32 \
+  --use_gradient_checkpointing \
+  --find_unused_parameters
+```
+
+**Training configuration suitable for Consumer GPUs (e.g., Nvidia RTX 4090):** Run the following scripts to start two-stage split training with gradient checkpointing offload. Requires 24GB VRAM.
+
+```shell
+accelerate launch examples/minimax_h3/model_training/train.py \
+  --dataset_base_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA \
+  --dataset_metadata_path data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/metadata.csv \
+  --data_file_keys "video,input_audio" \
+  --extra_inputs "input_audio" \
+  --height 480 \
+  --width 832 \
+  --num_frames 124 \
+  --dataset_repeat 1 \
+  --model_id_with_origin_paths "DiffSynth-Studio/MiniMax-H3-NF4:minimax-h3-text-encoder-nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:video_vae_nf4.safetensors,DiffSynth-Studio/MiniMax-H3-NF4:audio_vae_nf4.safetensors" \
+  --learning_rate 1e-4 \
+  --num_epochs 1 \
+  --remove_prefix_in_ckpt "pipe.dit." \
+  --output_path "./models/train/MiniMax-H3-T2VA-nf4-split-cache" \
+  --lora_base_model "dit" \
+  --lora_target_modules "qkv_proj,out_proj" \
+  --lora_rank 32 \
+  --use_gradient_checkpointing \
+  --use_gradient_checkpointing_offload \
+  --task "sft:data_process"
+
+accelerate launch examples/minimax_h3/model_training/train.py \
+  --dataset_base_path "./models/train/MiniMax-H3-T2VA-nf4-split-cache" \
+  --data_file_keys "video,input_audio" \
+  --extra_inputs "input_audio" \
+  --height 480 \
+  --width 832 \
+  --num_frames 124 \
+  --dataset_repeat 100 \
+  --model_id_with_origin_paths "DiffSynth-Studio/MiniMax-H3-NF4:minimax-h3-fl2va-nf4.safetensors" \
+  --learning_rate 1e-4 \
+  --num_epochs 5 \
+  --remove_prefix_in_ckpt "pipe.dit." \
+  --output_path "./models/train/MiniMax-H3-T2VA-nf4" \
+  --lora_base_model "dit" \
+  --lora_target_modules "qkv_proj,out_proj" \
+  --lora_rank 32 \
+  --use_gradient_checkpointing \
+  --use_gradient_checkpointing_offload \
+  --find_unused_parameters \
+  --task "sft:train"
+```
 
-Fully runnable scripts in the repository (examples in this README are derived from these):
+## References
 
-- `examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-FL2VA.py`
-- `examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-Ref2VA.py`
-- `examples/minimax_h3/model_inference/MiniMax-H3-NF4-FL2VA.py` (CPU offload)
-- `examples/minimax_h3/model_inference/MiniMax-H3-NF4-Ref2VA.py` (CPU offload)
+* DiffSynth-Studio Documentation: [Minimax-H3](https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/MiniMax-H3.html)
+* DiffSynth-Studio Documentation: [VRAM Management](https://diffsynth-studio-doc.readthedocs.io/en/latest/Pipeline_Usage/VRAM_management.html)
+* DiffSynth-Studio Documentation: [Two-Stage Split Training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Training/Split_Training.html)
+* DiffSynth-Studio Documentation: [Low VRAM Training](https://diffsynth-studio-doc.readthedocs.io/en/latest/Pipeline_Usage/Model_Training.html#low-vram-training)