Upload folder using huggingface_hub

.gitattributes

@@ -1,35 +1,60 @@
 *.7z filter=lfs diff=lfs merge=lfs -text
 *.arrow filter=lfs diff=lfs merge=lfs -text
 *.bin filter=lfs diff=lfs merge=lfs -text
+*.bin.* filter=lfs diff=lfs merge=lfs -text
 *.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
 *.ftz filter=lfs diff=lfs merge=lfs -text
 *.gz filter=lfs diff=lfs merge=lfs -text
 *.h5 filter=lfs diff=lfs merge=lfs -text
 *.joblib filter=lfs diff=lfs merge=lfs -text
 *.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
 *.model filter=lfs diff=lfs merge=lfs -text
 *.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
 *.onnx filter=lfs diff=lfs merge=lfs -text
 *.ot filter=lfs diff=lfs merge=lfs -text
 *.parquet filter=lfs diff=lfs merge=lfs -text
 *.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
 *.pth filter=lfs diff=lfs merge=lfs -text
 *.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
 saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
 *.tflite filter=lfs diff=lfs merge=lfs -text
 *.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
 *.xz filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
+*.zstandard filter=lfs diff=lfs merge=lfs -text
+*.tfevents* filter=lfs diff=lfs merge=lfs -text
+*.db* filter=lfs diff=lfs merge=lfs -text
+*.ark* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*data* filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.meta filter=lfs diff=lfs merge=lfs -text
+**/*ckpt*.index filter=lfs diff=lfs merge=lfs -text
+ 
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.gguf* filter=lfs diff=lfs merge=lfs -text
+*.ggml filter=lfs diff=lfs merge=lfs -text
+*.llamafile* filter=lfs diff=lfs merge=lfs -text
+*.pt2 filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+
+./minimax-h3-ref2va-nf4.safetensors filter=lfs diff=lfs merge=lfs -text
+./minimax-h3-text-encoder-nf4.safetensors filter=lfs diff=lfs merge=lfs -text
+./minimax-h3-fl2va-nf4.safetensors filter=lfs diff=lfs merge=lfs -text
+"/video_vae_nf4.safetensors" filter=lfs diff=lfs merge=lfs -text
+"/audio_vae_nf4.safetensors" filter=lfs diff=lfs merge=lfs -text
+"/minimax-h3-fl2va-nf4.safetensors" filter=lfs diff=lfs merge=lfs -text
+"/minimax-h3-ref2va-nf4.safetensors" filter=lfs diff=lfs merge=lfs -text
+"/minimax-h3-text-encoder-nf4.safetensors" filter=lfs diff=lfs merge=lfs -textaudio_vae_nf4.safetensors filter=lfs diff=lfs merge=lfs -text
+minimax-h3-fl2va-nf4.safetensors filter=lfs diff=lfs merge=lfs -text
+minimax-h3-ref2va-nf4.safetensors filter=lfs diff=lfs merge=lfs -text
+minimax-h3-text-encoder-nf4.safetensors filter=lfs diff=lfs merge=lfs -text
+video_vae_nf4.safetensors filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,212 @@
+---
+license: apache-2.0
+---
+# MiniMax-H3-NF4
+
+The **NF4 quantized version** of the MiniMax-H3 multimodal audio-video generation model (4-bit quantized via `bitsandbytes`), designed for use with [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio). Enables joint "text/image/video/audio → video + audio" generation on machines with limited GPU memory or system RAM.
+
+## File Overview
+
+| File | Size | Function | Shared? |
+|---|---|---|---|
+| `minimax-h3-fl2va-nf4.safetensors` | ~16 GB | DiT backbone for **FL2VA** task (text / start-end keyframes → video+audio) | FL2VA only |
+| `minimax-h3-ref2va-nf4.safetensors` | ~16 GB | DiT backbone for **Ref2VA** task (reference image/video/audio → video+audio) | Ref2VA only |
+| `minimax-h3-text-encoder-nf4.safetensors` | ~15 GB | Qwen3-VL text/visual encoder | Shared across tasks |
+| `video_vae_nf4.safetensors` | ~1.6 GB | Video VAE decoder | Shared across tasks |
+| `audio_vae_nf4.safetensors` | ~271 MB | Audio VAE decoder | Shared across tasks |
+
+> Note: Choose one DiT model depending on the task; the other three components (text_encoder / video_vae / audio_vae) are shared between both tasks. The framework automatically identifies component types and applies appropriate quantization configurations (including bf16 fallback for a few quantization-sensitive layers) based on file hashes—no manual configuration required.
+
+## System Requirements
+
+- CUDA GPU (NF4 dequantization relies on `bitsandbytes` CUDA kernels)
+- Processor and tokenizer must be obtained from the original repository `MiniMax/MiniMax-H3` (see `processor_config` below)
+
+### Install DiffSynth-Studio
+
+Install from source (recommended, ensures latest MiniMax-H3 support), including NF4 quantization dependencies:
+
+```bash
+git clone https://github.com/modelscope/DiffSynth-Studio.git
+cd DiffSynth-Studio
+pip install -e ".[quant]"
+```
+
+## Usage (Disk Offload, Low VRAM & RAM)
+
+Weights remain on disk and are streamed into GPU layer-by-layer during inference, minimizing VRAM usage. **Text-to-video+audio (t2v) can run with as little as ~6 GB VRAM.**
+
+> `vram_limit` (in GB) sets the VRAM threshold—lower values reduce memory usage at the cost of speed.
+
+> If you have sufficient CPU RAM, set `offload_device` / `offload_dtype` to `"cpu"` / `torch.bfloat16` (i.e., CPU offload). This keeps weights in main memory instead of reading from disk, resulting in faster performance; all other code remains unchanged.
+
+### FL2VA — Text / Start-End Keyframes → Video + Audio
+
+```python
+import torch
+from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
+from diffsynth.utils.data.audio_video import write_video_audio
+from modelscope import dataset_snapshot_download
+from PIL import Image
+```
+
+vram_config = {
+    "offload_dtype": "disk",
+    "offload_device": "disk",
+    "onload_dtype": torch.bfloat16,
+    "onload_device": "cpu",
+    "preparing_dtype": torch.bfloat16,
+    "preparing_device": "cuda",
+    "computation_dtype": torch.bfloat16,
+    "computation_device": "cuda",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-text-encoder-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
+    ],
+    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
+    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
+)
+
+# Text -> Video + Audio
+prompt = "A girl is very happy, she is speaking in english: “I enjoy working with Diffsynth-Studio, it's a perfect framework.”"
+video, audio = pipe(
+    prompt=prompt,
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=0,
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="t2va.mp4", fps=24, audio_sample_rate=32000,
+)
+
+# Text + First Frame + Last Frame -> Video + Audio
+dataset_snapshot_download(dataset_id="DiffSynth-Studio/diffsynth_example_dataset", local_dir="data/diffsynth_example_dataset", allow_file_pattern="minimax_h3/MiniMax-H3-FL2VA/*")
+first_frame = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/first.png")
+last_frame = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/last.png")
+prompt = "A short indoor drama scene of a family argument, vertical video format with short-form video aesthetics, realistic live-action performance, Chinese household or small restaurant interior setting, warm lighting, red decorations and calligraphy scrolls in the background, shallow depth of field, intense emotions, fast-paced editing. Performance requirements: authentic short-video acting style, no exaggerated theatrical tone. The man speaks with anger, grievance, and urgent rebuttal, saying 'What exactly do you want?' The middle-aged woman speaks sharply, assertively, and aggressively demanding, saying 'You must pay up!' There should be strong confrontation between them, escalating in intensity. Visual style: vertical 9:16 aspect ratio, smartphone short-video look, realistic live-action footage, shallow depth of field, warm indoor lighting, mostly medium and close-up shots, frequent shot-reverse-shot editing, background should remain everyday and realistic—no sci-fi, no historical costumes, no animation-like visuals. No subtitles, text, platform watermarks, or overlays should appear in the画面."
+video, audio = pipe(
+    prompt=prompt,
+    height=832, width=480, num_frames=124, num_inference_steps=50, seed=0,
+    keyframes=[first_frame, last_frame], keyframe_indices=[0, -1],
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="fl2va.mp4", fps=24, audio_sample_rate=32000,
+)
+```
+
+### Ref2VA — Reference Image/Video/Audio → Video + Audio
+
+Four types of references are supported, which can be combined within a single list (`video` is silent; for videos with sound, use `video_audio`):
+
+```python
+{"type": "image",       "image": PIL.Image}
+{"type": "video",       "video": list[PIL.Image]}                            # silent
+{"type": "audio",       "audio": Tensor[C, L], "sample_rate": int}
+{"type": "video_audio", "video": list[PIL.Image], "audio": Tensor[C, L], "sample_rate": int}
+```
+
+```python
+import torch
+from PIL import Image
+from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
+from diffsynth.utils.data.audio_video import write_video_audio
+from diffsynth.utils.data.audio import read_audio
+from diffsynth.utils.data import VideoData
+from modelscope import dataset_snapshot_download
+
+def align_frame_count(frame_count):
+    current = max(int(frame_count), 1)
+    while current % 17 != 5:
+        current += 1
+    return current
+
+def read_video_with_fps(path, num_out_frames, height, width, fps=24):
+    video = VideoData(path, height=height, width=width)
+    frames = video.raw_data()
+    src_fps = float(video.data.reader.get_meta_data()["fps"])
+    out = []
+    for k in range(num_out_frames):
+        idx = int(round(k * src_fps / fps))
+        if idx >= len(frames):
+            break
+        out.append(frames[idx])
+    return out
+
+vram_config = {
+    "offload_dtype": "disk",
+    "offload_device": "disk",
+    "onload_dtype": torch.bfloat16,
+    "onload_device": "cpu",
+    "preparing_dtype": torch.bfloat16,
+    "preparing_device": "cuda",
+    "computation_dtype": torch.bfloat16,
+    "computation_device": "cuda",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-ref2va-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-text-encoder-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
+    ],
+    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/processor/"),
+    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 5,
+)
+
+# Text + Reference Image -> Video + Audio
+dataset_snapshot_download(dataset_id="DiffSynth-Studio/diffsynth_example_dataset", local_dir="data/diffsynth_example_dataset", allow_file_pattern="minimax_h3/MiniMax-H3-Ref2VA/*")
+ref_image = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/0.png").convert("RGB")
+prompt = "A website page, UI design of a web page, web animation. The video demonstrates a smooth scrolling effect downward. A highly dynamic and energetic product official website-style landing page UI/UX demonstration video, with the main focus being product image 1. The layout features bold, slanted, oversized sans-serif typography. The background includes fast-paced dynamic lighting effects intertwined with dark carbon fiber or sporty breathable mesh textures in motion. The video showcases a tightly paced, powerful downward scroll effect, along with strong visual interactions such as significant zoom-in and color inversion when hovering over UI elements."
+video, audio = pipe(
+    prompt=prompt,
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=42,
+    references=[{"type": "image", "image": Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/0.png").convert("RGB")}]
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="ti2va.mp4", fps=24, audio_sample_rate=32000,
+)
+
+# Text + Reference Audio + Reference Video -> Video + Audio
+ref_video = read_video_with_fps("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/video.mp4", 124, 480, 832)
+ref_audio, sample_rate = read_audio("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/voice.mp3", duration=len(ref_video) / 24, resample=True, resample_rate=pipe.audio_vae.sample_rate)
+prompt = "subject_definitions:\n<Subject 1> is the young man with short wavy blonde hair, wearing a bright pink suit jacket, matching pink trousers, an unbuttoned white shirt, and silver rings, holding a small black lamb in his arms in <Video 1>.\n<Video 1> is the source video for the editing task.\n<Audio 1> is the synchronized audio track of <Video 1>, providing the background music.\n<Audio 2> is the voice timbre reference for <Subject 1>'s voice, containing a spoken male voiceover.\n\nsummary:\n[video editing + audio reference + audio reuse] The target video is an edited version of <Video 1>. <Subject 1>, wearing a bright pink suit and holding a black lamb, stands in a grassy field with other white lambs in the background. The edit animates <Subject 1>'s face to speak the user-provided dialogue. <Audio 1> is partially reused as the continuous background music, while the target references the calm male voice timbre of <Audio 2> for <Subject 1>'s spoken lines.\n\nretention_analysis:\n<Subject 1> (appears in [Shot 1]): fully_preserved - the man retains his identity, wavy blonde hair, pink suit, white shirt, accessories, and the black lamb he holds, with his mouth newly animated to speak.\n<Video 1> (source video editing): fully_preserved - the original camera framing, warm golden hour lighting, grassy hill setting, and background white lambs are maintained while the central character is edited.\n<Audio 1>: partially_copy - the atmospheric background music from <Audio 1> is reused in the target video, mixed beneath the newly added spoken dialogue.\n<Audio 2>: reference - the target audio references the male voice timbre from <Audio 2> to generate <Subject 1>'s spoken dialogue.\n\ndetailed_description:\nThe target video is in realistic photographic style.\n[Shot 1] The shot begins from the source <Video 1>, showing <Subject 1>, a young man with short wavy blonde hair, wearing a bright pink suit jacket, matching pink trousers, and a casually unbuttoned white shirt. He stands confidently in a sunlit green pasture, gently holding a small black lamb securely in his arms. The warm, golden hour lighting casts soft shadows across his face and the bright pink fabric of his suit. Behind him, several white lambs stand and graze on the rolling grassy hill against a clear, pale blue sky. The atmospheric background music from <Audio 1> plays continuously throughout the scene. <Subject 1> physically speaks, his mouth movements naturally syncing to the new dialogue, with his voice timbre referencing the calm male delivery from <Audio 2>. Looking thoughtfully forward, <Subject 1> (S1) speaks softly, <d>[English] Follow the wind, live free.</d> As he delivers the line, he subtly shifts his weight, cradling the resting black lamb while the camera slowly pushes in. <Subject 1> (S1) continues his thought, <d>[English] Leave worries behind, enjoy the moment.</d> Exactly as his voice stops, his lips meet in a relaxed, peaceful smile, and his jaw ceases speaking motion. He then turns his gaze slightly away toward the horizon, gently stroking the black lamb's fleece with his fingers as the camera holds on this tranquil, sunlit state through the end of the video.\n\noverall_soundscape:\nThe soundscape consists of the continuous, atmospheric background music from <Audio 1>, overlaid with the clear, calm male dialogue spoken by the main character, referencing the voice timbre of <Audio 2>.\n\nnon_diegetic_music:\nThe atmospheric, sustained background music from <Audio 1> is reused as the continuous score, playing quietly beneath the spoken dialogue."
+video, audio = pipe(
+    prompt=prompt,
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=42,
+    references=[
+        {"type": "video", "video": ref_video},
+        {"type": "audio", "audio": ref_audio, "sample_rate": sample_rate},
+    ],
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="tav2va.mp4", fps=24, audio_sample_rate=32000,
+)
+```
+
+## Common Parameters
+
+- `height` / `width`: Resolution, e.g., `480x832` (landscape) or `832x480` (portrait).
+- `num_frames`: Number of frames, must satisfy `num_frames % 17 == 5` (e.g., 124).
+- `num_inference_steps`: Denoising steps, example uses 50.
+- `keyframes` / `keyframe_indices`: FL2VA control for start and end frames (`[0, -1]` means first and last frame).
+- `references`: Ref2VA reference list, elements are `{"type": "image|video|audio|video_audio", ...}`.
+- Output: `write_video_audio(video, audio, output_path, fps=24, audio_sample_rate=32000)`.
+
+## Example Scripts
+
+Fully runnable scripts in the repository (examples in this README are derived from these):
+
+- `examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-FL2VA.py`
+- `examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-Ref2VA.py`
+- `examples/minimax_h3/model_inference/MiniMax-H3-NF4-FL2VA.py` (CPU offload)
+- `examples/minimax_h3/model_inference/MiniMax-H3-NF4-Ref2VA.py` (CPU offload)

README_from_modelscope.md

@@ -0,0 +1,216 @@
+---
+base_model:
+- MiniMax/MiniMax-H3
+frameworks:
+- ""
+license: Apache License 2.0
+base_model_relation: quantized
+---
+# MiniMax-H3-NF4
+
+MiniMax-H3 多模态音视频生成模型的 **NF4 量化版本**（通过 `bitsandbytes` 4-bit 量化），配合 [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) 使用，可在显存/内存受限的机器上进行「文本/图像/视频/音频 → 视频 + 音频」的联合生成。
+
+## 文件说明
+
+| 文件 | 大小 | 作用 | 是否共用 |
+|---|---|---|---|
+| `minimax-h3-fl2va-nf4.safetensors` | ~16 GB | **FL2VA** 任务的 DiT 主干（文本 / 首尾关键帧 → 视频+音频） | FL2VA 专用 |
+| `minimax-h3-ref2va-nf4.safetensors` | ~16 GB | **Ref2VA** 任务的 DiT 主干（参考图像/视频/音频 → 视频+音频） | Ref2VA 专用 |
+| `minimax-h3-text-encoder-nf4.safetensors` | ~15 GB | Qwen3-VL 文本/视觉编码器 | 两任务共用 |
+| `video_vae_nf4.safetensors` | ~1.6 GB | 视频 VAE 解码器 | 两任务共用 |
+| `audio_vae_nf4.safetensors` | ~271 MB | 音频 VAE 解码器 | 两任务共用 |
+
+> 说明：DiT 按任务二选一，其余三个（text_encoder / video_vae / audio_vae）在两种任务下通用。加载时框架会根据文件 hash 自动识别组件类型并套用对应的量化配置（含对少数量化敏感层的 bf16 保留），无需手动指定量化参数。
+
+## 环境要求
+
+- CUDA GPU（NF4 反量化依赖 `bitsandbytes` 的 CUDA kernel）
+- processor / tokenizer 需从原始仓库 `MiniMax/MiniMax-H3` 获取（下面 `processor_config`）
+
+### 安装 DiffSynth-Studio
+
+从源码安装（推荐，可获得最新的 MiniMax-H3 支持），并直接带上 NF4 量化依赖：
+
+```bash
+git clone https://github.com/modelscope/DiffSynth-Studio.git
+cd DiffSynth-Studio
+pip install -e ".[quant]"
+```
+
+## 使用（Disk offload，低显存和内存占用）
+
+权重存放磁盘、推理时按层流式加载到 GPU，显存占用最低。**纯文本生成视频+音频（t2v）最低约 6 GB 显存即可运行。**
+
+> `vram_limit`（单位 GB）是显存占用阈值，调小可降低显存占用（代价是更慢）。
+
+> 若 CPU 内存充足，可把 `offload_device` / `offload_dtype` 改为 `"cpu"` / `torch.bfloat16`（即 CPU offload），权重常驻内存、不走磁盘，速度更快；其余代码不变。
+
+### FL2VA — 文本 / 首尾关键帧 → 视频+音频
+
+```python
+import torch
+from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
+from diffsynth.utils.data.audio_video import write_video_audio
+from modelscope import dataset_snapshot_download
+from PIL import Image
+
+vram_config = {
+    "offload_dtype": "disk",
+    "offload_device": "disk",
+    "onload_dtype": torch.bfloat16,
+    "onload_device": "cpu",
+    "preparing_dtype": torch.bfloat16,
+    "preparing_device": "cuda",
+    "computation_dtype": torch.bfloat16,
+    "computation_device": "cuda",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-fl2va-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-text-encoder-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
+    ],
+    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="FL2VA/processor/"),
+    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 2,
+)
+
+# Text -> Video + Audio
+prompt = "A girl is very happy, she is speaking in english: “I enjoy working with Diffsynth-Studio, it's a perfect framework.”"
+video, audio = pipe(
+    prompt=prompt,
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=0,
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="t2va.mp4", fps=24, audio_sample_rate=32000,
+)
+
+# Text + First Frame + Last Frame -> Video + Audio
+dataset_snapshot_download(dataset_id="DiffSynth-Studio/diffsynth_example_dataset", local_dir="data/diffsynth_example_dataset", allow_file_pattern="minimax_h3/MiniMax-H3-FL2VA/*")
+first_frame = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/first.png")
+last_frame = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-FL2VA/last.png")
+prompt = "室内家庭争吵短剧场景，竖屏短剧质感，真实真人表演，中式家庭/小饭馆室内环境，暖色灯光，背景有红色装饰和书法字幅，浅景深，情绪强烈，剪辑节奏紧凑。表演要求：真实短剧表演风格，不要夸张舞台腔。男人的语气是愤怒、委屈、急切的反驳，他说“你到底想干什么？”；中老年女性的语气是尖锐、强势、咄咄���人的质问，她说“你必须赔钱！”。两人之间有强烈对峙感，节奏逐步升级。画面风格：竖屏9:16，手机短剧质感，真人实拍感，浅景深，室内暖光，中近景为主，频繁正反打剪辑，背景保持生活化，不要科幻、不要古装、不要动画感。画面中不要出现任何字幕、文字、平台水印或贴片。 "
+video, audio = pipe(
+    prompt=prompt,
+    height=832, width=480, num_frames=124, num_inference_steps=50, seed=0,
+    keyframes=[first_frame, last_frame], keyframe_indices=[0, -1],
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="fl2va.mp4", fps=24, audio_sample_rate=32000,
+)
+```
+
+### Ref2VA — 参考图像/视频/音频 → 视频+音频
+
+支持四种参考类型，可在一个列表里组合（`video` 无声，带声视频用 `video_audio`）：
+
+```python
+{"type": "image",       "image": PIL.Image}
+{"type": "video",       "video": list[PIL.Image]}                            # 无声
+{"type": "audio",       "audio": Tensor[C, L], "sample_rate": int}
+{"type": "video_audio", "video": list[PIL.Image], "audio": Tensor[C, L], "sample_rate": int}
+```
+
+```python
+import torch
+from PIL import Image
+from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
+from diffsynth.utils.data.audio_video import write_video_audio
+from diffsynth.utils.data.audio import read_audio
+from diffsynth.utils.data import VideoData
+from modelscope import dataset_snapshot_download
+
+def align_frame_count(frame_count):
+    current = max(int(frame_count), 1)
+    while current % 17 != 5:
+        current += 1
+    return current
+
+def read_video_with_fps(path, num_out_frames, height, width, fps=24):
+    video = VideoData(path, height=height, width=width)
+    frames = video.raw_data()
+    src_fps = float(video.data.reader.get_meta_data()["fps"])
+    out = []
+    for k in range(num_out_frames):
+        idx = int(round(k * src_fps / fps))
+        if idx >= len(frames):
+            break
+        out.append(frames[idx])
+    return out
+
+vram_config = {
+    "offload_dtype": "disk",
+    "offload_device": "disk",
+    "onload_dtype": torch.bfloat16,
+    "onload_device": "cpu",
+    "preparing_dtype": torch.bfloat16,
+    "preparing_device": "cuda",
+    "computation_dtype": torch.bfloat16,
+    "computation_device": "cuda",
+}
+pipe = MiniMaxH3Pipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-ref2va-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="minimax-h3-text-encoder-nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="video_vae_nf4.safetensors", **vram_config),
+        ModelConfig(model_id="DiffSynth-Studio/MiniMax-H3-NF4", origin_file_pattern="audio_vae_nf4.safetensors", **vram_config),
+    ],
+    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/processor/"),
+    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 5,
+)
+
+# Text + Reference Image -> Video + Audio
+dataset_snapshot_download(dataset_id="DiffSynth-Studio/diffsynth_example_dataset", local_dir="data/diffsynth_example_dataset", allow_file_pattern="minimax_h3/MiniMax-H3-Ref2VA/*")
+ref_image = Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/0.png").convert("RGB")
+prompt = "一个网站页面，网站页面UI设计，网站动效，视频展示了流畅的网页向下滚动效果。一个极具爆发力与动感的产品官网风格产品落地页 UI/UX 演示视频，核心展示主体是该产品图片1。页面采用粗犷有力、倾斜的超大号无衬线字体进行张扬的排版。背景有极具速度感的动态光影、暗色碳纤维或运动透气网眼纹理在交织变换。视频展示了节奏紧凑、充满力量感的网页向下滚动效果，以及鼠标悬停时强烈的视觉放大与颜色反转等 UI 交互动作。"
+video, audio = pipe(
+    prompt=prompt,
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=42,
+    references=[{"type": "image", "image": Image.open("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/0.png").convert("RGB")}]
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="ti2va.mp4", fps=24, audio_sample_rate=32000,
+)
+
+# Text + Reference Audio + Reference Video -> Video + Audio
+ref_video = read_video_with_fps("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/video.mp4", 124, 480, 832)
+ref_audio, sample_rate = read_audio("data/diffsynth_example_dataset/minimax_h3/MiniMax-H3-Ref2VA/voice.mp3", duration=len(ref_video) / 24, resample=True, resample_rate=pipe.audio_vae.sample_rate)
+prompt = "subject_definitions:\n<Subject 1> is the young man with short wavy blonde hair, wearing a bright pink suit jacket, matching pink trousers, an unbuttoned white shirt, and silver rings, holding a small black lamb in his arms in <Video 1>.\n<Video 1> is the source video for the editing task.\n<Audio 1> is the synchronized audio track of <Video 1>, providing the background music.\n<Audio 2> is the voice timbre reference for <Subject 1>'s voice, containing a spoken male voiceover.\n\nsummary:\n[video editing + audio reference + audio reuse] The target video is an edited version of <Video 1>. <Subject 1>, wearing a bright pink suit and holding a black lamb, stands in a grassy field with other white lambs in the background. The edit animates <Subject 1>'s face to speak the user-provided dialogue. <Audio 1> is partially reused as the continuous background music, while the target references the calm male voice timbre of <Audio 2> for <Subject 1>'s spoken lines.\n\nretention_analysis:\n<Subject 1> (appears in [Shot 1]): fully_preserved - the man retains his identity, wavy blonde hair, pink suit, white shirt, accessories, and the black lamb he holds, with his mouth newly animated to speak.\n<Video 1> (source video editing): fully_preserved - the original camera framing, warm golden hour lighting, grassy hill setting, and background white lambs are maintained while the central character is edited.\n<Audio 1>: partially_copy - the atmospheric background music from <Audio 1> is reused in the target video, mixed beneath the newly added spoken dialogue.\n<Audio 2>: reference - the target audio references the male voice timbre from <Audio 2> to generate <Subject 1>'s spoken dialogue.\n\ndetailed_description:\nThe target video is in realistic photographic style.\n[Shot 1] The shot begins from the source <Video 1>, showing <Subject 1>, a young man with short wavy blonde hair, wearing a bright pink suit jacket, matching pink trousers, and a casually unbuttoned white shirt. He stands confidently in a sunlit green pasture, gently holding a small black lamb securely in his arms. The warm, golden hour lighting casts soft shadows across his face and the bright pink fabric of his suit. Behind him, several white lambs stand and graze on the rolling grassy hill against a clear, pale blue sky. The atmospheric background music from <Audio 1> plays continuously throughout the scene. <Subject 1> physically speaks, his mouth movements naturally syncing to the new dialogue, with his voice timbre referencing the calm male delivery from <Audio 2>. Looking thoughtfully forward, <Subject 1> (S1) speaks softly, <d>[English] Follow the wind, live free.</d> As he delivers the line, he subtly shifts his weight, cradling the resting black lamb while the camera slowly pushes in. <Subject 1> (S1) continues his thought, <d>[English] Leave worries behind, enjoy the moment.</d> Exactly as his voice stops, his lips meet in a relaxed, peaceful smile, and his jaw ceases speaking motion. He then turns his gaze slightly away toward the horizon, gently stroking the black lamb's fleece with his fingers as the camera holds on this tranquil, sunlit state through the end of the video.\n\noverall_soundscape:\nThe soundscape consists of the continuous, atmospheric background music from <Audio 1>, overlaid with the clear, calm male dialogue spoken by the main character, referencing the voice timbre of <Audio 2>.\n\nnon_diegetic_music:\nThe atmospheric, sustained background music from <Audio 1> is reused as the continuous score, playing quietly beneath the spoken dialogue."
+video, audio = pipe(
+    prompt=prompt,
+    height=480, width=832, num_frames=124, num_inference_steps=50, seed=42,
+    references=[
+        {"type": "video", "video": ref_video},
+        {"type": "audio", "audio": ref_audio, "sample_rate": sample_rate},
+    ],
+)
+write_video_audio(
+    video=video, audio=audio,
+    output_path="tav2va.mp4", fps=24, audio_sample_rate=32000,
+)
+```
+
+## 常用参数
+
+- `height` / `width`：分辨率，如 `480x832`（横）或 `832x480`（竖）。
+- `num_frames`：帧数，需满足 `num_frames % 17 == 5`（如 124）。
+- `num_inference_steps`：去噪步数，示例用 50。
+- `keyframes` / `keyframe_indices`：FL2VA 首尾帧控制（`[0, -1]` 表示首帧和尾帧）。
+- `references`：Ref2VA 参考列表，元素为 `{"type": "image|video|audio|video_audio", ...}`。
+- 输出：`write_video_audio(video, audio, output_path, fps=24, audio_sample_rate=32000)`。
+
+## 参考示例脚本
+
+仓库内完整可运行脚本（本 README 的示例即取自这些脚本）：
+
+- `examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-FL2VA.py`
+- `examples/minimax_h3/model_inference_low_vram/MiniMax-H3-NF4-Ref2VA.py`
+- `examples/minimax_h3/model_inference/MiniMax-H3-NF4-FL2VA.py`（CPU offload）
+- `examples/minimax_h3/model_inference/MiniMax-H3-NF4-Ref2VA.py`（CPU offload）

audio_vae_nf4.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:759662130ba3618b7f196da8f983f857a1f1ec6af8110d657796d9792c0d64e5
+size 284004112

configuration.json

@@ -0,0 +1 @@
+{"task":"text-to-video-synthesis"}

minimax-h3-fl2va-nf4.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9416f2deb3a6bf1455457bdc373023f266bddcc8646edf7a73394ecd9f13e960
+size 17162138303

minimax-h3-ref2va-nf4.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:765f82272ad5b1e486beef4a5aca719430c6295f3d8d8d07de602a2a740253bb
+size 17162138284

minimax-h3-text-encoder-nf4.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0964f0634856e68522aa64b801afcd31b4e878619eff21a82324e2cd61690f13
+size 15324775807

video_vae_nf4.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d0cb4ff02ebb74cc6bca40018e6efae5082ccd7eb066fa1263098c6dbf8f6f1
+size 1613201536