MiniMax-H3 Character Swap: guide latent + native image reference port
Browse files- README.md +63 -8
- app.py +576 -0
- h3_charswap_blocks.py +294 -0
- h3_lora.py +138 -0
- h3_video.py +205 -0
- packages.txt +1 -0
- requirements.txt +22 -0
README.md
CHANGED
|
@@ -1,13 +1,68 @@
|
|
| 1 |
---
|
| 2 |
-
title:
|
| 3 |
-
emoji:
|
| 4 |
-
colorFrom:
|
| 5 |
-
colorTo:
|
| 6 |
sdk: gradio
|
| 7 |
-
sdk_version: 6.
|
| 8 |
-
python_version: '3.13'
|
| 9 |
app_file: app.py
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
---
|
| 12 |
|
| 13 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
title: MiniMax-H3 Character Swap LoRA
|
| 3 |
+
emoji: 🎭
|
| 4 |
+
colorFrom: purple
|
| 5 |
+
colorTo: blue
|
| 6 |
sdk: gradio
|
| 7 |
+
sdk_version: 6.26.0
|
|
|
|
| 8 |
app_file: app.py
|
| 9 |
+
python_version: "3.12"
|
| 10 |
+
startup_duration_timeout: 1h
|
| 11 |
+
short_description: Character swap on video with the MiniMax-H3 ref2va LoRA
|
| 12 |
+
models:
|
| 13 |
+
- Alissonerdx/Minimax-H3-ComfyUI
|
| 14 |
+
- multimodalart/MiniMax-H3-Pruned
|
| 15 |
+
- MiniMaxAI/MiniMax-H3
|
| 16 |
+
- lightx2v/Minimax-h3-Turbo
|
| 17 |
+
- briaai/RMBG-2.0
|
| 18 |
---
|
| 19 |
|
| 20 |
+
# MiniMax-H3 · Character Swap LoRA
|
| 21 |
+
|
| 22 |
+
Replace the **main subject of a guide video** with the **character in a reference image**, keeping the guide's
|
| 23 |
+
scene, camera, motion and performance. This is a `diffusers` port of the ComfyUI graph the LoRA ships
|
| 24 |
+
(`workflows/minimax_h3_character_swap_workflow.json`), minus its optional BFS multi-shot planner — one guide clip,
|
| 25 |
+
one reference character, one swap.
|
| 26 |
+
|
| 27 |
+
LoRA: [`Alissonerdx/Minimax-H3-ComfyUI`](https://huggingface.co/Alissonerdx/Minimax-H3-ComfyUI) →
|
| 28 |
+
`loras/minimax_h3_character_swap_v1.0_ref2va_r64_gl.safetensors` (rank 64, `ref2va`, guide-latent), by
|
| 29 |
+
[Alissonerdx](https://github.com/Alissonerdx/Minimax-H3-ComfyUI).
|
| 30 |
+
|
| 31 |
+
## How it conditions
|
| 32 |
+
|
| 33 |
+
The LoRA's own training metadata records that it sees **two** visual blocks at once:
|
| 34 |
+
|
| 35 |
+
```
|
| 36 |
+
minimax_h3_reference_rope = {"guide_rope_layout": "overlap", "reference_rope_layout": "native", ...}
|
| 37 |
+
minimax_h3_dataset_control_roles = [["guide", "reference", "reference"]]
|
| 38 |
+
guide_latent_only = true
|
| 39 |
+
```
|
| 40 |
+
|
| 41 |
+
| ComfyUI node | here |
|
| 42 |
+
|---|---|
|
| 43 |
+
| `BiRefNetRMBG` on the reference | [`briaai/RMBG-2.0`](https://huggingface.co/briaai/RMBG-2.0) via the `linoyts/bria-rmbg-service` client Space, cutout on `#808080` |
|
| 44 |
+
| `<Picture 1>` through the Qwen3-VL CLIP | [`multimodalart/qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner) `/encode_ref2va`, image kind |
|
| 45 |
+
| `MiniMaxH3AddGuide(frame_idx = 0)` | `h3_charswap_blocks.MiniMaxH3CharswapPrepareLayoutStep` — the guide on the target's own `(t, h, w)` grid (`overlap`) |
|
| 46 |
+
| native image-reference block | the same layout step — the reference image on H3's native rope, one integer slot (`native`) |
|
| 47 |
+
| `LoraLoaderModelOnly(character_swap_gl, 1.0)` | folded at startup by `h3_lora.fold_comfy_lora` |
|
| 48 |
+
| few-step turbo + `BasicScheduler("simple", 8)` | [`lightx2v/Minimax-h3-Turbo`](https://huggingface.co/lightx2v/Minimax-h3-Turbo) 4-step folded at 1.0, 8 Euler steps, no CFG |
|
| 49 |
+
| `LTXVSeparateAVLatent` -> `VAEDecode` (video only) | video-only decode; the guide's own soundtrack is muxed back |
|
| 50 |
+
|
| 51 |
+
So the packed layout is `[text (+ Picture 1 vision) | reference image | guide | target audio | target video]`. The
|
| 52 |
+
reference image takes one integer rotary slot; the guide overlaps the target on its own grid, frame for frame.
|
| 53 |
+
|
| 54 |
+
Weights are `multimodalart/MiniMax-H3-Pruned`'s `transformer_ref` — the released `ref2va` partition with its 2688-wide
|
| 55 |
+
AdaLN projections replaced by a rank-8 factorization, the same pruning the workflow's
|
| 56 |
+
`minimax_h3_ref2va_pruned_int8_convrot.safetensors` carries — so both LoRAs fold onto it natively. H3's 62 GiB
|
| 57 |
+
Qwen3-VL text tower will not fit alongside the transformer on one ZeroGPU worker, so the caption and the reference
|
| 58 |
+
image are encoded by the conditioner Space (billed to the visitor's own quota via `x-ip-token`).
|
| 59 |
+
|
| 60 |
+
## Notes and limits
|
| 61 |
+
|
| 62 |
+
* Experimental, not a guaranteed identity-transfer system. Results drift with small faces, occlusion, fast motion,
|
| 63 |
+
hard angles, lighting changes, or long clips; it is aimed at people and human-like characters.
|
| 64 |
+
* The workflow's `drbaph` DMD 8-step turbo carries `adaln_proj.linear.bias.diff` direct deltas the LoRA-fold path
|
| 65 |
+
does not apply, so this Space stacks the sibling LMS Space's proven `lightx2v` 4-step turbo instead.
|
| 66 |
+
* Use only with appropriate rights and consent for the reference images and guide videos. Not for impersonation,
|
| 67 |
+
fraud, harassment, or other unlawful purposes. See the
|
| 68 |
+
[Character Swap license](https://huggingface.co/Alissonerdx/Minimax-H3-ComfyUI/blob/main/LICENSE_CHARACTER_SWAP.md).
|
app.py
ADDED
|
@@ -0,0 +1,576 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""MiniMax-H3 Character Swap — the guide-latent + image-reference swap LoRA from `Alissonerdx/Minimax-H3-ComfyUI`.
|
| 2 |
+
|
| 3 |
+
`loras/minimax_h3_character_swap_v1.0_ref2va_r64_gl.safetensors` is a rank-64 LoRA for MiniMax-H3's `ref2va`
|
| 4 |
+
partition that **replaces the main subject of a guide video with the character in a reference image** while keeping
|
| 5 |
+
the guide's scene, camera, motion and performance. Its own training metadata spells out how it conditions:
|
| 6 |
+
|
| 7 |
+
minimax_h3_reference_rope = {"guide_rope_layout": "overlap", "reference_rope_layout": "native", ...}
|
| 8 |
+
minimax_h3_dataset_control_roles = [["guide", "reference", "reference"]]
|
| 9 |
+
guide_latent_only = true
|
| 10 |
+
|
| 11 |
+
so it sees two visual blocks at once — an **aligned guide** (the whole source clip VAE-encoded at the target
|
| 12 |
+
resolution and packed on the target's own `(t, h, w)` grid, ComfyUI's `MiniMaxH3AddGuide(frame_idx = 0)`) and a
|
| 13 |
+
**native image reference** (the character image as a Qwen3-VL `<Picture 1>` vision block *and* a VAE reference
|
| 14 |
+
block). This Space is a diffusers port of the repo's `workflows/minimax_h3_character_swap_workflow.json`, minus its
|
| 15 |
+
optional BFS multi-shot planner: one guide clip, one reference character, one swap.
|
| 16 |
+
|
| 17 |
+
| ComfyUI node | here |
|
| 18 |
+
|---|---|
|
| 19 |
+
| `BiRefNetRMBG` on the reference | `linoyts/bria-rmbg-service` (a `briaai/RMBG-2.0` client Space), cutout on `#808080` |
|
| 20 |
+
| `<Picture 1>` through the CLIP (Qwen3-VL) | `multimodalart/qwen3vl-conditioner` `/encode_ref2va`, image kind |
|
| 21 |
+
| `MiniMaxH3AddGuide(frame_idx = 0)` | `h3_charswap_blocks.MiniMaxH3CharswapPrepareLayoutStep`, guide on the target grid |
|
| 22 |
+
| native image reference block | the same layout step, reference on H3's native rope — see `h3_charswap_blocks.py` |
|
| 23 |
+
| `LoraLoaderModelOnly(character_swap_gl, 1.0)` | folded at startup by `h3_lora.fold_comfy_lora` |
|
| 24 |
+
| few-step turbo + `BasicScheduler("simple", 8)` | `lightx2v/Minimax-h3-Turbo` 4-step folded at 1.0, 8 Euler steps, no CFG |
|
| 25 |
+
| `PrimitiveStringMultiline` `body_swap:` | the trigger caption built from the reference description below |
|
| 26 |
+
| `LTXVSeparateAVLatent` -> `VAEDecode` (video only) | video-only decode; the guide's own soundtrack is muxed back |
|
| 27 |
+
|
| 28 |
+
Weights are `multimodalart/MiniMax-H3-Pruned`'s `transformer_ref` (the `ref2va` partition with rank-8 AdaLN factors),
|
| 29 |
+
the same pruning the workflow's own `minimax_h3_ref2va_pruned_int8_convrot.safetensors` carries, which is why both
|
| 30 |
+
LoRAs fold onto it natively. H3's 62 GiB Qwen3-VL does not fit next to the transformer on one ZeroGPU worker, so the
|
| 31 |
+
caption *and the reference image* are encoded by `multimodalart/qwen3vl-conditioner` over the gradio API (billed to
|
| 32 |
+
the visitor's own quota via `x-ip-token`).
|
| 33 |
+
"""
|
| 34 |
+
|
| 35 |
+
from __future__ import annotations
|
| 36 |
+
|
| 37 |
+
import os
|
| 38 |
+
|
| 39 |
+
# Before torch. See the sibling LMS Space: the guide VAE encode tripped the ZeroGPU allocator's expandable-segments
|
| 40 |
+
# path (`NVML_SUCCESS == r INTERNAL ASSERT FAILED`) with memory free; plain `cudaMalloc` never queries NVML.
|
| 41 |
+
_ALLOC_CONF_INHERITED = os.environ.get("PYTORCH_CUDA_ALLOC_CONF")
|
| 42 |
+
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = os.environ.get("H3_ALLOC_CONF", "expandable_segments:False")
|
| 43 |
+
|
| 44 |
+
import hashlib
|
| 45 |
+
import math
|
| 46 |
+
import tempfile
|
| 47 |
+
import time
|
| 48 |
+
import traceback
|
| 49 |
+
import uuid
|
| 50 |
+
from functools import cache
|
| 51 |
+
|
| 52 |
+
import spaces
|
| 53 |
+
import gradio as gr
|
| 54 |
+
import torch
|
| 55 |
+
|
| 56 |
+
import h3_video
|
| 57 |
+
|
| 58 |
+
MODEL_REPO = os.environ.get("H3_MODEL_REPO", "multimodalart/MiniMax-H3-Pruned")
|
| 59 |
+
CONDITIONER_SPACE = os.environ.get("H3_CONDITIONER", "multimodalart/qwen3vl-conditioner")
|
| 60 |
+
RMBG_SPACE = os.environ.get("H3_RMBG", "linoyts/bria-rmbg-service")
|
| 61 |
+
GPU_SIZE = os.environ.get("H3_GPU_SIZE", "xlarge")
|
| 62 |
+
ATTENTION = os.environ.get("H3_ATTENTION", "_native_cudnn")
|
| 63 |
+
|
| 64 |
+
# The task LoRA, stacked on the proven 4-step turbo the sibling LMS Space uses (the workflow's own drbaph DMD turbo
|
| 65 |
+
# carries `.bias.diff` direct deltas the fold path does not apply, so it would load half-merged). Both at 1.0,
|
| 66 |
+
# turbo first, folded into the base weights before the model reaches the GPU.
|
| 67 |
+
LORAS = [
|
| 68 |
+
(
|
| 69 |
+
"lightx2v/Minimax-h3-Turbo",
|
| 70 |
+
"minimax_h3_ref2v_turbo_4step_v0.1_comfyui_bf16.safetensors",
|
| 71 |
+
1.0,
|
| 72 |
+
"turbo-4step",
|
| 73 |
+
),
|
| 74 |
+
(
|
| 75 |
+
os.environ.get("H3_LORA_REPO", "Alissonerdx/Minimax-H3-ComfyUI"),
|
| 76 |
+
os.environ.get("H3_LORA_FILE", "loras/minimax_h3_character_swap_v1.0_ref2va_r64_gl.safetensors"),
|
| 77 |
+
1.0,
|
| 78 |
+
"character-swap-r64-gl",
|
| 79 |
+
),
|
| 80 |
+
]
|
| 81 |
+
|
| 82 |
+
# `workflows/minimax_h3_character_swap_workflow.json`'s `body_swap:` prompt, verbatim. `{details}` is the reference
|
| 83 |
+
# description, `{target}` names which guide subject to replace when more than one person appears.
|
| 84 |
+
PROMPT_MAIN = (
|
| 85 |
+
"body_swap: subject_definitions: <Subject 1> is the {details} shown in <Picture 1>, whose identity, face, "
|
| 86 |
+
"head, hair or fur, skin, body shape, proportions and clothing define the character in the target video. The "
|
| 87 |
+
"guide video provides the scene and the performance: its background, environment, lighting, camera, framing, "
|
| 88 |
+
"body pose, hand and arm movement, head rotation, gaze direction, lip movement and facial expression, frame by "
|
| 89 |
+
"frame; only the main subject of the guide video is replaced: its face, hair, body and clothing become "
|
| 90 |
+
"<Subject 1>. summary: [video generation + reference] Replace the guide subject with <Subject 1>, keeping the "
|
| 91 |
+
"scene, camera and the complete motion and performance of the guide video. detailed_description: [Shot 1] "
|
| 92 |
+
"<Subject 1> takes the exact place of the guide subject and moves with the same timing, body pose, gestures, "
|
| 93 |
+
"head motion, gaze, speech and facial expressions, while keeping the complete look from <Picture 1>: head, "
|
| 94 |
+
"face, skin, hair or fur, body shape and clothing stay stable in every frame, with no trace of the original "
|
| 95 |
+
"subject's face, hair or clothing. The background, lighting, colors and camera of the guide video are "
|
| 96 |
+
"preserved. Prevent identity drift, clothing changes, texture crawling and flicker. The result is a real, "
|
| 97 |
+
"photographic video."
|
| 98 |
+
)
|
| 99 |
+
PROMPT_TARGETED = (
|
| 100 |
+
"body_swap: subject_definitions: <Subject 1> is the {details} shown in <Picture 1>, whose identity, face, "
|
| 101 |
+
"head, hair or fur, skin, body shape, proportions and clothing define the character in the target video. The "
|
| 102 |
+
"guide video provides the scene and the performance: its background, environment, lighting, camera, framing, "
|
| 103 |
+
"body pose, hand and arm movement, head rotation, gaze direction, lip movement and facial expression, frame by "
|
| 104 |
+
"frame; only {target} is replaced: their face, hair, body and clothing become <Subject 1>. summary: [video "
|
| 105 |
+
"generation + reference] Replace {target} in the guide video with <Subject 1>, keeping the scene, camera and "
|
| 106 |
+
"the complete motion and performance of the guide video. detailed_description: [Shot 1] <Subject 1> takes the "
|
| 107 |
+
"exact place of {target} and moves with the same timing, body pose, gestures, head motion, gaze, speech and "
|
| 108 |
+
"facial expressions, while keeping the complete look from <Picture 1>: head, face, skin, hair or fur, body "
|
| 109 |
+
"shape and clothing stay stable in every frame, with no trace of the original subject's face, hair or "
|
| 110 |
+
"clothing. The background, lighting, colors and camera of the guide video are preserved. Prevent identity "
|
| 111 |
+
"drift, clothing changes, texture crawling and flicker. The result is a real, photographic video."
|
| 112 |
+
)
|
| 113 |
+
DEFAULT_DETAILS = "character"
|
| 114 |
+
|
| 115 |
+
FPS = h3_video.FPS
|
| 116 |
+
DEFAULT_STEPS = 8
|
| 117 |
+
DEFAULT_SEED = 46 # `RandomNoise(46, "fixed")` in the workflow.
|
| 118 |
+
|
| 119 |
+
# `AIToolkitMiniMaxH3RefVideo.max_length` — a cap on the 24 fps frame count, snapped down to 17n + 5 afterwards.
|
| 120 |
+
CLIP_LENGTHS = {
|
| 121 |
+
"0.9 s · 22 frames": 22,
|
| 122 |
+
"1.6 s · 39 frames": 39,
|
| 123 |
+
"2.3 s · 56 frames": 56,
|
| 124 |
+
"3.0 s · 73 frames": 73,
|
| 125 |
+
"3.8 s · 90 frames": 90,
|
| 126 |
+
}
|
| 127 |
+
DEFAULT_CLIP = "2.3 s · 56 frames"
|
| 128 |
+
|
| 129 |
+
# Working pixel area of the guide / generation, on the clip's own aspect ratio.
|
| 130 |
+
DETAIL_LEVELS = {
|
| 131 |
+
"0.26 MP · fastest (672x384 at 16:9)": 0.258,
|
| 132 |
+
"0.42 MP · balanced (864x480 at 16:9)": 0.42,
|
| 133 |
+
"0.59 MP · sharp (1024x576 at 16:9)": 0.59,
|
| 134 |
+
}
|
| 135 |
+
DEFAULT_DETAIL = "0.42 MP · balanced (864x480 at 16:9)"
|
| 136 |
+
|
| 137 |
+
# The reference image is encoded at its own resolution; the workflow sizes it to ~0.3 MP. The conditioner also
|
| 138 |
+
# validates a canvas/frame count it never uses for an image-only request — these just have to be in range.
|
| 139 |
+
REFERENCE_MEGAPIXELS = 0.3
|
| 140 |
+
CONDITIONER_CANVAS = "960x544 · 16:9 fast"
|
| 141 |
+
|
| 142 |
+
OUTPUT_DIR = os.path.join(tempfile.gettempdir(), "h3-charswap-out")
|
| 143 |
+
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
| 144 |
+
|
| 145 |
+
PIPE = None
|
| 146 |
+
LOAD_ERROR: str | None = None
|
| 147 |
+
LOAD_NOTES: list[str] = []
|
| 148 |
+
_PROMPT_CACHE: dict[str, tuple] = {}
|
| 149 |
+
_RMBG_CACHE: dict[str, str] = {}
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
# ── Model loading ───────────────────────────────────────────────────────────
|
| 153 |
+
|
| 154 |
+
|
| 155 |
+
def load_models() -> str | None:
|
| 156 |
+
"""Load the denoising half — `transformer_ref` + video VAE + schedulers — and fold both LoRAs in."""
|
| 157 |
+
global PIPE, LOAD_ERROR
|
| 158 |
+
|
| 159 |
+
if PIPE is not None or LOAD_ERROR is not None:
|
| 160 |
+
return LOAD_ERROR
|
| 161 |
+
|
| 162 |
+
started = time.time()
|
| 163 |
+
try:
|
| 164 |
+
from diffusers import ComponentsManager
|
| 165 |
+
from diffusers.modular_pipelines.minimax_h3.modular_pipeline import MiniMaxH3ModularPipeline
|
| 166 |
+
from huggingface_hub import hf_hub_download
|
| 167 |
+
|
| 168 |
+
from h3_charswap_blocks import MiniMaxH3CharswapGeneratorBlocks
|
| 169 |
+
from h3_lora import fold_comfy_lora
|
| 170 |
+
|
| 171 |
+
MiniMaxH3ModularPipeline.min_duration = property(lambda self: 0.2)
|
| 172 |
+
|
| 173 |
+
blocks = MiniMaxH3CharswapGeneratorBlocks()
|
| 174 |
+
print(f"[gen] loading {[c.name for c in blocks.expected_components]} from {MODEL_REPO} ...", flush=True)
|
| 175 |
+
pipe = blocks.init_pipeline(MODEL_REPO, components_manager=ComponentsManager(), collection="h3")
|
| 176 |
+
pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)
|
| 177 |
+
for name in ("transformer_ref", "vae", "scheduler", "audio_scheduler"):
|
| 178 |
+
if getattr(pipe, name, None) is None:
|
| 179 |
+
raise RuntimeError(f"Component `{name}` did not load; see the warning above for why.")
|
| 180 |
+
|
| 181 |
+
pipe.vae.set_attention_backend("native")
|
| 182 |
+
pipe.transformer_ref.set_attention_backend(ATTENTION)
|
| 183 |
+
|
| 184 |
+
for repo, filename, strength, label in LORAS:
|
| 185 |
+
path = hf_hub_download(repo, filename)
|
| 186 |
+
status = fold_comfy_lora(pipe.transformer_ref, path, strength=strength, label=label)
|
| 187 |
+
LOAD_NOTES.append(status)
|
| 188 |
+
print(f"[lora] {status}", flush=True)
|
| 189 |
+
|
| 190 |
+
pipe.transformer_ref.to("cuda")
|
| 191 |
+
PIPE = pipe
|
| 192 |
+
print(f"[gen] ready in {time.time() - started:.0f}s", flush=True)
|
| 193 |
+
except Exception as error:
|
| 194 |
+
traceback.print_exc()
|
| 195 |
+
LOAD_ERROR = f"**Loading failed** after {time.time() - started:.0f}s: `{type(error).__name__}: {error}`"
|
| 196 |
+
return LOAD_ERROR
|
| 197 |
+
|
| 198 |
+
|
| 199 |
+
# ── Remote conditioning + background removal ──────────────────────────────────
|
| 200 |
+
|
| 201 |
+
|
| 202 |
+
def _caller_ip_token() -> str | None:
|
| 203 |
+
from gradio.context import LocalContext
|
| 204 |
+
|
| 205 |
+
request = LocalContext.request.get()
|
| 206 |
+
return request.headers.get("x-ip-token") if request is not None else None
|
| 207 |
+
|
| 208 |
+
|
| 209 |
+
@cache
|
| 210 |
+
def _anonymous_client(space: str):
|
| 211 |
+
from gradio_client import Client
|
| 212 |
+
|
| 213 |
+
return Client(space)
|
| 214 |
+
|
| 215 |
+
|
| 216 |
+
def _client(space: str, ip_token: str | None):
|
| 217 |
+
"""A client for a helper Space, billed to the visitor when their `x-ip-token` is available."""
|
| 218 |
+
if not ip_token:
|
| 219 |
+
return _anonymous_client(space)
|
| 220 |
+
from gradio_client import Client
|
| 221 |
+
|
| 222 |
+
return Client(space, headers={"x-ip-token": ip_token})
|
| 223 |
+
|
| 224 |
+
|
| 225 |
+
def _file_digest(path: str) -> str:
|
| 226 |
+
with open(path, "rb") as handle:
|
| 227 |
+
return hashlib.sha256(handle.read()).hexdigest()
|
| 228 |
+
|
| 229 |
+
|
| 230 |
+
def remove_background(image_path: str, ip_token: str | None) -> str:
|
| 231 |
+
"""Cut the reference subject out onto neutral gray via the RMBG service; fall back to the raw image."""
|
| 232 |
+
digest = _file_digest(image_path)
|
| 233 |
+
if digest in _RMBG_CACHE and os.path.exists(_RMBG_CACHE[digest]):
|
| 234 |
+
return _RMBG_CACHE[digest]
|
| 235 |
+
from gradio_client import handle_file
|
| 236 |
+
|
| 237 |
+
try:
|
| 238 |
+
result = _client(RMBG_SPACE, ip_token).predict(
|
| 239 |
+
image_path=handle_file(image_path), background="gray", api_name="/rmbg"
|
| 240 |
+
)
|
| 241 |
+
except Exception as error:
|
| 242 |
+
print(f"[rmbg] keeping the raw reference, service failed: {error}", flush=True)
|
| 243 |
+
return image_path
|
| 244 |
+
_RMBG_CACHE[digest] = result
|
| 245 |
+
return result
|
| 246 |
+
|
| 247 |
+
|
| 248 |
+
def _read_conditioning(path: str):
|
| 249 |
+
from safetensors import safe_open
|
| 250 |
+
|
| 251 |
+
with safe_open(path, framework="pt") as handle:
|
| 252 |
+
return handle.get_tensor("prompt_embeds"), handle.get_tensor("text_token_tags")
|
| 253 |
+
|
| 254 |
+
|
| 255 |
+
def encode_prompt(prompt: str, image_path: str, num_frames: int):
|
| 256 |
+
"""`prompt_embeds` + `text_token_tags` for a `body_swap` caption with the reference image as `<Picture 1>`.
|
| 257 |
+
|
| 258 |
+
The reference image reaches Qwen3-VL as an ordered `ref2va` image reference, so the embedding carries its
|
| 259 |
+
`<Picture 1>` vision block. Cached per `(prompt, reference image)`; `num_frames` only has to be a valid
|
| 260 |
+
duration for the conditioner's own range check, it does not change an image-only embedding.
|
| 261 |
+
"""
|
| 262 |
+
key = f"{prompt}\n{_file_digest(image_path)}"
|
| 263 |
+
if key in _PROMPT_CACHE:
|
| 264 |
+
return _PROMPT_CACHE[key]
|
| 265 |
+
|
| 266 |
+
from gradio_client import handle_file
|
| 267 |
+
|
| 268 |
+
ip_token = _caller_ip_token()
|
| 269 |
+
|
| 270 |
+
def call():
|
| 271 |
+
return _client(CONDITIONER_SPACE, ip_token).predict(
|
| 272 |
+
prompt=prompt,
|
| 273 |
+
media=[handle_file(image_path)],
|
| 274 |
+
kinds="image",
|
| 275 |
+
canvas=CONDITIONER_CANVAS,
|
| 276 |
+
num_frames=int(num_frames),
|
| 277 |
+
rewrite_prompt=False,
|
| 278 |
+
api_name="/encode_ref2va",
|
| 279 |
+
)
|
| 280 |
+
|
| 281 |
+
try:
|
| 282 |
+
path, _plan = call()
|
| 283 |
+
except Exception as first:
|
| 284 |
+
print(f"[conditioner] retrying with a fresh client after: {first}", flush=True)
|
| 285 |
+
_anonymous_client.cache_clear()
|
| 286 |
+
try:
|
| 287 |
+
path, _plan = call()
|
| 288 |
+
except Exception as second:
|
| 289 |
+
raise gr.Error(
|
| 290 |
+
"The Qwen3-VL conditioner Space could not encode the reference "
|
| 291 |
+
f"({type(second).__name__}: {second})."
|
| 292 |
+
) from second
|
| 293 |
+
|
| 294 |
+
embeds = _read_conditioning(path)
|
| 295 |
+
_PROMPT_CACHE[key] = embeds
|
| 296 |
+
return embeds
|
| 297 |
+
|
| 298 |
+
|
| 299 |
+
# ── GPU budget ──────────────────────────────────────────────────────────────
|
| 300 |
+
|
| 301 |
+
# Seconds per step vs packed row count: linear matmuls, quadratic full self-attention. Calibrated by the sibling
|
| 302 |
+
# MiniMax-H3 Spaces on this pool.
|
| 303 |
+
_DUR_B, _DUR_C = 1.1745e-4, 3.8396e-9
|
| 304 |
+
_VAE_PER_REFERENCE_PIXELS, _REFERENCE_PIXELS = 26.0, 960 * 544 * 124
|
| 305 |
+
_PLACEMENT_ALLOWANCE = 15
|
| 306 |
+
_SAFETY = 1.2
|
| 307 |
+
|
| 308 |
+
|
| 309 |
+
def _reference_latent_patches(megapixels: float) -> int:
|
| 310 |
+
"""Patch rows a reference image sized to `megapixels` adds (one latent frame, /32 patch grid)."""
|
| 311 |
+
side = math.sqrt(megapixels * 1e6)
|
| 312 |
+
return max(1, round(side / 32) ** 2)
|
| 313 |
+
|
| 314 |
+
|
| 315 |
+
def estimate_duration(height: int, width: int, num_frames: int, steps: int) -> int:
|
| 316 |
+
"""Seconds of GPU time one swap pass needs: reference + guide conditioning rows and the target included."""
|
| 317 |
+
latent_frames = (num_frames - 5) // 17 * 5 + 2
|
| 318 |
+
patches = (height // 32) * (width // 32)
|
| 319 |
+
# Guide doubles the video rows (one conditioning row per target row); the reference adds one small frame.
|
| 320 |
+
rows = 2 * latent_frames * patches + _reference_latent_patches(REFERENCE_MEGAPIXELS)
|
| 321 |
+
denoise = steps * (_DUR_B * rows + _DUR_C * rows**2)
|
| 322 |
+
vae = _VAE_PER_REFERENCE_PIXELS * (height * width * num_frames) / _REFERENCE_PIXELS
|
| 323 |
+
return max(60, int((denoise + vae) * _SAFETY) + _PLACEMENT_ALLOWANCE)
|
| 324 |
+
|
| 325 |
+
|
| 326 |
+
def _gpu_duration(ref_pixels, pixels, prompt_embeds, tags, height, width, num_frames, steps, seed, *a, **k):
|
| 327 |
+
return estimate_duration(int(height), int(width), int(num_frames), int(steps))
|
| 328 |
+
|
| 329 |
+
|
| 330 |
+
# ── Inference ───────────────────────────────────────────────────────────────
|
| 331 |
+
|
| 332 |
+
|
| 333 |
+
@torch.no_grad()
|
| 334 |
+
def encode_condition(vae, pixels, pixel_mean, pixel_std, encode_seed):
|
| 335 |
+
"""`diffusers...minimax_h3.encoders.encode_vae_condition`, cast to the VAE's own dtype (see the LMS Space).
|
| 336 |
+
|
| 337 |
+
A single-frame `(1, 3, 1, H, W)` input (the reference image) runs the spatial encoder alone and returns one
|
| 338 |
+
latent frame; a clip runs the temporal chunking. `no_grad` keeps the float32 encoder's activations from being
|
| 339 |
+
retained for a backward pass that never comes.
|
| 340 |
+
"""
|
| 341 |
+
latents_mean = torch.tensor(vae.config.latents_mean).view(1, -1, 1, 1, 1)
|
| 342 |
+
latents_std = torch.tensor(vae.config.latents_std).view(1, -1, 1, 1, 1)
|
| 343 |
+
pixel_mean = torch.tensor(pixel_mean, device=pixels.device).view(1, -1, 1, 1, 1)
|
| 344 |
+
pixel_std = torch.tensor(pixel_std, device=pixels.device).view(1, -1, 1, 1, 1)
|
| 345 |
+
|
| 346 |
+
pixels = (pixels.to(torch.float32).div(255.0) - pixel_mean) / pixel_std
|
| 347 |
+
posterior = vae.encode(pixels.to(vae.dtype), return_dict=False)[0]
|
| 348 |
+
latents = posterior.sample(generator=torch.Generator().manual_seed(encode_seed))
|
| 349 |
+
latents = latents.to(torch.float16).float().cpu()
|
| 350 |
+
return (latents - latents_mean) / latents_std
|
| 351 |
+
|
| 352 |
+
|
| 353 |
+
@spaces.GPU(duration=_gpu_duration, size=GPU_SIZE)
|
| 354 |
+
def _swap(ref_pixels, pixels, prompt_embeds, text_token_tags, height, width, num_frames, steps, seed):
|
| 355 |
+
"""Encode the reference image and the guide clip, run the joint denoising loop, decode the video."""
|
| 356 |
+
PIPE.vae.to("cuda")
|
| 357 |
+
free, total = torch.cuda.mem_get_info()
|
| 358 |
+
print(
|
| 359 |
+
f"[gpu] {free / 2**30:.1f} GiB free of {total / 2**30:.1f} GiB · alloc conf "
|
| 360 |
+
f"{os.environ.get('PYTORCH_CUDA_ALLOC_CONF')!r}",
|
| 361 |
+
flush=True,
|
| 362 |
+
)
|
| 363 |
+
started = time.time()
|
| 364 |
+
reference = encode_condition(PIPE.vae, ref_pixels.to("cuda"), PIPE.pixel_mean, PIPE.pixel_std, PIPE.keyframe_encode_seed)
|
| 365 |
+
del ref_pixels
|
| 366 |
+
|
| 367 |
+
pixels = pixels.to("cuda")
|
| 368 |
+
try:
|
| 369 |
+
guide = encode_condition(PIPE.vae, pixels, PIPE.pixel_mean, PIPE.pixel_std, PIPE.keyframe_encode_seed)
|
| 370 |
+
except RuntimeError as error:
|
| 371 |
+
# The failed attempt's traceback pins every intermediate activation alive; drop it before retrying on tiles.
|
| 372 |
+
error.__traceback__ = None
|
| 373 |
+
torch.cuda.empty_cache()
|
| 374 |
+
print(f"[gpu] guide encode failed ({type(error).__name__}); retrying on 128px tiles.", flush=True)
|
| 375 |
+
PIPE.vae.enable_tiling(tile_sample_min_height=128, tile_sample_min_width=128)
|
| 376 |
+
guide = encode_condition(PIPE.vae, pixels, PIPE.pixel_mean, PIPE.pixel_std, PIPE.keyframe_encode_seed)
|
| 377 |
+
del pixels
|
| 378 |
+
torch.cuda.empty_cache()
|
| 379 |
+
encoded_in = time.time() - started
|
| 380 |
+
print(
|
| 381 |
+
f"[gpu] encoded reference {tuple(reference.shape)} + guide {tuple(guide.shape)} in {encoded_in:.1f}s",
|
| 382 |
+
flush=True,
|
| 383 |
+
)
|
| 384 |
+
|
| 385 |
+
started = time.time()
|
| 386 |
+
state = PIPE(
|
| 387 |
+
prompt_embeds=prompt_embeds.to("cuda"),
|
| 388 |
+
text_token_tags=text_token_tags,
|
| 389 |
+
# Packed order: reference image(s) first, then the aligned guide — the layout reserves them in this order.
|
| 390 |
+
condition_latents=[reference, guide],
|
| 391 |
+
num_reference_images=1,
|
| 392 |
+
height=int(height),
|
| 393 |
+
width=int(width),
|
| 394 |
+
num_frames=int(num_frames),
|
| 395 |
+
num_inference_steps=int(steps) + 1,
|
| 396 |
+
generator=torch.Generator("cpu").manual_seed(int(seed)),
|
| 397 |
+
output_type="np",
|
| 398 |
+
)
|
| 399 |
+
return state.get("videos")[0], encoded_in, time.time() - started
|
| 400 |
+
|
| 401 |
+
|
| 402 |
+
def swap(
|
| 403 |
+
video: str | None,
|
| 404 |
+
reference: str | None,
|
| 405 |
+
details: str = "",
|
| 406 |
+
target: str = "",
|
| 407 |
+
remove_bg: bool = True,
|
| 408 |
+
clip_length: str = DEFAULT_CLIP,
|
| 409 |
+
detail: str = DEFAULT_DETAIL,
|
| 410 |
+
steps: int = DEFAULT_STEPS,
|
| 411 |
+
seed: int = DEFAULT_SEED,
|
| 412 |
+
progress=gr.Progress(track_tqdm=True),
|
| 413 |
+
):
|
| 414 |
+
"""Swap the main subject of a guide video for the character in a reference image.
|
| 415 |
+
|
| 416 |
+
Args:
|
| 417 |
+
video: The guide clip — the scene and the performance to keep. Any resolution / frame rate.
|
| 418 |
+
reference: The character image (`<Picture 1>`) whose identity, body and clothing replace the subject.
|
| 419 |
+
details: A short description of the reference character, e.g. "an elderly man with a grey beard".
|
| 420 |
+
target: Which guide subject to replace when more than one person appears, e.g. "the woman on the left".
|
| 421 |
+
remove_bg: Cut the reference onto neutral gray with `briaai/RMBG-2.0` first (recommended).
|
| 422 |
+
clip_length: How much of the guide to process, as a 17n + 5 frame count at 24 fps.
|
| 423 |
+
detail: Working pixel area; the whole pass runs here, guide and result alike.
|
| 424 |
+
steps: Euler steps. The shipped workflow uses 8.
|
| 425 |
+
seed: Noise seed.
|
| 426 |
+
|
| 427 |
+
Returns:
|
| 428 |
+
The swapped clip, with the guide's own soundtrack, and a one-line report of what ran.
|
| 429 |
+
"""
|
| 430 |
+
if LOAD_ERROR:
|
| 431 |
+
raise gr.Error(LOAD_ERROR.replace("**", ""))
|
| 432 |
+
if not video:
|
| 433 |
+
raise gr.Error("Upload a guide video — the scene and performance to keep.")
|
| 434 |
+
if not reference:
|
| 435 |
+
raise gr.Error("Upload a reference character image — the identity to swap in.")
|
| 436 |
+
|
| 437 |
+
max_length = CLIP_LENGTHS.get(clip_length, CLIP_LENGTHS[DEFAULT_CLIP])
|
| 438 |
+
megapixels = DETAIL_LEVELS.get(detail, DETAIL_LEVELS[DEFAULT_DETAIL])
|
| 439 |
+
|
| 440 |
+
progress(0.02, desc="Planning the guide clip ...")
|
| 441 |
+
plan = h3_video.plan_clip(video, megapixels, max_length)
|
| 442 |
+
height, width, num_frames = plan["height"], plan["width"], plan["num_frames"]
|
| 443 |
+
|
| 444 |
+
ip_token = _caller_ip_token()
|
| 445 |
+
if remove_bg:
|
| 446 |
+
progress(0.08, desc="Removing the reference background ...")
|
| 447 |
+
reference = remove_background(reference, ip_token)
|
| 448 |
+
|
| 449 |
+
details = (details or "").strip() or DEFAULT_DETAILS
|
| 450 |
+
target = (target or "").strip()
|
| 451 |
+
prompt = (PROMPT_TARGETED if target else PROMPT_MAIN).format(details=details, target=target)
|
| 452 |
+
|
| 453 |
+
progress(0.14, desc="Encoding the reference and caption ...")
|
| 454 |
+
prompt_embeds, text_token_tags = encode_prompt(prompt, reference, num_frames)
|
| 455 |
+
|
| 456 |
+
progress(0.2, desc="Sizing the reference image ...")
|
| 457 |
+
ref_pixels = h3_video.load_image(reference, REFERENCE_MEGAPIXELS)
|
| 458 |
+
|
| 459 |
+
progress(0.26, desc=f"Re-timing {plan['source_frames']} frames to 24 fps ...")
|
| 460 |
+
pixels = h3_video.load_guide(video, plan)
|
| 461 |
+
|
| 462 |
+
progress(0.34, desc=f"Swapping {num_frames} frames at {width}x{height} in {int(steps)} steps ...")
|
| 463 |
+
frames, encoded_in, denoised_in = _swap(
|
| 464 |
+
ref_pixels, pixels, prompt_embeds, text_token_tags, height, width, num_frames, int(steps), int(seed)
|
| 465 |
+
)
|
| 466 |
+
|
| 467 |
+
progress(0.94, desc="Encoding the result ...")
|
| 468 |
+
stem = os.path.join(OUTPUT_DIR, uuid.uuid4().hex)
|
| 469 |
+
silent = h3_video.write_video(frames, f"{stem}.mp4")
|
| 470 |
+
out = h3_video.mux_source_audio(silent, video, num_frames, f"{stem}-audio.mp4")
|
| 471 |
+
|
| 472 |
+
report = (
|
| 473 |
+
f"**{width}x{height} · {num_frames} frames ({num_frames / FPS:.2f} s @ {FPS} fps)** — "
|
| 474 |
+
f"guide {plan['source_width']}x{plan['source_height']} @{plan['source_fps']:.4g} fps · "
|
| 475 |
+
f"encode {encoded_in:.1f} s · {int(steps)} steps in {denoised_in:.1f} s · seed {int(seed)}"
|
| 476 |
+
+ ("" if out is not silent else " · no soundtrack in the guide")
|
| 477 |
+
)
|
| 478 |
+
return out, report
|
| 479 |
+
|
| 480 |
+
|
| 481 |
+
# ── UI ──────────────────────────────────────────────────────────────────────
|
| 482 |
+
|
| 483 |
+
CSS = """
|
| 484 |
+
.contain { max-width: 1200px !important; margin: auto; }
|
| 485 |
+
#col-out video { max-height: 560px; }
|
| 486 |
+
"""
|
| 487 |
+
|
| 488 |
+
with gr.Blocks(title="MiniMax-H3 · character swap") as demo:
|
| 489 |
+
gr.Markdown(
|
| 490 |
+
"""
|
| 491 |
+
# MiniMax-H3 — character swap
|
| 492 |
+
`minimax_h3_character_swap_v1.0_ref2va_r64_gl` from [`Alissonerdx/Minimax-H3-ComfyUI`](https://huggingface.co/Alissonerdx/Minimax-H3-ComfyUI)
|
| 493 |
+
**replaces the main subject of a guide video with the character in a reference image**, keeping the guide's
|
| 494 |
+
scene, camera, motion and performance. The guide is packed as an aligned
|
| 495 |
+
[MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) guide latent (`MiniMaxH3AddGuide`), and the
|
| 496 |
+
character enters on H3's native image-reference path as `<Picture 1>`. Output resolution follows the guide;
|
| 497 |
+
the soundtrack is the guide's own.
|
| 498 |
+
"""
|
| 499 |
+
)
|
| 500 |
+
|
| 501 |
+
with gr.Row():
|
| 502 |
+
with gr.Column():
|
| 503 |
+
video = gr.Video(label="Guide video (scene + performance)", height=300)
|
| 504 |
+
reference = gr.Image(label="Reference character (Picture 1)", type="filepath", height=240)
|
| 505 |
+
run = gr.Button("Swap character", variant="primary")
|
| 506 |
+
details = gr.Textbox(
|
| 507 |
+
label="Reference description",
|
| 508 |
+
placeholder="an elderly man with a grey beard and a dark jacket",
|
| 509 |
+
lines=2,
|
| 510 |
+
info="Describe the reference character. Clear descriptions help the swap hold identity.",
|
| 511 |
+
)
|
| 512 |
+
with gr.Accordion("Advanced", open=False):
|
| 513 |
+
target = gr.Textbox(
|
| 514 |
+
label="Which subject to replace (optional)",
|
| 515 |
+
placeholder="the woman on the left",
|
| 516 |
+
info="Only needed when several people appear; otherwise the most prominent subject is swapped.",
|
| 517 |
+
)
|
| 518 |
+
remove_bg = gr.Checkbox(
|
| 519 |
+
label="Remove reference background (briaai/RMBG-2.0)", value=True,
|
| 520 |
+
info="Cuts the subject onto neutral gray first, as the ComfyUI workflow does.",
|
| 521 |
+
)
|
| 522 |
+
with gr.Row():
|
| 523 |
+
clip_length = gr.Dropdown(
|
| 524 |
+
label="Clip length", choices=list(CLIP_LENGTHS), value=DEFAULT_CLIP,
|
| 525 |
+
info="Trimmed from the start. Longer costs quadratically more.",
|
| 526 |
+
)
|
| 527 |
+
detail = gr.Dropdown(
|
| 528 |
+
label="Detail level", choices=list(DETAIL_LEVELS), value=DEFAULT_DETAIL,
|
| 529 |
+
info="Working pixel area, on the guide's own aspect ratio.",
|
| 530 |
+
)
|
| 531 |
+
steps = gr.Slider(
|
| 532 |
+
label="Steps", minimum=4, maximum=16, step=1, value=DEFAULT_STEPS,
|
| 533 |
+
info="Euler steps. The shipped workflow uses 8 on top of a few-step turbo LoRA.",
|
| 534 |
+
)
|
| 535 |
+
seed = gr.Slider(label="Seed", minimum=0, maximum=2**31 - 1, step=1, value=DEFAULT_SEED)
|
| 536 |
+
with gr.Column(elem_id="col-out"):
|
| 537 |
+
result = gr.Video(label="Swapped", height=300, autoplay=True)
|
| 538 |
+
report = gr.Markdown()
|
| 539 |
+
|
| 540 |
+
gr.Markdown(
|
| 541 |
+
"""
|
| 542 |
+
---
|
| 543 |
+
**How it runs.** Weights are `multimodalart/MiniMax-H3-Pruned`'s `transformer_ref` (the `ref2va` partition
|
| 544 |
+
with rank-8 AdaLN projections), with the Character Swap LoRA and
|
| 545 |
+
[`lightx2v/Minimax-h3-Turbo`](https://huggingface.co/lightx2v/Minimax-h3-Turbo)'s 4-step LoRA both folded in
|
| 546 |
+
at strength 1.0. The guide clip is prepared by
|
| 547 |
+
[`ostris/ComfyUI-AIToolkit-MiniMaxH3`](https://github.com/ostris/ComfyUI-AIToolkit-MiniMaxH3)'s recipe and
|
| 548 |
+
packed on the target's own grid; the reference image is background-removed by
|
| 549 |
+
[`briaai/RMBG-2.0`](https://huggingface.co/briaai/RMBG-2.0) and encoded — as `<Picture 1>` vision tokens and
|
| 550 |
+
a VAE reference block — by [`multimodalart/qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner).
|
| 551 |
+
The repo's optional BFS multi-shot planner is not ported: this is one guide, one reference, one swap.
|
| 552 |
+
|
| 553 |
+
Use only with appropriate rights and consent for the reference images and guide videos; not for
|
| 554 |
+
impersonation, fraud, harassment, or other unlawful purposes. See the
|
| 555 |
+
[Character Swap license](https://huggingface.co/Alissonerdx/Minimax-H3-ComfyUI/blob/main/LICENSE_CHARACTER_SWAP.md).
|
| 556 |
+
"""
|
| 557 |
+
)
|
| 558 |
+
|
| 559 |
+
run.click(
|
| 560 |
+
swap,
|
| 561 |
+
inputs=[video, reference, details, target, remove_bg, clip_length, detail, steps, seed],
|
| 562 |
+
outputs=[result, report],
|
| 563 |
+
api_name="swap",
|
| 564 |
+
)
|
| 565 |
+
|
| 566 |
+
if load_models():
|
| 567 |
+
print(LOAD_ERROR, flush=True)
|
| 568 |
+
|
| 569 |
+
if __name__ == "__main__":
|
| 570 |
+
demo.launch(
|
| 571 |
+
theme=gr.themes.Citrus(),
|
| 572 |
+
css=CSS,
|
| 573 |
+
show_error=True,
|
| 574 |
+
allowed_paths=[OUTPUT_DIR],
|
| 575 |
+
mcp_server=True,
|
| 576 |
+
)
|
h3_charswap_blocks.py
ADDED
|
@@ -0,0 +1,294 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""The **character-swap** conditioning branch of MiniMax-H3, which diffusers itself does not carry.
|
| 2 |
+
|
| 3 |
+
`Alissonerdx/Minimax-H3-ComfyUI`'s Character Swap LoRA
|
| 4 |
+
(`loras/minimax_h3_character_swap_v1.0_ref2va_r64_gl.safetensors`) replaces the main subject of a
|
| 5 |
+
*guide* video with the character in a *reference image*, while keeping the guide's scene, camera and
|
| 6 |
+
performance. It is trained on the `ref2va` partition and conditions through **two** visual blocks at
|
| 7 |
+
once, which is exactly what its own training metadata records:
|
| 8 |
+
|
| 9 |
+
minimax_h3_reference_rope = {"guide_rope_layout": "overlap", "reference_rope_layout": "native", ...}
|
| 10 |
+
minimax_h3_dataset_control_roles = [["guide", "reference", "reference"]]
|
| 11 |
+
guide_latent_only = true
|
| 12 |
+
|
| 13 |
+
* **guide — `overlap`.** The whole guide clip, VAE-encoded at the target resolution and packed on the
|
| 14 |
+
target's *own* `(t, h, w)` rotary grid: guide token `(t, y, x)` sits exactly where target token
|
| 15 |
+
`(t, y, x)` sits. This is ComfyUI's `MiniMaxH3AddGuide(frame_idx = 0)` and is identical to the
|
| 16 |
+
aligned guide the LMS sharpening Space uses — see the sibling `MiniMaxH3GuidePrepareLayoutStep`.
|
| 17 |
+
* **reference — `native`.** The character image on H3's native image-reference path: a Qwen3-VL
|
| 18 |
+
`<Picture 1>` vision block inside `prompt_embeds` (produced upstream by the conditioner Space's
|
| 19 |
+
`/encode_ref2va`) **and** a VAE-encoded reference block in the sequence, laid out exactly as
|
| 20 |
+
`diffusers`' own `MiniMaxH3Ref2VAPrepareLayoutStep` lays out an image reference — one integer rotary
|
| 21 |
+
slot, its own aspect-normalized spatial grid.
|
| 22 |
+
|
| 23 |
+
So the packed layout is `[text (+ Picture 1 vision) | reference image | guide | target audio | target
|
| 24 |
+
video]`. The reference image takes the single integer rotary slot `num_text_tokens`; everything after
|
| 25 |
+
it shares the origin `num_text_tokens + num_references`, so the guide and the target overlap each other
|
| 26 |
+
on that one grid. With no references this reduces byte for byte to the aligned-guide layout.
|
| 27 |
+
|
| 28 |
+
Everything downstream is stock diffusers and needs no change:
|
| 29 |
+
|
| 30 |
+
* `MiniMaxH3PrepareLatentsStep` draws one noise per entry of `condition_latents`, in packed order, noises
|
| 31 |
+
each to `t = keyframe_noise_aug` (0.999) and packs them ahead of the target rows. We hand it
|
| 32 |
+
`[reference_latents, guide_latents]`, so the reference rows land first and the guide rows next, which
|
| 33 |
+
is the order this layout reserves.
|
| 34 |
+
* `MiniMaxH3SetTimestepsStep` pins every one of the `num_condition_video_rows` leading video rows —
|
| 35 |
+
reference and guide alike — at `max(timestep, keyframe_noise_aug)` for every step, so both survive the
|
| 36 |
+
loop as conditioning.
|
| 37 |
+
* `MiniMaxH3Ref2VADenoiseStep` runs the loop against the `transformer_ref` partition the LoRA is trained on.
|
| 38 |
+
"""
|
| 39 |
+
|
| 40 |
+
import numpy as np
|
| 41 |
+
import torch
|
| 42 |
+
from diffusers.modular_pipelines.minimax_h3.before_denoise import (
|
| 43 |
+
MiniMaxH3PrepareLatentsStep,
|
| 44 |
+
MiniMaxH3PrepareLayoutStep,
|
| 45 |
+
MiniMaxH3SetTimestepsStep,
|
| 46 |
+
_fill_audio_positions,
|
| 47 |
+
_frame_position_grid,
|
| 48 |
+
_temporal_position_grid,
|
| 49 |
+
)
|
| 50 |
+
from diffusers.modular_pipelines.minimax_h3.decoders import (
|
| 51 |
+
MiniMaxH3AfterDenoiseStep,
|
| 52 |
+
MiniMaxH3VideoDecodeStep,
|
| 53 |
+
)
|
| 54 |
+
from diffusers.modular_pipelines.minimax_h3.denoise import MiniMaxH3Ref2VADenoiseStep
|
| 55 |
+
from diffusers.modular_pipelines.minimax_h3.modular_pipeline import (
|
| 56 |
+
MINIMAX_H3_AUDIO_CHANNELS,
|
| 57 |
+
MINIMAX_H3_AUDIO_TAG,
|
| 58 |
+
MINIMAX_H3_VIDEO_TAG,
|
| 59 |
+
align_num_frames,
|
| 60 |
+
audio_latent_num_frames,
|
| 61 |
+
resolve_canvas_size,
|
| 62 |
+
video_latent_num_frames,
|
| 63 |
+
)
|
| 64 |
+
from diffusers.modular_pipelines.modular_pipeline import SequentialPipelineBlocks
|
| 65 |
+
from diffusers.modular_pipelines.modular_pipeline_utils import InputParam
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
class MiniMaxH3CharswapPrepareLayoutStep(MiniMaxH3PrepareLayoutStep):
|
| 69 |
+
"""`MiniMaxH3PrepareLayoutStep` building the `[text | reference image | guide | audio | video]` layout.
|
| 70 |
+
|
| 71 |
+
The reference image(s) use H3's native image-reference rope; the guide overlaps the target grid. The
|
| 72 |
+
reference geometry is read off `condition_latents`, whose leading entries are the reference latents and
|
| 73 |
+
whose last entry is the aligned guide.
|
| 74 |
+
"""
|
| 75 |
+
|
| 76 |
+
model_name = "minimax-h3"
|
| 77 |
+
|
| 78 |
+
@property
|
| 79 |
+
def description(self) -> str:
|
| 80 |
+
return (
|
| 81 |
+
"Resolves the geometry of a character-swap request and builds the `[text | reference image | "
|
| 82 |
+
"guide | target audio | target video]` packed layout: the reference image on H3's native "
|
| 83 |
+
"image-reference rope, the guide overlapping the target's own grid (ComfyUI's "
|
| 84 |
+
"`MiniMaxH3AddGuide` at `frame_idx = 0`)."
|
| 85 |
+
)
|
| 86 |
+
|
| 87 |
+
@property
|
| 88 |
+
def inputs(self) -> list[InputParam]:
|
| 89 |
+
return super().inputs + [
|
| 90 |
+
InputParam(
|
| 91 |
+
name="condition_latents",
|
| 92 |
+
type_hint=list,
|
| 93 |
+
required=True,
|
| 94 |
+
description=(
|
| 95 |
+
"Encoded conditioning latents in packed order: one `(1, C, 1, h, w)` tensor per reference "
|
| 96 |
+
"image, then the aligned guide `(1, C, num_latent_frames, latent_h, latent_w)`."
|
| 97 |
+
),
|
| 98 |
+
),
|
| 99 |
+
InputParam(
|
| 100 |
+
name="num_reference_images",
|
| 101 |
+
type_hint=int,
|
| 102 |
+
default=1,
|
| 103 |
+
description="How many leading entries of `condition_latents` are reference images (the rest is the guide).",
|
| 104 |
+
),
|
| 105 |
+
]
|
| 106 |
+
|
| 107 |
+
def build_charswap_sequence(
|
| 108 |
+
self,
|
| 109 |
+
text_token_tags: torch.Tensor,
|
| 110 |
+
reference_shapes: list[tuple[int, int, int]],
|
| 111 |
+
num_latent_frames: int,
|
| 112 |
+
latent_height: int,
|
| 113 |
+
latent_width: int,
|
| 114 |
+
num_audio_latents: int,
|
| 115 |
+
patch_size: tuple[int, int, int],
|
| 116 |
+
):
|
| 117 |
+
r"""Build `[text (+Picture vision) | reference image(s) | guide | target audio | target video]`.
|
| 118 |
+
|
| 119 |
+
`reference_shapes` is one `(latent_frames, latent_h, latent_w)` per reference image, read off the
|
| 120 |
+
encoded reference latents so the reserved rows can never disagree with what the encoder produced.
|
| 121 |
+
"""
|
| 122 |
+
_, patch_h, patch_w = patch_size
|
| 123 |
+
num_text_tokens = text_token_tags.shape[0]
|
| 124 |
+
rows_per_frame = (latent_height // patch_h) * (latent_width // patch_w)
|
| 125 |
+
|
| 126 |
+
reference_rows = [
|
| 127 |
+
frames * (height // patch_h) * (width // patch_w) for frames, height, width in reference_shapes
|
| 128 |
+
]
|
| 129 |
+
num_reference_video_rows = int(sum(reference_rows))
|
| 130 |
+
num_guide_rows = num_latent_frames * rows_per_frame
|
| 131 |
+
num_condition_video_rows = num_reference_video_rows + num_guide_rows
|
| 132 |
+
num_audio_rows = num_audio_latents * MINIMAX_H3_AUDIO_CHANNELS
|
| 133 |
+
num_video_rows = num_latent_frames * rows_per_frame
|
| 134 |
+
sequence_length = (
|
| 135 |
+
num_text_tokens + num_reference_video_rows + num_guide_rows + num_audio_rows + num_video_rows
|
| 136 |
+
)
|
| 137 |
+
|
| 138 |
+
position_ids = torch.zeros(sequence_length, 3, dtype=torch.float64)
|
| 139 |
+
position_ids[:num_text_tokens, 0] = torch.arange(num_text_tokens, dtype=torch.float64)
|
| 140 |
+
|
| 141 |
+
video_index_blocks = []
|
| 142 |
+
cursor = num_text_tokens
|
| 143 |
+
rotary_time = float(num_text_tokens)
|
| 144 |
+
|
| 145 |
+
# Reference images — native rope. Each image is a single frame on its own aspect-normalized grid and
|
| 146 |
+
# takes one integer rotary slot, exactly as `MiniMaxH3Ref2VAPrepareLayoutStep` lays an image out.
|
| 147 |
+
for (frames, ref_h, ref_w), rows_count in zip(reference_shapes, reference_rows):
|
| 148 |
+
rows = slice(cursor, cursor + rows_count)
|
| 149 |
+
cursor = rows.stop
|
| 150 |
+
frame_grid, _ = _frame_position_grid(ref_h, ref_w, patch_h, patch_w)
|
| 151 |
+
position_ids[rows, 0] = rotary_time
|
| 152 |
+
position_ids[rows, 1:] = frame_grid.repeat(frames, 1)
|
| 153 |
+
video_index_blocks.append(torch.arange(rows.start, rows.stop))
|
| 154 |
+
rotary_time += 1.0
|
| 155 |
+
|
| 156 |
+
# Everything after the references shares this origin: the guide overlaps the target, both built on the
|
| 157 |
+
# target's own `(t, h, w)` grid starting here.
|
| 158 |
+
origin = rotary_time
|
| 159 |
+
target_frame_grid, target_width_grid = _frame_position_grid(latent_height, latent_width, patch_h, patch_w)
|
| 160 |
+
media_time = _temporal_position_grid(num_latent_frames, origin)
|
| 161 |
+
media_position_t = media_time.repeat_interleave(target_frame_grid.shape[0])
|
| 162 |
+
media_position_hw = target_frame_grid.repeat(num_latent_frames, 1)
|
| 163 |
+
|
| 164 |
+
# Guide block (overlap).
|
| 165 |
+
guide_rows = slice(cursor, cursor + num_guide_rows)
|
| 166 |
+
cursor = guide_rows.stop
|
| 167 |
+
position_ids[guide_rows, 0] = media_position_t
|
| 168 |
+
position_ids[guide_rows, 1:] = media_position_hw
|
| 169 |
+
video_index_blocks.append(torch.arange(guide_rows.start, guide_rows.stop))
|
| 170 |
+
|
| 171 |
+
# Target audio, then target video (overlapping the guide on the same grid).
|
| 172 |
+
audio_start = cursor
|
| 173 |
+
video_start = audio_start + num_audio_rows
|
| 174 |
+
_fill_audio_positions(
|
| 175 |
+
position_ids, slice(audio_start, video_start), num_audio_latents, origin, target_width_grid
|
| 176 |
+
)
|
| 177 |
+
position_ids[video_start:, 0] = media_position_t
|
| 178 |
+
position_ids[video_start:, 1:] = media_position_hw
|
| 179 |
+
|
| 180 |
+
video_indices = torch.cat(video_index_blocks + [torch.arange(video_start, sequence_length)])
|
| 181 |
+
audio_indices = torch.arange(audio_start, video_start)
|
| 182 |
+
text_indices = torch.arange(num_text_tokens)
|
| 183 |
+
|
| 184 |
+
token_tags = torch.empty(sequence_length, dtype=torch.long)
|
| 185 |
+
token_tags[text_indices] = text_token_tags.to(torch.long)
|
| 186 |
+
token_tags[audio_indices] = MINIMAX_H3_AUDIO_TAG
|
| 187 |
+
token_tags[video_indices] = MINIMAX_H3_VIDEO_TAG
|
| 188 |
+
|
| 189 |
+
return position_ids, token_tags, video_indices, audio_indices, text_indices, num_condition_video_rows, 0
|
| 190 |
+
|
| 191 |
+
@torch.no_grad()
|
| 192 |
+
def __call__(self, components, state):
|
| 193 |
+
block_state = self.get_block_state(state)
|
| 194 |
+
device = components._execution_device
|
| 195 |
+
|
| 196 |
+
if block_state.height is None:
|
| 197 |
+
block_state.height, block_state.width = resolve_canvas_size(16, 9, components.canvas_multiple)
|
| 198 |
+
if block_state.height % components.canvas_multiple or block_state.width % components.canvas_multiple:
|
| 199 |
+
raise ValueError(
|
| 200 |
+
f"`height` and `width` must be multiples of {components.canvas_multiple}, got "
|
| 201 |
+
f"{block_state.height}x{block_state.width}."
|
| 202 |
+
)
|
| 203 |
+
|
| 204 |
+
frames_per_chunk = components.vae_frames_per_chunk
|
| 205 |
+
latents_per_chunk = components.vae_latents_per_chunk
|
| 206 |
+
block_state.num_frames = align_num_frames(block_state.num_frames, frames_per_chunk, latents_per_chunk)
|
| 207 |
+
duration = block_state.num_frames / components.fps
|
| 208 |
+
if not components.min_duration <= duration <= components.max_duration:
|
| 209 |
+
raise ValueError(
|
| 210 |
+
f"MiniMax-H3 generates between {components.min_duration} and {components.max_duration} seconds "
|
| 211 |
+
f"at {components.fps} fps, got {block_state.num_frames} frames ({duration:g} s)."
|
| 212 |
+
)
|
| 213 |
+
|
| 214 |
+
ratio = components.vae_spatial_compression_ratio
|
| 215 |
+
block_state.num_latent_frames = video_latent_num_frames(
|
| 216 |
+
block_state.num_frames, frames_per_chunk, latents_per_chunk
|
| 217 |
+
)
|
| 218 |
+
block_state.latent_height = block_state.height // ratio
|
| 219 |
+
block_state.latent_width = block_state.width // ratio
|
| 220 |
+
block_state.num_audio_latents = audio_latent_num_frames(block_state.num_frames)
|
| 221 |
+
|
| 222 |
+
num_reference_images = int(getattr(block_state, "num_reference_images", 1) or 0)
|
| 223 |
+
reference_shapes = [
|
| 224 |
+
tuple(int(dim) for dim in latents.shape[2:5])
|
| 225 |
+
for latents in block_state.condition_latents[:num_reference_images]
|
| 226 |
+
]
|
| 227 |
+
|
| 228 |
+
(
|
| 229 |
+
position_ids,
|
| 230 |
+
token_tags,
|
| 231 |
+
video_indices,
|
| 232 |
+
audio_indices,
|
| 233 |
+
text_indices,
|
| 234 |
+
block_state.num_condition_video_rows,
|
| 235 |
+
block_state.num_condition_audio_rows,
|
| 236 |
+
) = self.build_charswap_sequence(
|
| 237 |
+
block_state.text_token_tags,
|
| 238 |
+
reference_shapes,
|
| 239 |
+
block_state.num_latent_frames,
|
| 240 |
+
block_state.latent_height,
|
| 241 |
+
block_state.latent_width,
|
| 242 |
+
block_state.num_audio_latents,
|
| 243 |
+
components.patch_size,
|
| 244 |
+
)
|
| 245 |
+
block_state.position_ids = position_ids.to(device)
|
| 246 |
+
block_state.token_tags = token_tags.to(device)
|
| 247 |
+
block_state.video_indices = video_indices.to(device)
|
| 248 |
+
block_state.audio_indices = audio_indices.to(device)
|
| 249 |
+
block_state.text_indices = text_indices.to(device)
|
| 250 |
+
|
| 251 |
+
self.set_block_state(state, block_state)
|
| 252 |
+
return components, state
|
| 253 |
+
|
| 254 |
+
|
| 255 |
+
class MiniMaxH3CharswapCoreDenoiseStep(SequentialPipelineBlocks):
|
| 256 |
+
"""The character-swap denoising core: combined layout, stock latents / timesteps, `transformer_ref` loop."""
|
| 257 |
+
|
| 258 |
+
model_name = "minimax-h3"
|
| 259 |
+
block_classes = [
|
| 260 |
+
MiniMaxH3CharswapPrepareLayoutStep,
|
| 261 |
+
MiniMaxH3PrepareLatentsStep,
|
| 262 |
+
MiniMaxH3SetTimestepsStep,
|
| 263 |
+
MiniMaxH3Ref2VADenoiseStep,
|
| 264 |
+
]
|
| 265 |
+
block_names = ["prepare_layout", "prepare_latents", "set_timesteps", "denoise"]
|
| 266 |
+
|
| 267 |
+
@property
|
| 268 |
+
def description(self):
|
| 269 |
+
return (
|
| 270 |
+
"Core denoising workflow for a character-swap request: builds the `[text | reference image | "
|
| 271 |
+
"guide | target audio | target video]` layout, packs the reference and guide conditioning latents, "
|
| 272 |
+
"plans the per-row timesteps and runs the loop against the `transformer_ref` partition."
|
| 273 |
+
)
|
| 274 |
+
|
| 275 |
+
|
| 276 |
+
class MiniMaxH3CharswapGeneratorBlocks(SequentialPipelineBlocks):
|
| 277 |
+
"""The denoising half of a split character-swap deployment.
|
| 278 |
+
|
| 279 |
+
`prompt_embeds` / `text_token_tags` (with the `<Picture 1>` vision block) arrive over the wire from the
|
| 280 |
+
conditioner Space, and both visual conditions arrive already VAE-encoded as `condition_latents` —
|
| 281 |
+
`[reference_image_latents, guide_latents]` — so neither the 62 GiB Qwen3-VL nor a resize step is needed
|
| 282 |
+
here. The soundtrack is the guide clip's own, muxed by the caller, so the audio VAE is not loaded either.
|
| 283 |
+
"""
|
| 284 |
+
|
| 285 |
+
model_name = "minimax-h3"
|
| 286 |
+
block_classes = [MiniMaxH3CharswapCoreDenoiseStep, MiniMaxH3AfterDenoiseStep, MiniMaxH3VideoDecodeStep]
|
| 287 |
+
block_names = ["denoise", "after_denoise", "decode"]
|
| 288 |
+
|
| 289 |
+
@property
|
| 290 |
+
def description(self):
|
| 291 |
+
return (
|
| 292 |
+
"The denoising half of a split MiniMax-H3 character-swap deployment: combined reference/guide "
|
| 293 |
+
"layout, joint audio/video denoising against `transformer_ref`, and a video-only decode."
|
| 294 |
+
)
|
h3_lora.py
ADDED
|
@@ -0,0 +1,138 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Fold a ComfyUI-format MiniMax-H3 LoRA into the diffusers `MiniMaxH3Transformer3DModel` module tree.
|
| 2 |
+
|
| 3 |
+
Both LoRAs this Space stacks are exported for ComfyUI and name the *reference* implementation's modules
|
| 4 |
+
(`blocks.N.attn.qkv_proj`, `mlp.fc1` / `fc2`, `attn.out_proj`, `adaln_proj.linear`), so each one has to be replayed
|
| 5 |
+
through the same layout transforms `scripts/convert_minimax_h3_to_diffusers.py` applies to the base weights — a
|
| 6 |
+
delta is only valid in the layout of the weight it is added to.
|
| 7 |
+
|
| 8 |
+
| ComfyUI target | diffusers parameter | transform |
|
| 9 |
+
|---|---|---|
|
| 10 |
+
| `blocks.N.…` | `transformer_blocks.N.…` | rename |
|
| 11 |
+
| `token_refiner.blocks.N.…` | `token_refiner.refiner_blocks.N.…` | rename |
|
| 12 |
+
| `attn.out_proj` | `attn.to_out.0` | rename |
|
| 13 |
+
| `mlp.fc2` | `ff.net.2` | rename |
|
| 14 |
+
| `adaln_proj.linear` | `adaln_proj.linear` | identity |
|
| 15 |
+
| `final_layer.adaln_proj.linear` | `norm_out.linear` | rename |
|
| 16 |
+
| `mlp.fc1` | `ff.net.0.proj` | fused halves swapped, ComfyUI `[gate; value]` -> diffusers `[value; gate]` |
|
| 17 |
+
| `attn.qkv_proj` | `attn.to_q` / `to_k` / `to_v` | split into contiguous thirds of `inner_dim` rows |
|
| 18 |
+
|
| 19 |
+
Two details that are easy to get wrong, and that the LoRAs themselves document:
|
| 20 |
+
|
| 21 |
+
* **SwiGLU order.** `lightx2v/Minimax-h3-Turbo`'s own metadata records the mapping it applied on export as
|
| 22 |
+
`"Diffusers [value;gate] -> ComfyUI [gate;value]"`, so coming back the other way the two halves swap.
|
| 23 |
+
* **QKV rows are already contiguous.** Raw `MiniMaxAI/MiniMax-H3` shards store fused QKV rows *per-head
|
| 24 |
+
interleaved*, and a LoRA trained against those has to be de-interleaved first. A ComfyUI checkpoint does not:
|
| 25 |
+
both ComfyUI (`comfy/ldm/minimax/model.py`) and ai-toolkit read the projection with
|
| 26 |
+
`.split(heads * head_dim, dim=-1)` / `.chunk(3, dim=-1)`, i.e. `[q_all; k_all; v_all]`. Both LoRAs here are
|
| 27 |
+
ComfyUI exports, so the thirds are contiguous and de-interleaving them would scatter every head.
|
| 28 |
+
|
| 29 |
+
The fold is `W' = W + scale * (lora_B @ lora_A)`, accumulated in float32, with `scale = alpha / rank` read off the
|
| 30 |
+
file's own `.alpha` scalars when it carries them (`lightx2v`'s does: `alpha 8 / rank 128`, and `alpha 24 / rank 384`
|
| 31 |
+
on the block-diagonal fused QKV, both `0.0625`) and `1.0` when it does not (ai-toolkit writes no alphas, so alpha
|
| 32 |
+
equals rank). Merging rather than attaching a PEFT adapter keeps the served graph a plain transformer, which is what
|
| 33 |
+
ZeroGPU's weight packing and any later AoTI export want.
|
| 34 |
+
|
| 35 |
+
Every LoRA target has to resolve, or loading fails: a silently skipped target means the name mapping is wrong and
|
| 36 |
+
the Space would serve a half-applied adapter.
|
| 37 |
+
"""
|
| 38 |
+
|
| 39 |
+
import torch
|
| 40 |
+
from safetensors.torch import load_file
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def _diffusers_stem(name: str) -> str:
|
| 44 |
+
"""Map a ComfyUI module path (already stripped of `diffusion_model.`) onto the diffusers tree."""
|
| 45 |
+
if name.startswith("token_refiner.blocks."):
|
| 46 |
+
stem = name.replace("token_refiner.blocks.", "token_refiner.refiner_blocks.", 1)
|
| 47 |
+
elif name.startswith("blocks."):
|
| 48 |
+
stem = name.replace("blocks.", "transformer_blocks.", 1)
|
| 49 |
+
else:
|
| 50 |
+
stem = name
|
| 51 |
+
return stem.replace("final_layer.adaln_proj.linear", "norm_out.linear")
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def _targets(name: str, lora_b: torch.Tensor, inner_dim: int):
|
| 55 |
+
"""Yield `(diffusers parameter key, row-transformed lora_B)` for one ComfyUI LoRA target."""
|
| 56 |
+
stem = _diffusers_stem(name)
|
| 57 |
+
|
| 58 |
+
if stem.endswith(".attn.qkv_proj"):
|
| 59 |
+
prefix = stem.removesuffix("qkv_proj")
|
| 60 |
+
if lora_b.shape[0] != 3 * inner_dim:
|
| 61 |
+
raise ValueError(
|
| 62 |
+
f"{name}: fused QKV lora_B has {lora_b.shape[0]} rows, expected 3 * {inner_dim}. This does not look "
|
| 63 |
+
"like a ComfyUI-layout MiniMax-H3 LoRA."
|
| 64 |
+
)
|
| 65 |
+
for projection, rows in zip(("to_q", "to_k", "to_v"), lora_b.split(inner_dim, dim=0)):
|
| 66 |
+
yield f"{prefix}{projection}.weight", rows
|
| 67 |
+
elif stem.endswith(".mlp.fc1"):
|
| 68 |
+
gate, value = lora_b.chunk(2, dim=0)
|
| 69 |
+
yield stem.replace(".mlp.fc1", ".ff.net.0.proj") + ".weight", torch.cat([value, gate])
|
| 70 |
+
elif stem.endswith(".mlp.fc2"):
|
| 71 |
+
yield stem.replace(".mlp.fc2", ".ff.net.2") + ".weight", lora_b
|
| 72 |
+
elif stem.endswith(".attn.out_proj"):
|
| 73 |
+
yield stem.replace(".attn.out_proj", ".attn.to_out.0") + ".weight", lora_b
|
| 74 |
+
else:
|
| 75 |
+
# `adaln_proj.linear` (per block) and `norm_out.linear`: same row layout on both sides.
|
| 76 |
+
yield stem + ".weight", lora_b
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def fold_comfy_lora(transformer, path: str, strength: float = 1.0, label: str = "lora") -> str:
|
| 80 |
+
"""Merge one ComfyUI-format MiniMax-H3 LoRA into `transformer`'s weights in place.
|
| 81 |
+
|
| 82 |
+
Args:
|
| 83 |
+
transformer: A `MiniMaxH3Transformer3DModel` (or the AdaLN-pruned subclass of it).
|
| 84 |
+
path: Local path to the LoRA `.safetensors`.
|
| 85 |
+
strength: The ComfyUI `LoraLoaderModelOnly` strength, applied on top of the file's own `alpha / rank`.
|
| 86 |
+
label: Name used in the returned status line.
|
| 87 |
+
|
| 88 |
+
Returns:
|
| 89 |
+
A one-line status string describing what was folded.
|
| 90 |
+
"""
|
| 91 |
+
lora = load_file(path)
|
| 92 |
+
heads = transformer.config.num_attention_heads
|
| 93 |
+
inner_dim = heads * transformer.config.attention_head_dim
|
| 94 |
+
|
| 95 |
+
bases = sorted({key[: -len(".lora_A.weight")] for key in lora if key.endswith(".lora_A.weight")})
|
| 96 |
+
if not bases:
|
| 97 |
+
raise ValueError(f"{path} holds no `.lora_A.weight` tensors; this is not a ComfyUI LoRA.")
|
| 98 |
+
|
| 99 |
+
params = dict(transformer.named_parameters())
|
| 100 |
+
folded, ranks, scales, missed = 0, set(), set(), []
|
| 101 |
+
|
| 102 |
+
for base in bases:
|
| 103 |
+
lora_a = lora[f"{base}.lora_A.weight"]
|
| 104 |
+
lora_b = lora.get(f"{base}.lora_B.weight")
|
| 105 |
+
if lora_b is None:
|
| 106 |
+
raise ValueError(f"{base} has a lora_A with no lora_B twin in {path}.")
|
| 107 |
+
rank = int(lora_a.shape[0])
|
| 108 |
+
ranks.add(rank)
|
| 109 |
+
alpha = lora.get(f"{base}.alpha")
|
| 110 |
+
scale = strength * (float(alpha) / rank if alpha is not None else 1.0)
|
| 111 |
+
scales.add(round(scale, 6))
|
| 112 |
+
|
| 113 |
+
name = base.removeprefix("diffusion_model.")
|
| 114 |
+
for key, rows in _targets(name, lora_b, inner_dim):
|
| 115 |
+
param = params.get(key)
|
| 116 |
+
if param is None:
|
| 117 |
+
missed.append(key)
|
| 118 |
+
continue
|
| 119 |
+
# float32 accumulation: a bf16 base would otherwise swallow most of the update.
|
| 120 |
+
delta = scale * (rows.to(torch.float32) @ lora_a.to(torch.float32))
|
| 121 |
+
if delta.shape != param.shape:
|
| 122 |
+
raise ValueError(
|
| 123 |
+
f"LoRA delta for `{key}` is {tuple(delta.shape)} but the base weight is {tuple(param.shape)}. "
|
| 124 |
+
"The LoRA and this transformer disagree on layout."
|
| 125 |
+
)
|
| 126 |
+
param.data = (param.data.float() + delta.to(param.device)).to(param.dtype)
|
| 127 |
+
folded += 1
|
| 128 |
+
|
| 129 |
+
if missed:
|
| 130 |
+
raise ValueError(
|
| 131 |
+
f"{len(missed)} targets of {path} matched no transformer parameter, e.g. {missed[:5]}. The LoRA and the "
|
| 132 |
+
"diffusers transformer disagree on module naming; nothing was folded for them."
|
| 133 |
+
)
|
| 134 |
+
|
| 135 |
+
return (
|
| 136 |
+
f"{label}: {len(bases)} targets -> {folded} weight deltas, rank {sorted(ranks)}, "
|
| 137 |
+
f"scale {sorted(scales)} (strength {strength:g})"
|
| 138 |
+
)
|
h3_video.py
ADDED
|
@@ -0,0 +1,205 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Clip I/O for the guide-latent pass, following `AIToolkitMiniMaxH3RefVideo` (`ostris/ComfyUI-AIToolkit-MiniMaxH3`).
|
| 2 |
+
|
| 3 |
+
The workflow the LoRA ships (`workflows/minimax_h3_lms_workflow.json`) prepares the source clip with that node at
|
| 4 |
+
`target_megapixels = 0.59`, `max_length = 0`, and the node's docstring is explicit that this is the recipe
|
| 5 |
+
ai-toolkit itself used to *cache the guide during training* — so it is the preparation the LoRA expects, not an
|
| 6 |
+
arbitrary resize. Reproduced here verbatim:
|
| 7 |
+
|
| 8 |
+
* **frame count** — `int(total / src_fps * 24)`, capped by `max_length`, snapped **down** to the model's `17n + 5`
|
| 9 |
+
grid. Down, not up: an up-snap would need frames the source does not have.
|
| 10 |
+
* **frame picks** — real-time pacing from frame 0, `source_frame(i) = min(round(i * src_fps / 24), total - 1)`, tail
|
| 11 |
+
trimmed. Any source fps re-times correctly to H3's fixed 24 fps.
|
| 12 |
+
* **size** — the clip's *own* aspect at `target_megapixels`, rounded to the /32 canvas grid, resampled with bicubic
|
| 13 |
+
+ antialias. The guide has to land on the target's grid exactly, so this one size is both the guide's and the
|
| 14 |
+
generation's.
|
| 15 |
+
* **soundtrack** — trimmed to the same real-time window, `round(n / 24 * sr)` samples.
|
| 16 |
+
|
| 17 |
+
The soundtrack is carried through rather than generated: the workflow decodes video only
|
| 18 |
+
(`LTXVSeparateAVLatent` -> `VAEDecode`) and muxes the original audio back, since an enhancement pass has no reason
|
| 19 |
+
to resynthesize a soundtrack it was handed.
|
| 20 |
+
"""
|
| 21 |
+
|
| 22 |
+
import math
|
| 23 |
+
import shutil
|
| 24 |
+
import subprocess
|
| 25 |
+
|
| 26 |
+
import av
|
| 27 |
+
import numpy as np
|
| 28 |
+
import torch
|
| 29 |
+
|
| 30 |
+
FPS = 24
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def snap_down_17n5(n: int) -> int:
|
| 34 |
+
"""Largest `17k + 5 <= n`, floored at 5 — ai-toolkit's `snap_down_17n5`."""
|
| 35 |
+
n = max(5, int(n))
|
| 36 |
+
return ((n - 5) // 17) * 17 + 5
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def probe(path: str) -> tuple[int, float, int, int]:
|
| 40 |
+
"""Return `(total_frames, fps, width, height)` of a clip, without decoding it."""
|
| 41 |
+
with av.open(path) as container:
|
| 42 |
+
stream = container.streams.video[0]
|
| 43 |
+
fps = float(stream.average_rate or stream.guessed_rate or FPS)
|
| 44 |
+
width, height = int(stream.codec_context.width), int(stream.codec_context.height)
|
| 45 |
+
total = int(stream.frames or 0)
|
| 46 |
+
if total <= 0:
|
| 47 |
+
# Some containers (webm, fragmented mp4) carry no frame count; fall back to the duration.
|
| 48 |
+
seconds = None
|
| 49 |
+
if stream.duration is not None and stream.time_base is not None:
|
| 50 |
+
seconds = float(stream.duration * stream.time_base)
|
| 51 |
+
elif container.duration is not None:
|
| 52 |
+
seconds = float(container.duration) / av.time_base
|
| 53 |
+
total = int(round(seconds * fps)) if seconds else 0
|
| 54 |
+
if total <= 0:
|
| 55 |
+
raise ValueError(f"Could not determine the frame count of {path}.")
|
| 56 |
+
return total, fps, width, height
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def plan_clip(path: str, target_megapixels: float, max_length: int = 0) -> dict:
|
| 60 |
+
"""Resolve the geometry of the guide pass without touching a single pixel.
|
| 61 |
+
|
| 62 |
+
Returns the `num_frames` / `height` / `width` triple `MiniMaxH3GuidePrepareLayoutStep` will be given, so a
|
| 63 |
+
caller can size its GPU budget and show the user the plan before paying for a decode.
|
| 64 |
+
"""
|
| 65 |
+
total, fps, width, height = probe(path)
|
| 66 |
+
|
| 67 |
+
num_frames = int(total / fps * FPS)
|
| 68 |
+
if max_length and max_length > 0:
|
| 69 |
+
num_frames = min(num_frames, int(max_length))
|
| 70 |
+
num_frames = snap_down_17n5(num_frames)
|
| 71 |
+
|
| 72 |
+
scale = math.sqrt(float(target_megapixels) * 1e6 / (width * height))
|
| 73 |
+
out_w = max(32, round(width * scale / 32) * 32)
|
| 74 |
+
out_h = max(32, round(height * scale / 32) * 32)
|
| 75 |
+
return {
|
| 76 |
+
"num_frames": num_frames,
|
| 77 |
+
"height": out_h,
|
| 78 |
+
"width": out_w,
|
| 79 |
+
"source_frames": total,
|
| 80 |
+
"source_fps": fps,
|
| 81 |
+
"source_height": height,
|
| 82 |
+
"source_width": width,
|
| 83 |
+
}
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
def load_guide(path: str, plan: dict) -> torch.Tensor:
|
| 87 |
+
"""Decode and resample the source clip into the guide's `(1, 3, num_frames, height, width)` uint8 tensor.
|
| 88 |
+
|
| 89 |
+
Only the source frames the pacing actually lands on are kept, so a long clip that gets trimmed to a couple of
|
| 90 |
+
seconds never materializes in full.
|
| 91 |
+
"""
|
| 92 |
+
num_frames, total, fps = plan["num_frames"], plan["source_frames"], plan["source_fps"]
|
| 93 |
+
picks = [min(round(i * fps / FPS), total - 1) for i in range(num_frames)]
|
| 94 |
+
wanted = sorted(set(picks))
|
| 95 |
+
|
| 96 |
+
decoded: dict[int, np.ndarray] = {}
|
| 97 |
+
wanted_set, last = set(wanted), wanted[-1]
|
| 98 |
+
with av.open(path) as container:
|
| 99 |
+
container.streams.video[0].thread_type = "AUTO"
|
| 100 |
+
for index, frame in enumerate(container.decode(video=0)):
|
| 101 |
+
if index in wanted_set:
|
| 102 |
+
decoded[index] = frame.to_ndarray(format="rgb24")
|
| 103 |
+
if index >= last:
|
| 104 |
+
break
|
| 105 |
+
if not decoded:
|
| 106 |
+
raise ValueError(f"Decoded no frames from {path}.")
|
| 107 |
+
# A container can stop short of the frame count it advertises; clamp onto what actually decoded.
|
| 108 |
+
available = max(decoded)
|
| 109 |
+
frames = np.stack([decoded[min(pick, available)] for pick in picks])
|
| 110 |
+
|
| 111 |
+
pixels = torch.from_numpy(frames).permute(0, 3, 1, 2) # (T, 3, H, W) uint8
|
| 112 |
+
if (pixels.shape[2], pixels.shape[3]) != (plan["height"], plan["width"]):
|
| 113 |
+
pixels = (
|
| 114 |
+
torch.nn.functional.interpolate(
|
| 115 |
+
pixels.float().div_(255.0),
|
| 116 |
+
size=(plan["height"], plan["width"]),
|
| 117 |
+
mode="bicubic",
|
| 118 |
+
antialias=True,
|
| 119 |
+
align_corners=False,
|
| 120 |
+
)
|
| 121 |
+
.clamp_(0.0, 1.0)
|
| 122 |
+
.mul_(255.0)
|
| 123 |
+
.round_()
|
| 124 |
+
.to(torch.uint8)
|
| 125 |
+
)
|
| 126 |
+
return pixels.permute(1, 0, 2, 3)[None].contiguous() # (1, 3, T, H, W)
|
| 127 |
+
|
| 128 |
+
|
| 129 |
+
def write_video(frames, path: str, fps: int = FPS, crf: int = 14) -> str:
|
| 130 |
+
"""Encode `(T, H, W, 3)` uint8 frames to H.264 at 24 fps."""
|
| 131 |
+
frames = np.asarray(frames)
|
| 132 |
+
if frames.dtype != np.uint8:
|
| 133 |
+
frames = np.clip(frames * 255.0 if frames.max() <= 1.0 else frames, 0, 255).astype(np.uint8)
|
| 134 |
+
height, width = int(frames.shape[1]), int(frames.shape[2])
|
| 135 |
+
with av.open(path, "w") as container:
|
| 136 |
+
stream = container.add_stream("libx264", rate=fps)
|
| 137 |
+
stream.width, stream.height, stream.pix_fmt = width, height, "yuv420p"
|
| 138 |
+
stream.options = {"crf": str(crf), "preset": "medium"}
|
| 139 |
+
for frame in frames:
|
| 140 |
+
for packet in stream.encode(av.VideoFrame.from_ndarray(np.ascontiguousarray(frame), format="rgb24")):
|
| 141 |
+
container.mux(packet)
|
| 142 |
+
for packet in stream.encode():
|
| 143 |
+
container.mux(packet)
|
| 144 |
+
return path
|
| 145 |
+
|
| 146 |
+
|
| 147 |
+
def has_audio(path: str) -> bool:
|
| 148 |
+
try:
|
| 149 |
+
with av.open(path) as container:
|
| 150 |
+
return bool(container.streams.audio)
|
| 151 |
+
except Exception:
|
| 152 |
+
return False
|
| 153 |
+
|
| 154 |
+
|
| 155 |
+
def mux_source_audio(video_path: str, source_path: str, num_frames: int, out_path: str) -> str:
|
| 156 |
+
"""Attach the source clip's own soundtrack, trimmed to the `num_frames / 24` window the pass covers.
|
| 157 |
+
|
| 158 |
+
ai-toolkit trims the reference soundtrack to exactly `round(n / 24 * sr)` samples; `-t` on the audio input is
|
| 159 |
+
the same window expressed in seconds. Falls back to the silent video if ffmpeg is unavailable or the mux fails,
|
| 160 |
+
since a missing soundtrack is not worth failing a generation over.
|
| 161 |
+
"""
|
| 162 |
+
if not shutil.which("ffmpeg") or not has_audio(source_path):
|
| 163 |
+
return video_path
|
| 164 |
+
seconds = num_frames / FPS
|
| 165 |
+
command = [
|
| 166 |
+
"ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
| 167 |
+
"-i", video_path,
|
| 168 |
+
"-t", f"{seconds:.6f}", "-i", source_path,
|
| 169 |
+
"-map", "0:v:0", "-map", "1:a:0",
|
| 170 |
+
"-c:v", "copy", "-c:a", "aac", "-b:a", "192k", "-shortest",
|
| 171 |
+
out_path,
|
| 172 |
+
]
|
| 173 |
+
try:
|
| 174 |
+
subprocess.run(command, check=True, capture_output=True)
|
| 175 |
+
except Exception as error:
|
| 176 |
+
print(f"[audio] keeping the silent video, mux failed: {error}", flush=True)
|
| 177 |
+
return video_path
|
| 178 |
+
return out_path
|
| 179 |
+
|
| 180 |
+
|
| 181 |
+
def size_to_megapixels(width: int, height: int, target_megapixels: float) -> tuple[int, int]:
|
| 182 |
+
"""`(out_height, out_width)` for `width x height` resized to `target_megapixels` on the /32 canvas grid."""
|
| 183 |
+
scale = math.sqrt(float(target_megapixels) * 1e6 / (width * height))
|
| 184 |
+
out_w = max(32, round(width * scale / 32) * 32)
|
| 185 |
+
out_h = max(32, round(height * scale / 32) * 32)
|
| 186 |
+
return out_h, out_w
|
| 187 |
+
|
| 188 |
+
|
| 189 |
+
def load_image(path: str, target_megapixels: float) -> torch.Tensor:
|
| 190 |
+
"""Load a reference image into a `(1, 3, 1, H, W)` uint8 tensor, sized to `target_megapixels` on its own aspect.
|
| 191 |
+
|
| 192 |
+
The single temporal frame is what `encode_vae_condition` runs through the VAE's spatial encoder alone, yielding
|
| 193 |
+
one latent frame — the geometry H3's native image reference occupies.
|
| 194 |
+
"""
|
| 195 |
+
from PIL import Image, ImageOps
|
| 196 |
+
|
| 197 |
+
image = ImageOps.exif_transpose(Image.open(path)).convert("RGB")
|
| 198 |
+
out_h, out_w = size_to_megapixels(image.width, image.height, target_megapixels)
|
| 199 |
+
pixels = torch.from_numpy(np.asarray(image)).permute(2, 0, 1)[None].float().div_(255.0) # (1, 3, H, W)
|
| 200 |
+
if (pixels.shape[2], pixels.shape[3]) != (out_h, out_w):
|
| 201 |
+
pixels = torch.nn.functional.interpolate(
|
| 202 |
+
pixels, size=(out_h, out_w), mode="bicubic", antialias=True, align_corners=False
|
| 203 |
+
).clamp_(0.0, 1.0)
|
| 204 |
+
pixels = pixels.mul_(255.0).round_().to(torch.uint8)
|
| 205 |
+
return pixels.unsqueeze(2).contiguous() # (1, 3, 1, H, W)
|
packages.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
ffmpeg
|
requirements.txt
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# `diffusers` is installed from the canonical MiniMax-H3 pull request,
|
| 2 |
+
# https://github.com/huggingface/diffusers/pull/14371 ("Minimax h3 follow up (review & refactor)"), pinned to a
|
| 3 |
+
# **commit** rather than to its `minimax-h3-refactor` branch: the PR is a WIP and its head moves, and this Space's
|
| 4 |
+
# `h3_guide_blocks.py` subclasses its block classes. Re-pin — and re-check the block/step names against the new
|
| 5 |
+
# head — whenever the PR updates.
|
| 6 |
+
#
|
| 7 |
+
# 665f578278365ea4a3318cb8c9b66ce6c01204b9 = refs/pull/14371/head at the time of this deploy
|
| 8 |
+
--extra-index-url https://download.pytorch.org/whl/cu130
|
| 9 |
+
diffusers @ git+https://github.com/huggingface/diffusers.git@665f578278365ea4a3318cb8c9b66ce6c01204b9
|
| 10 |
+
torch==2.11.0
|
| 11 |
+
torchvision==0.26.0
|
| 12 |
+
# `diffusers.modular_pipelines.minimax_h3` imports Qwen3-VL at module import, so transformers has to carry it.
|
| 13 |
+
transformers==5.8.0
|
| 14 |
+
accelerate==1.14.0
|
| 15 |
+
# No `kernels` pin on purpose: the Hub attention backends want `kernels>=0.12.3`, and that version breaks
|
| 16 |
+
# transformers 5.8.0 at import.
|
| 17 |
+
# PyAV decodes the source clip and encodes the result; ffmpeg (packages.txt) muxes the source soundtrack back on.
|
| 18 |
+
av
|
| 19 |
+
pillow
|
| 20 |
+
numpy
|
| 21 |
+
requests
|
| 22 |
+
safetensors>=0.8.0
|