linoyts HF Staff commited on
Commit
ac62bb3
·
verified ·
1 Parent(s): 368ae59

MiniMax-H3 Character Swap: guide latent + native image reference port

Browse files
Files changed (7) hide show
  1. README.md +63 -8
  2. app.py +576 -0
  3. h3_charswap_blocks.py +294 -0
  4. h3_lora.py +138 -0
  5. h3_video.py +205 -0
  6. packages.txt +1 -0
  7. requirements.txt +22 -0
README.md CHANGED
@@ -1,13 +1,68 @@
1
  ---
2
- title: Minimax H3 Character Swap
3
- emoji: 🏃
4
- colorFrom: gray
5
- colorTo: pink
6
  sdk: gradio
7
- sdk_version: 6.30.0
8
- python_version: '3.13'
9
  app_file: app.py
10
- pinned: false
 
 
 
 
 
 
 
 
11
  ---
12
 
13
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: MiniMax-H3 Character Swap LoRA
3
+ emoji: 🎭
4
+ colorFrom: purple
5
+ colorTo: blue
6
  sdk: gradio
7
+ sdk_version: 6.26.0
 
8
  app_file: app.py
9
+ python_version: "3.12"
10
+ startup_duration_timeout: 1h
11
+ short_description: Character swap on video with the MiniMax-H3 ref2va LoRA
12
+ models:
13
+ - Alissonerdx/Minimax-H3-ComfyUI
14
+ - multimodalart/MiniMax-H3-Pruned
15
+ - MiniMaxAI/MiniMax-H3
16
+ - lightx2v/Minimax-h3-Turbo
17
+ - briaai/RMBG-2.0
18
  ---
19
 
20
+ # MiniMax-H3 · Character Swap LoRA
21
+
22
+ Replace the **main subject of a guide video** with the **character in a reference image**, keeping the guide's
23
+ scene, camera, motion and performance. This is a `diffusers` port of the ComfyUI graph the LoRA ships
24
+ (`workflows/minimax_h3_character_swap_workflow.json`), minus its optional BFS multi-shot planner — one guide clip,
25
+ one reference character, one swap.
26
+
27
+ LoRA: [`Alissonerdx/Minimax-H3-ComfyUI`](https://huggingface.co/Alissonerdx/Minimax-H3-ComfyUI) →
28
+ `loras/minimax_h3_character_swap_v1.0_ref2va_r64_gl.safetensors` (rank 64, `ref2va`, guide-latent), by
29
+ [Alissonerdx](https://github.com/Alissonerdx/Minimax-H3-ComfyUI).
30
+
31
+ ## How it conditions
32
+
33
+ The LoRA's own training metadata records that it sees **two** visual blocks at once:
34
+
35
+ ```
36
+ minimax_h3_reference_rope = {"guide_rope_layout": "overlap", "reference_rope_layout": "native", ...}
37
+ minimax_h3_dataset_control_roles = [["guide", "reference", "reference"]]
38
+ guide_latent_only = true
39
+ ```
40
+
41
+ | ComfyUI node | here |
42
+ |---|---|
43
+ | `BiRefNetRMBG` on the reference | [`briaai/RMBG-2.0`](https://huggingface.co/briaai/RMBG-2.0) via the `linoyts/bria-rmbg-service` client Space, cutout on `#808080` |
44
+ | `<Picture 1>` through the Qwen3-VL CLIP | [`multimodalart/qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner) `/encode_ref2va`, image kind |
45
+ | `MiniMaxH3AddGuide(frame_idx = 0)` | `h3_charswap_blocks.MiniMaxH3CharswapPrepareLayoutStep` — the guide on the target's own `(t, h, w)` grid (`overlap`) |
46
+ | native image-reference block | the same layout step — the reference image on H3's native rope, one integer slot (`native`) |
47
+ | `LoraLoaderModelOnly(character_swap_gl, 1.0)` | folded at startup by `h3_lora.fold_comfy_lora` |
48
+ | few-step turbo + `BasicScheduler("simple", 8)` | [`lightx2v/Minimax-h3-Turbo`](https://huggingface.co/lightx2v/Minimax-h3-Turbo) 4-step folded at 1.0, 8 Euler steps, no CFG |
49
+ | `LTXVSeparateAVLatent` -> `VAEDecode` (video only) | video-only decode; the guide's own soundtrack is muxed back |
50
+
51
+ So the packed layout is `[text (+ Picture 1 vision) | reference image | guide | target audio | target video]`. The
52
+ reference image takes one integer rotary slot; the guide overlaps the target on its own grid, frame for frame.
53
+
54
+ Weights are `multimodalart/MiniMax-H3-Pruned`'s `transformer_ref` — the released `ref2va` partition with its 2688-wide
55
+ AdaLN projections replaced by a rank-8 factorization, the same pruning the workflow's
56
+ `minimax_h3_ref2va_pruned_int8_convrot.safetensors` carries — so both LoRAs fold onto it natively. H3's 62 GiB
57
+ Qwen3-VL text tower will not fit alongside the transformer on one ZeroGPU worker, so the caption and the reference
58
+ image are encoded by the conditioner Space (billed to the visitor's own quota via `x-ip-token`).
59
+
60
+ ## Notes and limits
61
+
62
+ * Experimental, not a guaranteed identity-transfer system. Results drift with small faces, occlusion, fast motion,
63
+ hard angles, lighting changes, or long clips; it is aimed at people and human-like characters.
64
+ * The workflow's `drbaph` DMD 8-step turbo carries `adaln_proj.linear.bias.diff` direct deltas the LoRA-fold path
65
+ does not apply, so this Space stacks the sibling LMS Space's proven `lightx2v` 4-step turbo instead.
66
+ * Use only with appropriate rights and consent for the reference images and guide videos. Not for impersonation,
67
+ fraud, harassment, or other unlawful purposes. See the
68
+ [Character Swap license](https://huggingface.co/Alissonerdx/Minimax-H3-ComfyUI/blob/main/LICENSE_CHARACTER_SWAP.md).
app.py ADDED
@@ -0,0 +1,576 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """MiniMax-H3 Character Swap — the guide-latent + image-reference swap LoRA from `Alissonerdx/Minimax-H3-ComfyUI`.
2
+
3
+ `loras/minimax_h3_character_swap_v1.0_ref2va_r64_gl.safetensors` is a rank-64 LoRA for MiniMax-H3's `ref2va`
4
+ partition that **replaces the main subject of a guide video with the character in a reference image** while keeping
5
+ the guide's scene, camera, motion and performance. Its own training metadata spells out how it conditions:
6
+
7
+ minimax_h3_reference_rope = {"guide_rope_layout": "overlap", "reference_rope_layout": "native", ...}
8
+ minimax_h3_dataset_control_roles = [["guide", "reference", "reference"]]
9
+ guide_latent_only = true
10
+
11
+ so it sees two visual blocks at once — an **aligned guide** (the whole source clip VAE-encoded at the target
12
+ resolution and packed on the target's own `(t, h, w)` grid, ComfyUI's `MiniMaxH3AddGuide(frame_idx = 0)`) and a
13
+ **native image reference** (the character image as a Qwen3-VL `<Picture 1>` vision block *and* a VAE reference
14
+ block). This Space is a diffusers port of the repo's `workflows/minimax_h3_character_swap_workflow.json`, minus its
15
+ optional BFS multi-shot planner: one guide clip, one reference character, one swap.
16
+
17
+ | ComfyUI node | here |
18
+ |---|---|
19
+ | `BiRefNetRMBG` on the reference | `linoyts/bria-rmbg-service` (a `briaai/RMBG-2.0` client Space), cutout on `#808080` |
20
+ | `<Picture 1>` through the CLIP (Qwen3-VL) | `multimodalart/qwen3vl-conditioner` `/encode_ref2va`, image kind |
21
+ | `MiniMaxH3AddGuide(frame_idx = 0)` | `h3_charswap_blocks.MiniMaxH3CharswapPrepareLayoutStep`, guide on the target grid |
22
+ | native image reference block | the same layout step, reference on H3's native rope — see `h3_charswap_blocks.py` |
23
+ | `LoraLoaderModelOnly(character_swap_gl, 1.0)` | folded at startup by `h3_lora.fold_comfy_lora` |
24
+ | few-step turbo + `BasicScheduler("simple", 8)` | `lightx2v/Minimax-h3-Turbo` 4-step folded at 1.0, 8 Euler steps, no CFG |
25
+ | `PrimitiveStringMultiline` `body_swap:` | the trigger caption built from the reference description below |
26
+ | `LTXVSeparateAVLatent` -> `VAEDecode` (video only) | video-only decode; the guide's own soundtrack is muxed back |
27
+
28
+ Weights are `multimodalart/MiniMax-H3-Pruned`'s `transformer_ref` (the `ref2va` partition with rank-8 AdaLN factors),
29
+ the same pruning the workflow's own `minimax_h3_ref2va_pruned_int8_convrot.safetensors` carries, which is why both
30
+ LoRAs fold onto it natively. H3's 62 GiB Qwen3-VL does not fit next to the transformer on one ZeroGPU worker, so the
31
+ caption *and the reference image* are encoded by `multimodalart/qwen3vl-conditioner` over the gradio API (billed to
32
+ the visitor's own quota via `x-ip-token`).
33
+ """
34
+
35
+ from __future__ import annotations
36
+
37
+ import os
38
+
39
+ # Before torch. See the sibling LMS Space: the guide VAE encode tripped the ZeroGPU allocator's expandable-segments
40
+ # path (`NVML_SUCCESS == r INTERNAL ASSERT FAILED`) with memory free; plain `cudaMalloc` never queries NVML.
41
+ _ALLOC_CONF_INHERITED = os.environ.get("PYTORCH_CUDA_ALLOC_CONF")
42
+ os.environ["PYTORCH_CUDA_ALLOC_CONF"] = os.environ.get("H3_ALLOC_CONF", "expandable_segments:False")
43
+
44
+ import hashlib
45
+ import math
46
+ import tempfile
47
+ import time
48
+ import traceback
49
+ import uuid
50
+ from functools import cache
51
+
52
+ import spaces
53
+ import gradio as gr
54
+ import torch
55
+
56
+ import h3_video
57
+
58
+ MODEL_REPO = os.environ.get("H3_MODEL_REPO", "multimodalart/MiniMax-H3-Pruned")
59
+ CONDITIONER_SPACE = os.environ.get("H3_CONDITIONER", "multimodalart/qwen3vl-conditioner")
60
+ RMBG_SPACE = os.environ.get("H3_RMBG", "linoyts/bria-rmbg-service")
61
+ GPU_SIZE = os.environ.get("H3_GPU_SIZE", "xlarge")
62
+ ATTENTION = os.environ.get("H3_ATTENTION", "_native_cudnn")
63
+
64
+ # The task LoRA, stacked on the proven 4-step turbo the sibling LMS Space uses (the workflow's own drbaph DMD turbo
65
+ # carries `.bias.diff` direct deltas the fold path does not apply, so it would load half-merged). Both at 1.0,
66
+ # turbo first, folded into the base weights before the model reaches the GPU.
67
+ LORAS = [
68
+ (
69
+ "lightx2v/Minimax-h3-Turbo",
70
+ "minimax_h3_ref2v_turbo_4step_v0.1_comfyui_bf16.safetensors",
71
+ 1.0,
72
+ "turbo-4step",
73
+ ),
74
+ (
75
+ os.environ.get("H3_LORA_REPO", "Alissonerdx/Minimax-H3-ComfyUI"),
76
+ os.environ.get("H3_LORA_FILE", "loras/minimax_h3_character_swap_v1.0_ref2va_r64_gl.safetensors"),
77
+ 1.0,
78
+ "character-swap-r64-gl",
79
+ ),
80
+ ]
81
+
82
+ # `workflows/minimax_h3_character_swap_workflow.json`'s `body_swap:` prompt, verbatim. `{details}` is the reference
83
+ # description, `{target}` names which guide subject to replace when more than one person appears.
84
+ PROMPT_MAIN = (
85
+ "body_swap: subject_definitions: <Subject 1> is the {details} shown in <Picture 1>, whose identity, face, "
86
+ "head, hair or fur, skin, body shape, proportions and clothing define the character in the target video. The "
87
+ "guide video provides the scene and the performance: its background, environment, lighting, camera, framing, "
88
+ "body pose, hand and arm movement, head rotation, gaze direction, lip movement and facial expression, frame by "
89
+ "frame; only the main subject of the guide video is replaced: its face, hair, body and clothing become "
90
+ "<Subject 1>. summary: [video generation + reference] Replace the guide subject with <Subject 1>, keeping the "
91
+ "scene, camera and the complete motion and performance of the guide video. detailed_description: [Shot 1] "
92
+ "<Subject 1> takes the exact place of the guide subject and moves with the same timing, body pose, gestures, "
93
+ "head motion, gaze, speech and facial expressions, while keeping the complete look from <Picture 1>: head, "
94
+ "face, skin, hair or fur, body shape and clothing stay stable in every frame, with no trace of the original "
95
+ "subject's face, hair or clothing. The background, lighting, colors and camera of the guide video are "
96
+ "preserved. Prevent identity drift, clothing changes, texture crawling and flicker. The result is a real, "
97
+ "photographic video."
98
+ )
99
+ PROMPT_TARGETED = (
100
+ "body_swap: subject_definitions: <Subject 1> is the {details} shown in <Picture 1>, whose identity, face, "
101
+ "head, hair or fur, skin, body shape, proportions and clothing define the character in the target video. The "
102
+ "guide video provides the scene and the performance: its background, environment, lighting, camera, framing, "
103
+ "body pose, hand and arm movement, head rotation, gaze direction, lip movement and facial expression, frame by "
104
+ "frame; only {target} is replaced: their face, hair, body and clothing become <Subject 1>. summary: [video "
105
+ "generation + reference] Replace {target} in the guide video with <Subject 1>, keeping the scene, camera and "
106
+ "the complete motion and performance of the guide video. detailed_description: [Shot 1] <Subject 1> takes the "
107
+ "exact place of {target} and moves with the same timing, body pose, gestures, head motion, gaze, speech and "
108
+ "facial expressions, while keeping the complete look from <Picture 1>: head, face, skin, hair or fur, body "
109
+ "shape and clothing stay stable in every frame, with no trace of the original subject's face, hair or "
110
+ "clothing. The background, lighting, colors and camera of the guide video are preserved. Prevent identity "
111
+ "drift, clothing changes, texture crawling and flicker. The result is a real, photographic video."
112
+ )
113
+ DEFAULT_DETAILS = "character"
114
+
115
+ FPS = h3_video.FPS
116
+ DEFAULT_STEPS = 8
117
+ DEFAULT_SEED = 46 # `RandomNoise(46, "fixed")` in the workflow.
118
+
119
+ # `AIToolkitMiniMaxH3RefVideo.max_length` — a cap on the 24 fps frame count, snapped down to 17n + 5 afterwards.
120
+ CLIP_LENGTHS = {
121
+ "0.9 s · 22 frames": 22,
122
+ "1.6 s · 39 frames": 39,
123
+ "2.3 s · 56 frames": 56,
124
+ "3.0 s · 73 frames": 73,
125
+ "3.8 s · 90 frames": 90,
126
+ }
127
+ DEFAULT_CLIP = "2.3 s · 56 frames"
128
+
129
+ # Working pixel area of the guide / generation, on the clip's own aspect ratio.
130
+ DETAIL_LEVELS = {
131
+ "0.26 MP · fastest (672x384 at 16:9)": 0.258,
132
+ "0.42 MP · balanced (864x480 at 16:9)": 0.42,
133
+ "0.59 MP · sharp (1024x576 at 16:9)": 0.59,
134
+ }
135
+ DEFAULT_DETAIL = "0.42 MP · balanced (864x480 at 16:9)"
136
+
137
+ # The reference image is encoded at its own resolution; the workflow sizes it to ~0.3 MP. The conditioner also
138
+ # validates a canvas/frame count it never uses for an image-only request — these just have to be in range.
139
+ REFERENCE_MEGAPIXELS = 0.3
140
+ CONDITIONER_CANVAS = "960x544 · 16:9 fast"
141
+
142
+ OUTPUT_DIR = os.path.join(tempfile.gettempdir(), "h3-charswap-out")
143
+ os.makedirs(OUTPUT_DIR, exist_ok=True)
144
+
145
+ PIPE = None
146
+ LOAD_ERROR: str | None = None
147
+ LOAD_NOTES: list[str] = []
148
+ _PROMPT_CACHE: dict[str, tuple] = {}
149
+ _RMBG_CACHE: dict[str, str] = {}
150
+
151
+
152
+ # ── Model loading ───────────────────────────────────────────────────────────
153
+
154
+
155
+ def load_models() -> str | None:
156
+ """Load the denoising half — `transformer_ref` + video VAE + schedulers — and fold both LoRAs in."""
157
+ global PIPE, LOAD_ERROR
158
+
159
+ if PIPE is not None or LOAD_ERROR is not None:
160
+ return LOAD_ERROR
161
+
162
+ started = time.time()
163
+ try:
164
+ from diffusers import ComponentsManager
165
+ from diffusers.modular_pipelines.minimax_h3.modular_pipeline import MiniMaxH3ModularPipeline
166
+ from huggingface_hub import hf_hub_download
167
+
168
+ from h3_charswap_blocks import MiniMaxH3CharswapGeneratorBlocks
169
+ from h3_lora import fold_comfy_lora
170
+
171
+ MiniMaxH3ModularPipeline.min_duration = property(lambda self: 0.2)
172
+
173
+ blocks = MiniMaxH3CharswapGeneratorBlocks()
174
+ print(f"[gen] loading {[c.name for c in blocks.expected_components]} from {MODEL_REPO} ...", flush=True)
175
+ pipe = blocks.init_pipeline(MODEL_REPO, components_manager=ComponentsManager(), collection="h3")
176
+ pipe.load_components(dtype=torch.bfloat16, trust_remote_code=True)
177
+ for name in ("transformer_ref", "vae", "scheduler", "audio_scheduler"):
178
+ if getattr(pipe, name, None) is None:
179
+ raise RuntimeError(f"Component `{name}` did not load; see the warning above for why.")
180
+
181
+ pipe.vae.set_attention_backend("native")
182
+ pipe.transformer_ref.set_attention_backend(ATTENTION)
183
+
184
+ for repo, filename, strength, label in LORAS:
185
+ path = hf_hub_download(repo, filename)
186
+ status = fold_comfy_lora(pipe.transformer_ref, path, strength=strength, label=label)
187
+ LOAD_NOTES.append(status)
188
+ print(f"[lora] {status}", flush=True)
189
+
190
+ pipe.transformer_ref.to("cuda")
191
+ PIPE = pipe
192
+ print(f"[gen] ready in {time.time() - started:.0f}s", flush=True)
193
+ except Exception as error:
194
+ traceback.print_exc()
195
+ LOAD_ERROR = f"**Loading failed** after {time.time() - started:.0f}s: `{type(error).__name__}: {error}`"
196
+ return LOAD_ERROR
197
+
198
+
199
+ # ── Remote conditioning + background removal ──────────────────────────────────
200
+
201
+
202
+ def _caller_ip_token() -> str | None:
203
+ from gradio.context import LocalContext
204
+
205
+ request = LocalContext.request.get()
206
+ return request.headers.get("x-ip-token") if request is not None else None
207
+
208
+
209
+ @cache
210
+ def _anonymous_client(space: str):
211
+ from gradio_client import Client
212
+
213
+ return Client(space)
214
+
215
+
216
+ def _client(space: str, ip_token: str | None):
217
+ """A client for a helper Space, billed to the visitor when their `x-ip-token` is available."""
218
+ if not ip_token:
219
+ return _anonymous_client(space)
220
+ from gradio_client import Client
221
+
222
+ return Client(space, headers={"x-ip-token": ip_token})
223
+
224
+
225
+ def _file_digest(path: str) -> str:
226
+ with open(path, "rb") as handle:
227
+ return hashlib.sha256(handle.read()).hexdigest()
228
+
229
+
230
+ def remove_background(image_path: str, ip_token: str | None) -> str:
231
+ """Cut the reference subject out onto neutral gray via the RMBG service; fall back to the raw image."""
232
+ digest = _file_digest(image_path)
233
+ if digest in _RMBG_CACHE and os.path.exists(_RMBG_CACHE[digest]):
234
+ return _RMBG_CACHE[digest]
235
+ from gradio_client import handle_file
236
+
237
+ try:
238
+ result = _client(RMBG_SPACE, ip_token).predict(
239
+ image_path=handle_file(image_path), background="gray", api_name="/rmbg"
240
+ )
241
+ except Exception as error:
242
+ print(f"[rmbg] keeping the raw reference, service failed: {error}", flush=True)
243
+ return image_path
244
+ _RMBG_CACHE[digest] = result
245
+ return result
246
+
247
+
248
+ def _read_conditioning(path: str):
249
+ from safetensors import safe_open
250
+
251
+ with safe_open(path, framework="pt") as handle:
252
+ return handle.get_tensor("prompt_embeds"), handle.get_tensor("text_token_tags")
253
+
254
+
255
+ def encode_prompt(prompt: str, image_path: str, num_frames: int):
256
+ """`prompt_embeds` + `text_token_tags` for a `body_swap` caption with the reference image as `<Picture 1>`.
257
+
258
+ The reference image reaches Qwen3-VL as an ordered `ref2va` image reference, so the embedding carries its
259
+ `<Picture 1>` vision block. Cached per `(prompt, reference image)`; `num_frames` only has to be a valid
260
+ duration for the conditioner's own range check, it does not change an image-only embedding.
261
+ """
262
+ key = f"{prompt}\n{_file_digest(image_path)}"
263
+ if key in _PROMPT_CACHE:
264
+ return _PROMPT_CACHE[key]
265
+
266
+ from gradio_client import handle_file
267
+
268
+ ip_token = _caller_ip_token()
269
+
270
+ def call():
271
+ return _client(CONDITIONER_SPACE, ip_token).predict(
272
+ prompt=prompt,
273
+ media=[handle_file(image_path)],
274
+ kinds="image",
275
+ canvas=CONDITIONER_CANVAS,
276
+ num_frames=int(num_frames),
277
+ rewrite_prompt=False,
278
+ api_name="/encode_ref2va",
279
+ )
280
+
281
+ try:
282
+ path, _plan = call()
283
+ except Exception as first:
284
+ print(f"[conditioner] retrying with a fresh client after: {first}", flush=True)
285
+ _anonymous_client.cache_clear()
286
+ try:
287
+ path, _plan = call()
288
+ except Exception as second:
289
+ raise gr.Error(
290
+ "The Qwen3-VL conditioner Space could not encode the reference "
291
+ f"({type(second).__name__}: {second})."
292
+ ) from second
293
+
294
+ embeds = _read_conditioning(path)
295
+ _PROMPT_CACHE[key] = embeds
296
+ return embeds
297
+
298
+
299
+ # ── GPU budget ──────────────────────────────────────────────────────────────
300
+
301
+ # Seconds per step vs packed row count: linear matmuls, quadratic full self-attention. Calibrated by the sibling
302
+ # MiniMax-H3 Spaces on this pool.
303
+ _DUR_B, _DUR_C = 1.1745e-4, 3.8396e-9
304
+ _VAE_PER_REFERENCE_PIXELS, _REFERENCE_PIXELS = 26.0, 960 * 544 * 124
305
+ _PLACEMENT_ALLOWANCE = 15
306
+ _SAFETY = 1.2
307
+
308
+
309
+ def _reference_latent_patches(megapixels: float) -> int:
310
+ """Patch rows a reference image sized to `megapixels` adds (one latent frame, /32 patch grid)."""
311
+ side = math.sqrt(megapixels * 1e6)
312
+ return max(1, round(side / 32) ** 2)
313
+
314
+
315
+ def estimate_duration(height: int, width: int, num_frames: int, steps: int) -> int:
316
+ """Seconds of GPU time one swap pass needs: reference + guide conditioning rows and the target included."""
317
+ latent_frames = (num_frames - 5) // 17 * 5 + 2
318
+ patches = (height // 32) * (width // 32)
319
+ # Guide doubles the video rows (one conditioning row per target row); the reference adds one small frame.
320
+ rows = 2 * latent_frames * patches + _reference_latent_patches(REFERENCE_MEGAPIXELS)
321
+ denoise = steps * (_DUR_B * rows + _DUR_C * rows**2)
322
+ vae = _VAE_PER_REFERENCE_PIXELS * (height * width * num_frames) / _REFERENCE_PIXELS
323
+ return max(60, int((denoise + vae) * _SAFETY) + _PLACEMENT_ALLOWANCE)
324
+
325
+
326
+ def _gpu_duration(ref_pixels, pixels, prompt_embeds, tags, height, width, num_frames, steps, seed, *a, **k):
327
+ return estimate_duration(int(height), int(width), int(num_frames), int(steps))
328
+
329
+
330
+ # ── Inference ───────────────────────────────────────────────────────────────
331
+
332
+
333
+ @torch.no_grad()
334
+ def encode_condition(vae, pixels, pixel_mean, pixel_std, encode_seed):
335
+ """`diffusers...minimax_h3.encoders.encode_vae_condition`, cast to the VAE's own dtype (see the LMS Space).
336
+
337
+ A single-frame `(1, 3, 1, H, W)` input (the reference image) runs the spatial encoder alone and returns one
338
+ latent frame; a clip runs the temporal chunking. `no_grad` keeps the float32 encoder's activations from being
339
+ retained for a backward pass that never comes.
340
+ """
341
+ latents_mean = torch.tensor(vae.config.latents_mean).view(1, -1, 1, 1, 1)
342
+ latents_std = torch.tensor(vae.config.latents_std).view(1, -1, 1, 1, 1)
343
+ pixel_mean = torch.tensor(pixel_mean, device=pixels.device).view(1, -1, 1, 1, 1)
344
+ pixel_std = torch.tensor(pixel_std, device=pixels.device).view(1, -1, 1, 1, 1)
345
+
346
+ pixels = (pixels.to(torch.float32).div(255.0) - pixel_mean) / pixel_std
347
+ posterior = vae.encode(pixels.to(vae.dtype), return_dict=False)[0]
348
+ latents = posterior.sample(generator=torch.Generator().manual_seed(encode_seed))
349
+ latents = latents.to(torch.float16).float().cpu()
350
+ return (latents - latents_mean) / latents_std
351
+
352
+
353
+ @spaces.GPU(duration=_gpu_duration, size=GPU_SIZE)
354
+ def _swap(ref_pixels, pixels, prompt_embeds, text_token_tags, height, width, num_frames, steps, seed):
355
+ """Encode the reference image and the guide clip, run the joint denoising loop, decode the video."""
356
+ PIPE.vae.to("cuda")
357
+ free, total = torch.cuda.mem_get_info()
358
+ print(
359
+ f"[gpu] {free / 2**30:.1f} GiB free of {total / 2**30:.1f} GiB · alloc conf "
360
+ f"{os.environ.get('PYTORCH_CUDA_ALLOC_CONF')!r}",
361
+ flush=True,
362
+ )
363
+ started = time.time()
364
+ reference = encode_condition(PIPE.vae, ref_pixels.to("cuda"), PIPE.pixel_mean, PIPE.pixel_std, PIPE.keyframe_encode_seed)
365
+ del ref_pixels
366
+
367
+ pixels = pixels.to("cuda")
368
+ try:
369
+ guide = encode_condition(PIPE.vae, pixels, PIPE.pixel_mean, PIPE.pixel_std, PIPE.keyframe_encode_seed)
370
+ except RuntimeError as error:
371
+ # The failed attempt's traceback pins every intermediate activation alive; drop it before retrying on tiles.
372
+ error.__traceback__ = None
373
+ torch.cuda.empty_cache()
374
+ print(f"[gpu] guide encode failed ({type(error).__name__}); retrying on 128px tiles.", flush=True)
375
+ PIPE.vae.enable_tiling(tile_sample_min_height=128, tile_sample_min_width=128)
376
+ guide = encode_condition(PIPE.vae, pixels, PIPE.pixel_mean, PIPE.pixel_std, PIPE.keyframe_encode_seed)
377
+ del pixels
378
+ torch.cuda.empty_cache()
379
+ encoded_in = time.time() - started
380
+ print(
381
+ f"[gpu] encoded reference {tuple(reference.shape)} + guide {tuple(guide.shape)} in {encoded_in:.1f}s",
382
+ flush=True,
383
+ )
384
+
385
+ started = time.time()
386
+ state = PIPE(
387
+ prompt_embeds=prompt_embeds.to("cuda"),
388
+ text_token_tags=text_token_tags,
389
+ # Packed order: reference image(s) first, then the aligned guide — the layout reserves them in this order.
390
+ condition_latents=[reference, guide],
391
+ num_reference_images=1,
392
+ height=int(height),
393
+ width=int(width),
394
+ num_frames=int(num_frames),
395
+ num_inference_steps=int(steps) + 1,
396
+ generator=torch.Generator("cpu").manual_seed(int(seed)),
397
+ output_type="np",
398
+ )
399
+ return state.get("videos")[0], encoded_in, time.time() - started
400
+
401
+
402
+ def swap(
403
+ video: str | None,
404
+ reference: str | None,
405
+ details: str = "",
406
+ target: str = "",
407
+ remove_bg: bool = True,
408
+ clip_length: str = DEFAULT_CLIP,
409
+ detail: str = DEFAULT_DETAIL,
410
+ steps: int = DEFAULT_STEPS,
411
+ seed: int = DEFAULT_SEED,
412
+ progress=gr.Progress(track_tqdm=True),
413
+ ):
414
+ """Swap the main subject of a guide video for the character in a reference image.
415
+
416
+ Args:
417
+ video: The guide clip — the scene and the performance to keep. Any resolution / frame rate.
418
+ reference: The character image (`<Picture 1>`) whose identity, body and clothing replace the subject.
419
+ details: A short description of the reference character, e.g. "an elderly man with a grey beard".
420
+ target: Which guide subject to replace when more than one person appears, e.g. "the woman on the left".
421
+ remove_bg: Cut the reference onto neutral gray with `briaai/RMBG-2.0` first (recommended).
422
+ clip_length: How much of the guide to process, as a 17n + 5 frame count at 24 fps.
423
+ detail: Working pixel area; the whole pass runs here, guide and result alike.
424
+ steps: Euler steps. The shipped workflow uses 8.
425
+ seed: Noise seed.
426
+
427
+ Returns:
428
+ The swapped clip, with the guide's own soundtrack, and a one-line report of what ran.
429
+ """
430
+ if LOAD_ERROR:
431
+ raise gr.Error(LOAD_ERROR.replace("**", ""))
432
+ if not video:
433
+ raise gr.Error("Upload a guide video — the scene and performance to keep.")
434
+ if not reference:
435
+ raise gr.Error("Upload a reference character image — the identity to swap in.")
436
+
437
+ max_length = CLIP_LENGTHS.get(clip_length, CLIP_LENGTHS[DEFAULT_CLIP])
438
+ megapixels = DETAIL_LEVELS.get(detail, DETAIL_LEVELS[DEFAULT_DETAIL])
439
+
440
+ progress(0.02, desc="Planning the guide clip ...")
441
+ plan = h3_video.plan_clip(video, megapixels, max_length)
442
+ height, width, num_frames = plan["height"], plan["width"], plan["num_frames"]
443
+
444
+ ip_token = _caller_ip_token()
445
+ if remove_bg:
446
+ progress(0.08, desc="Removing the reference background ...")
447
+ reference = remove_background(reference, ip_token)
448
+
449
+ details = (details or "").strip() or DEFAULT_DETAILS
450
+ target = (target or "").strip()
451
+ prompt = (PROMPT_TARGETED if target else PROMPT_MAIN).format(details=details, target=target)
452
+
453
+ progress(0.14, desc="Encoding the reference and caption ...")
454
+ prompt_embeds, text_token_tags = encode_prompt(prompt, reference, num_frames)
455
+
456
+ progress(0.2, desc="Sizing the reference image ...")
457
+ ref_pixels = h3_video.load_image(reference, REFERENCE_MEGAPIXELS)
458
+
459
+ progress(0.26, desc=f"Re-timing {plan['source_frames']} frames to 24 fps ...")
460
+ pixels = h3_video.load_guide(video, plan)
461
+
462
+ progress(0.34, desc=f"Swapping {num_frames} frames at {width}x{height} in {int(steps)} steps ...")
463
+ frames, encoded_in, denoised_in = _swap(
464
+ ref_pixels, pixels, prompt_embeds, text_token_tags, height, width, num_frames, int(steps), int(seed)
465
+ )
466
+
467
+ progress(0.94, desc="Encoding the result ...")
468
+ stem = os.path.join(OUTPUT_DIR, uuid.uuid4().hex)
469
+ silent = h3_video.write_video(frames, f"{stem}.mp4")
470
+ out = h3_video.mux_source_audio(silent, video, num_frames, f"{stem}-audio.mp4")
471
+
472
+ report = (
473
+ f"**{width}x{height} · {num_frames} frames ({num_frames / FPS:.2f} s @ {FPS} fps)** — "
474
+ f"guide {plan['source_width']}x{plan['source_height']} @{plan['source_fps']:.4g} fps · "
475
+ f"encode {encoded_in:.1f} s · {int(steps)} steps in {denoised_in:.1f} s · seed {int(seed)}"
476
+ + ("" if out is not silent else " · no soundtrack in the guide")
477
+ )
478
+ return out, report
479
+
480
+
481
+ # ── UI ──────────────────────────────────────────────────────────────────────
482
+
483
+ CSS = """
484
+ .contain { max-width: 1200px !important; margin: auto; }
485
+ #col-out video { max-height: 560px; }
486
+ """
487
+
488
+ with gr.Blocks(title="MiniMax-H3 · character swap") as demo:
489
+ gr.Markdown(
490
+ """
491
+ # MiniMax-H3 — character swap
492
+ `minimax_h3_character_swap_v1.0_ref2va_r64_gl` from [`Alissonerdx/Minimax-H3-ComfyUI`](https://huggingface.co/Alissonerdx/Minimax-H3-ComfyUI)
493
+ **replaces the main subject of a guide video with the character in a reference image**, keeping the guide's
494
+ scene, camera, motion and performance. The guide is packed as an aligned
495
+ [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) guide latent (`MiniMaxH3AddGuide`), and the
496
+ character enters on H3's native image-reference path as `<Picture 1>`. Output resolution follows the guide;
497
+ the soundtrack is the guide's own.
498
+ """
499
+ )
500
+
501
+ with gr.Row():
502
+ with gr.Column():
503
+ video = gr.Video(label="Guide video (scene + performance)", height=300)
504
+ reference = gr.Image(label="Reference character (Picture 1)", type="filepath", height=240)
505
+ run = gr.Button("Swap character", variant="primary")
506
+ details = gr.Textbox(
507
+ label="Reference description",
508
+ placeholder="an elderly man with a grey beard and a dark jacket",
509
+ lines=2,
510
+ info="Describe the reference character. Clear descriptions help the swap hold identity.",
511
+ )
512
+ with gr.Accordion("Advanced", open=False):
513
+ target = gr.Textbox(
514
+ label="Which subject to replace (optional)",
515
+ placeholder="the woman on the left",
516
+ info="Only needed when several people appear; otherwise the most prominent subject is swapped.",
517
+ )
518
+ remove_bg = gr.Checkbox(
519
+ label="Remove reference background (briaai/RMBG-2.0)", value=True,
520
+ info="Cuts the subject onto neutral gray first, as the ComfyUI workflow does.",
521
+ )
522
+ with gr.Row():
523
+ clip_length = gr.Dropdown(
524
+ label="Clip length", choices=list(CLIP_LENGTHS), value=DEFAULT_CLIP,
525
+ info="Trimmed from the start. Longer costs quadratically more.",
526
+ )
527
+ detail = gr.Dropdown(
528
+ label="Detail level", choices=list(DETAIL_LEVELS), value=DEFAULT_DETAIL,
529
+ info="Working pixel area, on the guide's own aspect ratio.",
530
+ )
531
+ steps = gr.Slider(
532
+ label="Steps", minimum=4, maximum=16, step=1, value=DEFAULT_STEPS,
533
+ info="Euler steps. The shipped workflow uses 8 on top of a few-step turbo LoRA.",
534
+ )
535
+ seed = gr.Slider(label="Seed", minimum=0, maximum=2**31 - 1, step=1, value=DEFAULT_SEED)
536
+ with gr.Column(elem_id="col-out"):
537
+ result = gr.Video(label="Swapped", height=300, autoplay=True)
538
+ report = gr.Markdown()
539
+
540
+ gr.Markdown(
541
+ """
542
+ ---
543
+ **How it runs.** Weights are `multimodalart/MiniMax-H3-Pruned`'s `transformer_ref` (the `ref2va` partition
544
+ with rank-8 AdaLN projections), with the Character Swap LoRA and
545
+ [`lightx2v/Minimax-h3-Turbo`](https://huggingface.co/lightx2v/Minimax-h3-Turbo)'s 4-step LoRA both folded in
546
+ at strength 1.0. The guide clip is prepared by
547
+ [`ostris/ComfyUI-AIToolkit-MiniMaxH3`](https://github.com/ostris/ComfyUI-AIToolkit-MiniMaxH3)'s recipe and
548
+ packed on the target's own grid; the reference image is background-removed by
549
+ [`briaai/RMBG-2.0`](https://huggingface.co/briaai/RMBG-2.0) and encoded — as `<Picture 1>` vision tokens and
550
+ a VAE reference block — by [`multimodalart/qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner).
551
+ The repo's optional BFS multi-shot planner is not ported: this is one guide, one reference, one swap.
552
+
553
+ Use only with appropriate rights and consent for the reference images and guide videos; not for
554
+ impersonation, fraud, harassment, or other unlawful purposes. See the
555
+ [Character Swap license](https://huggingface.co/Alissonerdx/Minimax-H3-ComfyUI/blob/main/LICENSE_CHARACTER_SWAP.md).
556
+ """
557
+ )
558
+
559
+ run.click(
560
+ swap,
561
+ inputs=[video, reference, details, target, remove_bg, clip_length, detail, steps, seed],
562
+ outputs=[result, report],
563
+ api_name="swap",
564
+ )
565
+
566
+ if load_models():
567
+ print(LOAD_ERROR, flush=True)
568
+
569
+ if __name__ == "__main__":
570
+ demo.launch(
571
+ theme=gr.themes.Citrus(),
572
+ css=CSS,
573
+ show_error=True,
574
+ allowed_paths=[OUTPUT_DIR],
575
+ mcp_server=True,
576
+ )
h3_charswap_blocks.py ADDED
@@ -0,0 +1,294 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """The **character-swap** conditioning branch of MiniMax-H3, which diffusers itself does not carry.
2
+
3
+ `Alissonerdx/Minimax-H3-ComfyUI`'s Character Swap LoRA
4
+ (`loras/minimax_h3_character_swap_v1.0_ref2va_r64_gl.safetensors`) replaces the main subject of a
5
+ *guide* video with the character in a *reference image*, while keeping the guide's scene, camera and
6
+ performance. It is trained on the `ref2va` partition and conditions through **two** visual blocks at
7
+ once, which is exactly what its own training metadata records:
8
+
9
+ minimax_h3_reference_rope = {"guide_rope_layout": "overlap", "reference_rope_layout": "native", ...}
10
+ minimax_h3_dataset_control_roles = [["guide", "reference", "reference"]]
11
+ guide_latent_only = true
12
+
13
+ * **guide — `overlap`.** The whole guide clip, VAE-encoded at the target resolution and packed on the
14
+ target's *own* `(t, h, w)` rotary grid: guide token `(t, y, x)` sits exactly where target token
15
+ `(t, y, x)` sits. This is ComfyUI's `MiniMaxH3AddGuide(frame_idx = 0)` and is identical to the
16
+ aligned guide the LMS sharpening Space uses — see the sibling `MiniMaxH3GuidePrepareLayoutStep`.
17
+ * **reference — `native`.** The character image on H3's native image-reference path: a Qwen3-VL
18
+ `<Picture 1>` vision block inside `prompt_embeds` (produced upstream by the conditioner Space's
19
+ `/encode_ref2va`) **and** a VAE-encoded reference block in the sequence, laid out exactly as
20
+ `diffusers`' own `MiniMaxH3Ref2VAPrepareLayoutStep` lays out an image reference — one integer rotary
21
+ slot, its own aspect-normalized spatial grid.
22
+
23
+ So the packed layout is `[text (+ Picture 1 vision) | reference image | guide | target audio | target
24
+ video]`. The reference image takes the single integer rotary slot `num_text_tokens`; everything after
25
+ it shares the origin `num_text_tokens + num_references`, so the guide and the target overlap each other
26
+ on that one grid. With no references this reduces byte for byte to the aligned-guide layout.
27
+
28
+ Everything downstream is stock diffusers and needs no change:
29
+
30
+ * `MiniMaxH3PrepareLatentsStep` draws one noise per entry of `condition_latents`, in packed order, noises
31
+ each to `t = keyframe_noise_aug` (0.999) and packs them ahead of the target rows. We hand it
32
+ `[reference_latents, guide_latents]`, so the reference rows land first and the guide rows next, which
33
+ is the order this layout reserves.
34
+ * `MiniMaxH3SetTimestepsStep` pins every one of the `num_condition_video_rows` leading video rows —
35
+ reference and guide alike — at `max(timestep, keyframe_noise_aug)` for every step, so both survive the
36
+ loop as conditioning.
37
+ * `MiniMaxH3Ref2VADenoiseStep` runs the loop against the `transformer_ref` partition the LoRA is trained on.
38
+ """
39
+
40
+ import numpy as np
41
+ import torch
42
+ from diffusers.modular_pipelines.minimax_h3.before_denoise import (
43
+ MiniMaxH3PrepareLatentsStep,
44
+ MiniMaxH3PrepareLayoutStep,
45
+ MiniMaxH3SetTimestepsStep,
46
+ _fill_audio_positions,
47
+ _frame_position_grid,
48
+ _temporal_position_grid,
49
+ )
50
+ from diffusers.modular_pipelines.minimax_h3.decoders import (
51
+ MiniMaxH3AfterDenoiseStep,
52
+ MiniMaxH3VideoDecodeStep,
53
+ )
54
+ from diffusers.modular_pipelines.minimax_h3.denoise import MiniMaxH3Ref2VADenoiseStep
55
+ from diffusers.modular_pipelines.minimax_h3.modular_pipeline import (
56
+ MINIMAX_H3_AUDIO_CHANNELS,
57
+ MINIMAX_H3_AUDIO_TAG,
58
+ MINIMAX_H3_VIDEO_TAG,
59
+ align_num_frames,
60
+ audio_latent_num_frames,
61
+ resolve_canvas_size,
62
+ video_latent_num_frames,
63
+ )
64
+ from diffusers.modular_pipelines.modular_pipeline import SequentialPipelineBlocks
65
+ from diffusers.modular_pipelines.modular_pipeline_utils import InputParam
66
+
67
+
68
+ class MiniMaxH3CharswapPrepareLayoutStep(MiniMaxH3PrepareLayoutStep):
69
+ """`MiniMaxH3PrepareLayoutStep` building the `[text | reference image | guide | audio | video]` layout.
70
+
71
+ The reference image(s) use H3's native image-reference rope; the guide overlaps the target grid. The
72
+ reference geometry is read off `condition_latents`, whose leading entries are the reference latents and
73
+ whose last entry is the aligned guide.
74
+ """
75
+
76
+ model_name = "minimax-h3"
77
+
78
+ @property
79
+ def description(self) -> str:
80
+ return (
81
+ "Resolves the geometry of a character-swap request and builds the `[text | reference image | "
82
+ "guide | target audio | target video]` packed layout: the reference image on H3's native "
83
+ "image-reference rope, the guide overlapping the target's own grid (ComfyUI's "
84
+ "`MiniMaxH3AddGuide` at `frame_idx = 0`)."
85
+ )
86
+
87
+ @property
88
+ def inputs(self) -> list[InputParam]:
89
+ return super().inputs + [
90
+ InputParam(
91
+ name="condition_latents",
92
+ type_hint=list,
93
+ required=True,
94
+ description=(
95
+ "Encoded conditioning latents in packed order: one `(1, C, 1, h, w)` tensor per reference "
96
+ "image, then the aligned guide `(1, C, num_latent_frames, latent_h, latent_w)`."
97
+ ),
98
+ ),
99
+ InputParam(
100
+ name="num_reference_images",
101
+ type_hint=int,
102
+ default=1,
103
+ description="How many leading entries of `condition_latents` are reference images (the rest is the guide).",
104
+ ),
105
+ ]
106
+
107
+ def build_charswap_sequence(
108
+ self,
109
+ text_token_tags: torch.Tensor,
110
+ reference_shapes: list[tuple[int, int, int]],
111
+ num_latent_frames: int,
112
+ latent_height: int,
113
+ latent_width: int,
114
+ num_audio_latents: int,
115
+ patch_size: tuple[int, int, int],
116
+ ):
117
+ r"""Build `[text (+Picture vision) | reference image(s) | guide | target audio | target video]`.
118
+
119
+ `reference_shapes` is one `(latent_frames, latent_h, latent_w)` per reference image, read off the
120
+ encoded reference latents so the reserved rows can never disagree with what the encoder produced.
121
+ """
122
+ _, patch_h, patch_w = patch_size
123
+ num_text_tokens = text_token_tags.shape[0]
124
+ rows_per_frame = (latent_height // patch_h) * (latent_width // patch_w)
125
+
126
+ reference_rows = [
127
+ frames * (height // patch_h) * (width // patch_w) for frames, height, width in reference_shapes
128
+ ]
129
+ num_reference_video_rows = int(sum(reference_rows))
130
+ num_guide_rows = num_latent_frames * rows_per_frame
131
+ num_condition_video_rows = num_reference_video_rows + num_guide_rows
132
+ num_audio_rows = num_audio_latents * MINIMAX_H3_AUDIO_CHANNELS
133
+ num_video_rows = num_latent_frames * rows_per_frame
134
+ sequence_length = (
135
+ num_text_tokens + num_reference_video_rows + num_guide_rows + num_audio_rows + num_video_rows
136
+ )
137
+
138
+ position_ids = torch.zeros(sequence_length, 3, dtype=torch.float64)
139
+ position_ids[:num_text_tokens, 0] = torch.arange(num_text_tokens, dtype=torch.float64)
140
+
141
+ video_index_blocks = []
142
+ cursor = num_text_tokens
143
+ rotary_time = float(num_text_tokens)
144
+
145
+ # Reference images — native rope. Each image is a single frame on its own aspect-normalized grid and
146
+ # takes one integer rotary slot, exactly as `MiniMaxH3Ref2VAPrepareLayoutStep` lays an image out.
147
+ for (frames, ref_h, ref_w), rows_count in zip(reference_shapes, reference_rows):
148
+ rows = slice(cursor, cursor + rows_count)
149
+ cursor = rows.stop
150
+ frame_grid, _ = _frame_position_grid(ref_h, ref_w, patch_h, patch_w)
151
+ position_ids[rows, 0] = rotary_time
152
+ position_ids[rows, 1:] = frame_grid.repeat(frames, 1)
153
+ video_index_blocks.append(torch.arange(rows.start, rows.stop))
154
+ rotary_time += 1.0
155
+
156
+ # Everything after the references shares this origin: the guide overlaps the target, both built on the
157
+ # target's own `(t, h, w)` grid starting here.
158
+ origin = rotary_time
159
+ target_frame_grid, target_width_grid = _frame_position_grid(latent_height, latent_width, patch_h, patch_w)
160
+ media_time = _temporal_position_grid(num_latent_frames, origin)
161
+ media_position_t = media_time.repeat_interleave(target_frame_grid.shape[0])
162
+ media_position_hw = target_frame_grid.repeat(num_latent_frames, 1)
163
+
164
+ # Guide block (overlap).
165
+ guide_rows = slice(cursor, cursor + num_guide_rows)
166
+ cursor = guide_rows.stop
167
+ position_ids[guide_rows, 0] = media_position_t
168
+ position_ids[guide_rows, 1:] = media_position_hw
169
+ video_index_blocks.append(torch.arange(guide_rows.start, guide_rows.stop))
170
+
171
+ # Target audio, then target video (overlapping the guide on the same grid).
172
+ audio_start = cursor
173
+ video_start = audio_start + num_audio_rows
174
+ _fill_audio_positions(
175
+ position_ids, slice(audio_start, video_start), num_audio_latents, origin, target_width_grid
176
+ )
177
+ position_ids[video_start:, 0] = media_position_t
178
+ position_ids[video_start:, 1:] = media_position_hw
179
+
180
+ video_indices = torch.cat(video_index_blocks + [torch.arange(video_start, sequence_length)])
181
+ audio_indices = torch.arange(audio_start, video_start)
182
+ text_indices = torch.arange(num_text_tokens)
183
+
184
+ token_tags = torch.empty(sequence_length, dtype=torch.long)
185
+ token_tags[text_indices] = text_token_tags.to(torch.long)
186
+ token_tags[audio_indices] = MINIMAX_H3_AUDIO_TAG
187
+ token_tags[video_indices] = MINIMAX_H3_VIDEO_TAG
188
+
189
+ return position_ids, token_tags, video_indices, audio_indices, text_indices, num_condition_video_rows, 0
190
+
191
+ @torch.no_grad()
192
+ def __call__(self, components, state):
193
+ block_state = self.get_block_state(state)
194
+ device = components._execution_device
195
+
196
+ if block_state.height is None:
197
+ block_state.height, block_state.width = resolve_canvas_size(16, 9, components.canvas_multiple)
198
+ if block_state.height % components.canvas_multiple or block_state.width % components.canvas_multiple:
199
+ raise ValueError(
200
+ f"`height` and `width` must be multiples of {components.canvas_multiple}, got "
201
+ f"{block_state.height}x{block_state.width}."
202
+ )
203
+
204
+ frames_per_chunk = components.vae_frames_per_chunk
205
+ latents_per_chunk = components.vae_latents_per_chunk
206
+ block_state.num_frames = align_num_frames(block_state.num_frames, frames_per_chunk, latents_per_chunk)
207
+ duration = block_state.num_frames / components.fps
208
+ if not components.min_duration <= duration <= components.max_duration:
209
+ raise ValueError(
210
+ f"MiniMax-H3 generates between {components.min_duration} and {components.max_duration} seconds "
211
+ f"at {components.fps} fps, got {block_state.num_frames} frames ({duration:g} s)."
212
+ )
213
+
214
+ ratio = components.vae_spatial_compression_ratio
215
+ block_state.num_latent_frames = video_latent_num_frames(
216
+ block_state.num_frames, frames_per_chunk, latents_per_chunk
217
+ )
218
+ block_state.latent_height = block_state.height // ratio
219
+ block_state.latent_width = block_state.width // ratio
220
+ block_state.num_audio_latents = audio_latent_num_frames(block_state.num_frames)
221
+
222
+ num_reference_images = int(getattr(block_state, "num_reference_images", 1) or 0)
223
+ reference_shapes = [
224
+ tuple(int(dim) for dim in latents.shape[2:5])
225
+ for latents in block_state.condition_latents[:num_reference_images]
226
+ ]
227
+
228
+ (
229
+ position_ids,
230
+ token_tags,
231
+ video_indices,
232
+ audio_indices,
233
+ text_indices,
234
+ block_state.num_condition_video_rows,
235
+ block_state.num_condition_audio_rows,
236
+ ) = self.build_charswap_sequence(
237
+ block_state.text_token_tags,
238
+ reference_shapes,
239
+ block_state.num_latent_frames,
240
+ block_state.latent_height,
241
+ block_state.latent_width,
242
+ block_state.num_audio_latents,
243
+ components.patch_size,
244
+ )
245
+ block_state.position_ids = position_ids.to(device)
246
+ block_state.token_tags = token_tags.to(device)
247
+ block_state.video_indices = video_indices.to(device)
248
+ block_state.audio_indices = audio_indices.to(device)
249
+ block_state.text_indices = text_indices.to(device)
250
+
251
+ self.set_block_state(state, block_state)
252
+ return components, state
253
+
254
+
255
+ class MiniMaxH3CharswapCoreDenoiseStep(SequentialPipelineBlocks):
256
+ """The character-swap denoising core: combined layout, stock latents / timesteps, `transformer_ref` loop."""
257
+
258
+ model_name = "minimax-h3"
259
+ block_classes = [
260
+ MiniMaxH3CharswapPrepareLayoutStep,
261
+ MiniMaxH3PrepareLatentsStep,
262
+ MiniMaxH3SetTimestepsStep,
263
+ MiniMaxH3Ref2VADenoiseStep,
264
+ ]
265
+ block_names = ["prepare_layout", "prepare_latents", "set_timesteps", "denoise"]
266
+
267
+ @property
268
+ def description(self):
269
+ return (
270
+ "Core denoising workflow for a character-swap request: builds the `[text | reference image | "
271
+ "guide | target audio | target video]` layout, packs the reference and guide conditioning latents, "
272
+ "plans the per-row timesteps and runs the loop against the `transformer_ref` partition."
273
+ )
274
+
275
+
276
+ class MiniMaxH3CharswapGeneratorBlocks(SequentialPipelineBlocks):
277
+ """The denoising half of a split character-swap deployment.
278
+
279
+ `prompt_embeds` / `text_token_tags` (with the `<Picture 1>` vision block) arrive over the wire from the
280
+ conditioner Space, and both visual conditions arrive already VAE-encoded as `condition_latents` —
281
+ `[reference_image_latents, guide_latents]` — so neither the 62 GiB Qwen3-VL nor a resize step is needed
282
+ here. The soundtrack is the guide clip's own, muxed by the caller, so the audio VAE is not loaded either.
283
+ """
284
+
285
+ model_name = "minimax-h3"
286
+ block_classes = [MiniMaxH3CharswapCoreDenoiseStep, MiniMaxH3AfterDenoiseStep, MiniMaxH3VideoDecodeStep]
287
+ block_names = ["denoise", "after_denoise", "decode"]
288
+
289
+ @property
290
+ def description(self):
291
+ return (
292
+ "The denoising half of a split MiniMax-H3 character-swap deployment: combined reference/guide "
293
+ "layout, joint audio/video denoising against `transformer_ref`, and a video-only decode."
294
+ )
h3_lora.py ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Fold a ComfyUI-format MiniMax-H3 LoRA into the diffusers `MiniMaxH3Transformer3DModel` module tree.
2
+
3
+ Both LoRAs this Space stacks are exported for ComfyUI and name the *reference* implementation's modules
4
+ (`blocks.N.attn.qkv_proj`, `mlp.fc1` / `fc2`, `attn.out_proj`, `adaln_proj.linear`), so each one has to be replayed
5
+ through the same layout transforms `scripts/convert_minimax_h3_to_diffusers.py` applies to the base weights — a
6
+ delta is only valid in the layout of the weight it is added to.
7
+
8
+ | ComfyUI target | diffusers parameter | transform |
9
+ |---|---|---|
10
+ | `blocks.N.…` | `transformer_blocks.N.…` | rename |
11
+ | `token_refiner.blocks.N.…` | `token_refiner.refiner_blocks.N.…` | rename |
12
+ | `attn.out_proj` | `attn.to_out.0` | rename |
13
+ | `mlp.fc2` | `ff.net.2` | rename |
14
+ | `adaln_proj.linear` | `adaln_proj.linear` | identity |
15
+ | `final_layer.adaln_proj.linear` | `norm_out.linear` | rename |
16
+ | `mlp.fc1` | `ff.net.0.proj` | fused halves swapped, ComfyUI `[gate; value]` -> diffusers `[value; gate]` |
17
+ | `attn.qkv_proj` | `attn.to_q` / `to_k` / `to_v` | split into contiguous thirds of `inner_dim` rows |
18
+
19
+ Two details that are easy to get wrong, and that the LoRAs themselves document:
20
+
21
+ * **SwiGLU order.** `lightx2v/Minimax-h3-Turbo`'s own metadata records the mapping it applied on export as
22
+ `"Diffusers [value;gate] -> ComfyUI [gate;value]"`, so coming back the other way the two halves swap.
23
+ * **QKV rows are already contiguous.** Raw `MiniMaxAI/MiniMax-H3` shards store fused QKV rows *per-head
24
+ interleaved*, and a LoRA trained against those has to be de-interleaved first. A ComfyUI checkpoint does not:
25
+ both ComfyUI (`comfy/ldm/minimax/model.py`) and ai-toolkit read the projection with
26
+ `.split(heads * head_dim, dim=-1)` / `.chunk(3, dim=-1)`, i.e. `[q_all; k_all; v_all]`. Both LoRAs here are
27
+ ComfyUI exports, so the thirds are contiguous and de-interleaving them would scatter every head.
28
+
29
+ The fold is `W' = W + scale * (lora_B @ lora_A)`, accumulated in float32, with `scale = alpha / rank` read off the
30
+ file's own `.alpha` scalars when it carries them (`lightx2v`'s does: `alpha 8 / rank 128`, and `alpha 24 / rank 384`
31
+ on the block-diagonal fused QKV, both `0.0625`) and `1.0` when it does not (ai-toolkit writes no alphas, so alpha
32
+ equals rank). Merging rather than attaching a PEFT adapter keeps the served graph a plain transformer, which is what
33
+ ZeroGPU's weight packing and any later AoTI export want.
34
+
35
+ Every LoRA target has to resolve, or loading fails: a silently skipped target means the name mapping is wrong and
36
+ the Space would serve a half-applied adapter.
37
+ """
38
+
39
+ import torch
40
+ from safetensors.torch import load_file
41
+
42
+
43
+ def _diffusers_stem(name: str) -> str:
44
+ """Map a ComfyUI module path (already stripped of `diffusion_model.`) onto the diffusers tree."""
45
+ if name.startswith("token_refiner.blocks."):
46
+ stem = name.replace("token_refiner.blocks.", "token_refiner.refiner_blocks.", 1)
47
+ elif name.startswith("blocks."):
48
+ stem = name.replace("blocks.", "transformer_blocks.", 1)
49
+ else:
50
+ stem = name
51
+ return stem.replace("final_layer.adaln_proj.linear", "norm_out.linear")
52
+
53
+
54
+ def _targets(name: str, lora_b: torch.Tensor, inner_dim: int):
55
+ """Yield `(diffusers parameter key, row-transformed lora_B)` for one ComfyUI LoRA target."""
56
+ stem = _diffusers_stem(name)
57
+
58
+ if stem.endswith(".attn.qkv_proj"):
59
+ prefix = stem.removesuffix("qkv_proj")
60
+ if lora_b.shape[0] != 3 * inner_dim:
61
+ raise ValueError(
62
+ f"{name}: fused QKV lora_B has {lora_b.shape[0]} rows, expected 3 * {inner_dim}. This does not look "
63
+ "like a ComfyUI-layout MiniMax-H3 LoRA."
64
+ )
65
+ for projection, rows in zip(("to_q", "to_k", "to_v"), lora_b.split(inner_dim, dim=0)):
66
+ yield f"{prefix}{projection}.weight", rows
67
+ elif stem.endswith(".mlp.fc1"):
68
+ gate, value = lora_b.chunk(2, dim=0)
69
+ yield stem.replace(".mlp.fc1", ".ff.net.0.proj") + ".weight", torch.cat([value, gate])
70
+ elif stem.endswith(".mlp.fc2"):
71
+ yield stem.replace(".mlp.fc2", ".ff.net.2") + ".weight", lora_b
72
+ elif stem.endswith(".attn.out_proj"):
73
+ yield stem.replace(".attn.out_proj", ".attn.to_out.0") + ".weight", lora_b
74
+ else:
75
+ # `adaln_proj.linear` (per block) and `norm_out.linear`: same row layout on both sides.
76
+ yield stem + ".weight", lora_b
77
+
78
+
79
+ def fold_comfy_lora(transformer, path: str, strength: float = 1.0, label: str = "lora") -> str:
80
+ """Merge one ComfyUI-format MiniMax-H3 LoRA into `transformer`'s weights in place.
81
+
82
+ Args:
83
+ transformer: A `MiniMaxH3Transformer3DModel` (or the AdaLN-pruned subclass of it).
84
+ path: Local path to the LoRA `.safetensors`.
85
+ strength: The ComfyUI `LoraLoaderModelOnly` strength, applied on top of the file's own `alpha / rank`.
86
+ label: Name used in the returned status line.
87
+
88
+ Returns:
89
+ A one-line status string describing what was folded.
90
+ """
91
+ lora = load_file(path)
92
+ heads = transformer.config.num_attention_heads
93
+ inner_dim = heads * transformer.config.attention_head_dim
94
+
95
+ bases = sorted({key[: -len(".lora_A.weight")] for key in lora if key.endswith(".lora_A.weight")})
96
+ if not bases:
97
+ raise ValueError(f"{path} holds no `.lora_A.weight` tensors; this is not a ComfyUI LoRA.")
98
+
99
+ params = dict(transformer.named_parameters())
100
+ folded, ranks, scales, missed = 0, set(), set(), []
101
+
102
+ for base in bases:
103
+ lora_a = lora[f"{base}.lora_A.weight"]
104
+ lora_b = lora.get(f"{base}.lora_B.weight")
105
+ if lora_b is None:
106
+ raise ValueError(f"{base} has a lora_A with no lora_B twin in {path}.")
107
+ rank = int(lora_a.shape[0])
108
+ ranks.add(rank)
109
+ alpha = lora.get(f"{base}.alpha")
110
+ scale = strength * (float(alpha) / rank if alpha is not None else 1.0)
111
+ scales.add(round(scale, 6))
112
+
113
+ name = base.removeprefix("diffusion_model.")
114
+ for key, rows in _targets(name, lora_b, inner_dim):
115
+ param = params.get(key)
116
+ if param is None:
117
+ missed.append(key)
118
+ continue
119
+ # float32 accumulation: a bf16 base would otherwise swallow most of the update.
120
+ delta = scale * (rows.to(torch.float32) @ lora_a.to(torch.float32))
121
+ if delta.shape != param.shape:
122
+ raise ValueError(
123
+ f"LoRA delta for `{key}` is {tuple(delta.shape)} but the base weight is {tuple(param.shape)}. "
124
+ "The LoRA and this transformer disagree on layout."
125
+ )
126
+ param.data = (param.data.float() + delta.to(param.device)).to(param.dtype)
127
+ folded += 1
128
+
129
+ if missed:
130
+ raise ValueError(
131
+ f"{len(missed)} targets of {path} matched no transformer parameter, e.g. {missed[:5]}. The LoRA and the "
132
+ "diffusers transformer disagree on module naming; nothing was folded for them."
133
+ )
134
+
135
+ return (
136
+ f"{label}: {len(bases)} targets -> {folded} weight deltas, rank {sorted(ranks)}, "
137
+ f"scale {sorted(scales)} (strength {strength:g})"
138
+ )
h3_video.py ADDED
@@ -0,0 +1,205 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Clip I/O for the guide-latent pass, following `AIToolkitMiniMaxH3RefVideo` (`ostris/ComfyUI-AIToolkit-MiniMaxH3`).
2
+
3
+ The workflow the LoRA ships (`workflows/minimax_h3_lms_workflow.json`) prepares the source clip with that node at
4
+ `target_megapixels = 0.59`, `max_length = 0`, and the node's docstring is explicit that this is the recipe
5
+ ai-toolkit itself used to *cache the guide during training* — so it is the preparation the LoRA expects, not an
6
+ arbitrary resize. Reproduced here verbatim:
7
+
8
+ * **frame count** — `int(total / src_fps * 24)`, capped by `max_length`, snapped **down** to the model's `17n + 5`
9
+ grid. Down, not up: an up-snap would need frames the source does not have.
10
+ * **frame picks** — real-time pacing from frame 0, `source_frame(i) = min(round(i * src_fps / 24), total - 1)`, tail
11
+ trimmed. Any source fps re-times correctly to H3's fixed 24 fps.
12
+ * **size** — the clip's *own* aspect at `target_megapixels`, rounded to the /32 canvas grid, resampled with bicubic
13
+ + antialias. The guide has to land on the target's grid exactly, so this one size is both the guide's and the
14
+ generation's.
15
+ * **soundtrack** — trimmed to the same real-time window, `round(n / 24 * sr)` samples.
16
+
17
+ The soundtrack is carried through rather than generated: the workflow decodes video only
18
+ (`LTXVSeparateAVLatent` -> `VAEDecode`) and muxes the original audio back, since an enhancement pass has no reason
19
+ to resynthesize a soundtrack it was handed.
20
+ """
21
+
22
+ import math
23
+ import shutil
24
+ import subprocess
25
+
26
+ import av
27
+ import numpy as np
28
+ import torch
29
+
30
+ FPS = 24
31
+
32
+
33
+ def snap_down_17n5(n: int) -> int:
34
+ """Largest `17k + 5 <= n`, floored at 5 — ai-toolkit's `snap_down_17n5`."""
35
+ n = max(5, int(n))
36
+ return ((n - 5) // 17) * 17 + 5
37
+
38
+
39
+ def probe(path: str) -> tuple[int, float, int, int]:
40
+ """Return `(total_frames, fps, width, height)` of a clip, without decoding it."""
41
+ with av.open(path) as container:
42
+ stream = container.streams.video[0]
43
+ fps = float(stream.average_rate or stream.guessed_rate or FPS)
44
+ width, height = int(stream.codec_context.width), int(stream.codec_context.height)
45
+ total = int(stream.frames or 0)
46
+ if total <= 0:
47
+ # Some containers (webm, fragmented mp4) carry no frame count; fall back to the duration.
48
+ seconds = None
49
+ if stream.duration is not None and stream.time_base is not None:
50
+ seconds = float(stream.duration * stream.time_base)
51
+ elif container.duration is not None:
52
+ seconds = float(container.duration) / av.time_base
53
+ total = int(round(seconds * fps)) if seconds else 0
54
+ if total <= 0:
55
+ raise ValueError(f"Could not determine the frame count of {path}.")
56
+ return total, fps, width, height
57
+
58
+
59
+ def plan_clip(path: str, target_megapixels: float, max_length: int = 0) -> dict:
60
+ """Resolve the geometry of the guide pass without touching a single pixel.
61
+
62
+ Returns the `num_frames` / `height` / `width` triple `MiniMaxH3GuidePrepareLayoutStep` will be given, so a
63
+ caller can size its GPU budget and show the user the plan before paying for a decode.
64
+ """
65
+ total, fps, width, height = probe(path)
66
+
67
+ num_frames = int(total / fps * FPS)
68
+ if max_length and max_length > 0:
69
+ num_frames = min(num_frames, int(max_length))
70
+ num_frames = snap_down_17n5(num_frames)
71
+
72
+ scale = math.sqrt(float(target_megapixels) * 1e6 / (width * height))
73
+ out_w = max(32, round(width * scale / 32) * 32)
74
+ out_h = max(32, round(height * scale / 32) * 32)
75
+ return {
76
+ "num_frames": num_frames,
77
+ "height": out_h,
78
+ "width": out_w,
79
+ "source_frames": total,
80
+ "source_fps": fps,
81
+ "source_height": height,
82
+ "source_width": width,
83
+ }
84
+
85
+
86
+ def load_guide(path: str, plan: dict) -> torch.Tensor:
87
+ """Decode and resample the source clip into the guide's `(1, 3, num_frames, height, width)` uint8 tensor.
88
+
89
+ Only the source frames the pacing actually lands on are kept, so a long clip that gets trimmed to a couple of
90
+ seconds never materializes in full.
91
+ """
92
+ num_frames, total, fps = plan["num_frames"], plan["source_frames"], plan["source_fps"]
93
+ picks = [min(round(i * fps / FPS), total - 1) for i in range(num_frames)]
94
+ wanted = sorted(set(picks))
95
+
96
+ decoded: dict[int, np.ndarray] = {}
97
+ wanted_set, last = set(wanted), wanted[-1]
98
+ with av.open(path) as container:
99
+ container.streams.video[0].thread_type = "AUTO"
100
+ for index, frame in enumerate(container.decode(video=0)):
101
+ if index in wanted_set:
102
+ decoded[index] = frame.to_ndarray(format="rgb24")
103
+ if index >= last:
104
+ break
105
+ if not decoded:
106
+ raise ValueError(f"Decoded no frames from {path}.")
107
+ # A container can stop short of the frame count it advertises; clamp onto what actually decoded.
108
+ available = max(decoded)
109
+ frames = np.stack([decoded[min(pick, available)] for pick in picks])
110
+
111
+ pixels = torch.from_numpy(frames).permute(0, 3, 1, 2) # (T, 3, H, W) uint8
112
+ if (pixels.shape[2], pixels.shape[3]) != (plan["height"], plan["width"]):
113
+ pixels = (
114
+ torch.nn.functional.interpolate(
115
+ pixels.float().div_(255.0),
116
+ size=(plan["height"], plan["width"]),
117
+ mode="bicubic",
118
+ antialias=True,
119
+ align_corners=False,
120
+ )
121
+ .clamp_(0.0, 1.0)
122
+ .mul_(255.0)
123
+ .round_()
124
+ .to(torch.uint8)
125
+ )
126
+ return pixels.permute(1, 0, 2, 3)[None].contiguous() # (1, 3, T, H, W)
127
+
128
+
129
+ def write_video(frames, path: str, fps: int = FPS, crf: int = 14) -> str:
130
+ """Encode `(T, H, W, 3)` uint8 frames to H.264 at 24 fps."""
131
+ frames = np.asarray(frames)
132
+ if frames.dtype != np.uint8:
133
+ frames = np.clip(frames * 255.0 if frames.max() <= 1.0 else frames, 0, 255).astype(np.uint8)
134
+ height, width = int(frames.shape[1]), int(frames.shape[2])
135
+ with av.open(path, "w") as container:
136
+ stream = container.add_stream("libx264", rate=fps)
137
+ stream.width, stream.height, stream.pix_fmt = width, height, "yuv420p"
138
+ stream.options = {"crf": str(crf), "preset": "medium"}
139
+ for frame in frames:
140
+ for packet in stream.encode(av.VideoFrame.from_ndarray(np.ascontiguousarray(frame), format="rgb24")):
141
+ container.mux(packet)
142
+ for packet in stream.encode():
143
+ container.mux(packet)
144
+ return path
145
+
146
+
147
+ def has_audio(path: str) -> bool:
148
+ try:
149
+ with av.open(path) as container:
150
+ return bool(container.streams.audio)
151
+ except Exception:
152
+ return False
153
+
154
+
155
+ def mux_source_audio(video_path: str, source_path: str, num_frames: int, out_path: str) -> str:
156
+ """Attach the source clip's own soundtrack, trimmed to the `num_frames / 24` window the pass covers.
157
+
158
+ ai-toolkit trims the reference soundtrack to exactly `round(n / 24 * sr)` samples; `-t` on the audio input is
159
+ the same window expressed in seconds. Falls back to the silent video if ffmpeg is unavailable or the mux fails,
160
+ since a missing soundtrack is not worth failing a generation over.
161
+ """
162
+ if not shutil.which("ffmpeg") or not has_audio(source_path):
163
+ return video_path
164
+ seconds = num_frames / FPS
165
+ command = [
166
+ "ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
167
+ "-i", video_path,
168
+ "-t", f"{seconds:.6f}", "-i", source_path,
169
+ "-map", "0:v:0", "-map", "1:a:0",
170
+ "-c:v", "copy", "-c:a", "aac", "-b:a", "192k", "-shortest",
171
+ out_path,
172
+ ]
173
+ try:
174
+ subprocess.run(command, check=True, capture_output=True)
175
+ except Exception as error:
176
+ print(f"[audio] keeping the silent video, mux failed: {error}", flush=True)
177
+ return video_path
178
+ return out_path
179
+
180
+
181
+ def size_to_megapixels(width: int, height: int, target_megapixels: float) -> tuple[int, int]:
182
+ """`(out_height, out_width)` for `width x height` resized to `target_megapixels` on the /32 canvas grid."""
183
+ scale = math.sqrt(float(target_megapixels) * 1e6 / (width * height))
184
+ out_w = max(32, round(width * scale / 32) * 32)
185
+ out_h = max(32, round(height * scale / 32) * 32)
186
+ return out_h, out_w
187
+
188
+
189
+ def load_image(path: str, target_megapixels: float) -> torch.Tensor:
190
+ """Load a reference image into a `(1, 3, 1, H, W)` uint8 tensor, sized to `target_megapixels` on its own aspect.
191
+
192
+ The single temporal frame is what `encode_vae_condition` runs through the VAE's spatial encoder alone, yielding
193
+ one latent frame — the geometry H3's native image reference occupies.
194
+ """
195
+ from PIL import Image, ImageOps
196
+
197
+ image = ImageOps.exif_transpose(Image.open(path)).convert("RGB")
198
+ out_h, out_w = size_to_megapixels(image.width, image.height, target_megapixels)
199
+ pixels = torch.from_numpy(np.asarray(image)).permute(2, 0, 1)[None].float().div_(255.0) # (1, 3, H, W)
200
+ if (pixels.shape[2], pixels.shape[3]) != (out_h, out_w):
201
+ pixels = torch.nn.functional.interpolate(
202
+ pixels, size=(out_h, out_w), mode="bicubic", antialias=True, align_corners=False
203
+ ).clamp_(0.0, 1.0)
204
+ pixels = pixels.mul_(255.0).round_().to(torch.uint8)
205
+ return pixels.unsqueeze(2).contiguous() # (1, 3, 1, H, W)
packages.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ ffmpeg
requirements.txt ADDED
@@ -0,0 +1,22 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # `diffusers` is installed from the canonical MiniMax-H3 pull request,
2
+ # https://github.com/huggingface/diffusers/pull/14371 ("Minimax h3 follow up (review & refactor)"), pinned to a
3
+ # **commit** rather than to its `minimax-h3-refactor` branch: the PR is a WIP and its head moves, and this Space's
4
+ # `h3_guide_blocks.py` subclasses its block classes. Re-pin — and re-check the block/step names against the new
5
+ # head — whenever the PR updates.
6
+ #
7
+ # 665f578278365ea4a3318cb8c9b66ce6c01204b9 = refs/pull/14371/head at the time of this deploy
8
+ --extra-index-url https://download.pytorch.org/whl/cu130
9
+ diffusers @ git+https://github.com/huggingface/diffusers.git@665f578278365ea4a3318cb8c9b66ce6c01204b9
10
+ torch==2.11.0
11
+ torchvision==0.26.0
12
+ # `diffusers.modular_pipelines.minimax_h3` imports Qwen3-VL at module import, so transformers has to carry it.
13
+ transformers==5.8.0
14
+ accelerate==1.14.0
15
+ # No `kernels` pin on purpose: the Hub attention backends want `kernels>=0.12.3`, and that version breaks
16
+ # transformers 5.8.0 at import.
17
+ # PyAV decodes the source clip and encodes the result; ffmpeg (packages.txt) muxes the source soundtrack back on.
18
+ av
19
+ pillow
20
+ numpy
21
+ requests
22
+ safetensors>=0.8.0