dima0987654321 commited on
Commit
8b21446
·
verified ·
1 Parent(s): bd7344e

Upload 5 files

Browse files
Files changed (4) hide show
  1. README.md +14 -67
  2. app.py +170 -869
  3. h3_split_blocks.py +9 -64
  4. requirements.txt +1 -5
README.md CHANGED
@@ -1,78 +1,25 @@
1
  ---
2
- title: MiniMax-H3 4-Step FlashGen LoRA
3
- emoji: ⚡
4
- colorFrom: blue
5
- colorTo: red
6
  sdk: gradio
7
- sdk_version: 6.25.0
8
  app_file: app.py
 
9
  python_version: "3.12"
10
  startup_duration_timeout: 1h
11
- short_description: 4-step text-to-video+audio with MiniMax-H3 FlashGen LoRA
12
- models:
13
- - MiniMaxAI/MiniMax-H3
14
- - Beidouqixing/minimax-h3-4step-lora-flashgen
15
  ---
16
 
17
- # MiniMax-H3 4-Step FlashGen LoRA
18
 
19
- A demo of the [FlashGen 4-step distillation LoRA](https://huggingface.co/Beidouqixing/minimax-h3-4step-lora-flashgen)
20
- applied to [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3), MiniMax's 33B omni-modal model that generates
21
- video with native stereo audio.
22
 
23
- The FlashGen LoRA uses distribution-matching distillation (DMD2 / VSD, data-free) to reduce inference from ~20 steps
24
- to just **4 steps** — approximately a 5x speedup. The distilled `base_schedule` `[1.0, 0.7, 0.4, 0.15, 0.0]` replaces
25
- the *uniform* `linspace(1, 0, num_inference_steps)` grid `MiniMaxH3Scheduler.set_timesteps` would otherwise build, and
26
- H3's per-modality exponential sigma shift still applies on top of it — a single base schedule can serve both
27
- modalities precisely because it is pre-shift. So the video scheduler runs
28
- σ `1 → 0.9655 → 0.8889 → 0.6792 → 0` (`shift = 12.0`) and the audio scheduler σ `1 → 0.875 → 0.6667 → 0.3462 → 0`
29
- (`shift = 3.0`). Each shifted grid is injected through `MiniMaxH3Scheduler.set_timesteps(sigmas=...)`, which takes it
30
- verbatim, and *pinned* onto its scheduler, because the pipeline's own denoise step calls
31
- `set_timesteps(num_inference_steps, ...)` again — and for this scheduler `num_inference_steps` counts sigma grid
32
- points (terminal `0.0` included), so it runs `len(sigmas) - 1` model evaluations. Pinning is the runtime equivalent of
33
- `merge_lora_ckpt.py` writing `_minimax_h3.base_schedule` into `model_index.json`. Each generation reports the number
34
- of Euler steps the scheduler actually took, read back off `scheduler.step_index`, rather than a hardcoded 4.
35
 
36
- ## How it works
 
 
37
 
38
- | Piece | What runs here |
39
- |---|---|
40
- | Transformer | `MiniMaxH3Transformer3DModel` — the 33B / 50-block H3 DiT, bf16, with the FlashGen LoRA folded in |
41
- | VAEs | `AutoencoderKLMiniMaxH3` (video) + `AutoencoderKLMiniMaxH3Audio` (audio), float32 |
42
- | Schedulers | `MiniMaxH3Scheduler` with the 4-step distilled `base_schedule` `[1.0, 0.7, 0.4, 0.15, 0.0]` under each modality's own shift (video `12.0`, audio `3.0`) |
43
- | Conditioning | Delegated to [`multimodalart/qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner) over the gradio API (the 62 GB Qwen3-VL text encoder does not fit alongside the transformer on a single ZeroGPU worker) |
44
-
45
- ## How the LoRA is applied
46
-
47
- The LoRA repo ships its own merge script,
48
- [`merge_lora_ckpt.py`](https://huggingface.co/Beidouqixing/minimax-h3-4step-lora-flashgen/blob/main/merge_lora_ckpt.py),
49
- and this Space follows it rather than a default diffusers LoRA load. The adapter is **merged** into the base weights
50
- at startup — `W' = W + scale * (lora_B @ lora_A)`, accumulated in fp32, at `scale = lora_alpha / rank = 1.0`
51
- (FlashGen's alpha equals its rank of 64) — instead of being attached as a runtime PEFT adapter, and all 259 LoRA
52
- targets are required to resolve or startup fails, exactly as the reference script does.
53
-
54
- The reference script merges into the *original* MiniMax-H3 partition, whose keys are the LoRA's own
55
- (`blocks.N.attn.qkv_proj.weight`, `mlp.fc1/fc2`, `attn.out_proj`, `adaln_proj.linear`). Merging into the diffusers
56
- port instead requires replaying the same layout transforms
57
- [`convert_minimax_h3_to_diffusers.py`](https://github.com/huggingface/diffusers/blob/main/scripts/convert_minimax_h3_to_diffusers.py)
58
- applied to the base weights, because a delta is only valid in the layout of the weight it is added to:
59
-
60
- | Original LoRA target | diffusers parameter | Transform |
61
- |---|---|---|
62
- | `blocks.N.…` / `token_refiner.blocks.N.…` | `transformer_blocks.N.…` / `token_refiner.refiner_blocks.N.…` | rename |
63
- | `attn.out_proj` | `attn.to_out.0` | rename |
64
- | `mlp.fc2` | `ff.net.2` | rename |
65
- | `final_layer.adaln_proj.linear` | `norm_out.linear` | rename |
66
- | `adaln_proj.linear` | `adaln_proj.linear` | identity |
67
- | `mlp.fc1` | `ff.net.0.proj` | fused halves swapped, `[gate; value]` → `[value; gate]` for diffusers' `SwiGLU` |
68
- | `attn.qkv_proj` | `attn.to_q` / `to_k` / `to_v` | **de-interleave the per-head rows** (`[head0: q k v, head1: q k v, …]` → `[q_all; k_all; v_all]`), *then* split thirds |
69
-
70
- That last row is the one that is easy to get wrong: the checkpoint's fused QKV rows are per-head interleaved, so
71
- splitting them into contiguous thirds — the naive reading — scatters every head's q/k/v across all three projections
72
- and turns the adapter into noise on all 52 attention blocks. With the de-interleave in place, the merged weights
73
- reproduce `merge_lora_ckpt.py`-then-convert to within bf16 rounding.
74
-
75
- ## License
76
-
77
- MiniMax-H3 is covered by the **MiniMax H3 Community License Agreement**. The FlashGen LoRA is Apache-2.0.
78
- Read both before using any output of this Space.
 
1
  ---
2
+ title: MiniMax-H3 Motion Enhancer
3
+ emoji: 🎬
4
+ colorFrom: indigo
5
+ colorTo: purple
6
  sdk: gradio
7
+ sdk_version: 6.26.0
8
  app_file: app.py
9
+ short_description: Animate a still image with MiniMax-H3 motion
10
  python_version: "3.12"
11
  startup_duration_timeout: 1h
 
 
 
 
12
  ---
13
 
14
+ # MiniMax-H3 Motion Enhancer
15
 
16
+ Animate a still image with the 8-step motion-enhancer LoRA for [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3).
17
+ The app generates a short video and the model's native soundtrack.
 
18
 
19
+ ## Model
 
 
 
 
 
 
 
 
 
 
 
20
 
21
+ - LoRA: [rzgar/minimax-h3_fl2v_8Step_motion_enhancer](https://huggingface.co/rzgar/minimax-h3_fl2v_8Step_motion_enhancer)
22
+ - Base: [MiniMaxAI/MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3)
23
+ - License: Apache-2.0 for the LoRA; see the base model's [MiniMax H3 Community License](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE)
24
 
25
+ Use only images you own or have permission to animate. Do not upload private or intimate imagery of real people without their explicit consent.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
app.py CHANGED
@@ -1,282 +1,66 @@
1
- """MiniMax-H3 4-Step FlashGen LoRA — text (and optional keyframes) to video + audio in 4 steps.
2
-
3
- The FlashGen LoRA (``Beidouqixing/minimax-h3-4step-lora-flashgen``) is a data-free DMD2 distillation
4
- adapter (rank 64, alpha == rank) that reduces MiniMax-H3's T2VA inference from ~20 steps to **4**. It
5
- targets the ``transformer/`` partition in the original checkpoint's naming (``blocks.N.attn.qkv_proj``,
6
- ``mlp.fc1/fc2``, ``attn.out_proj``, ``adaln_proj.linear``). Following ``merge_lora_ckpt.py`` — the merge
7
- script the LoRA repo ships — it is *merged* into the base weights at startup rather than attached as a
8
- runtime PEFT adapter: ``W' = W + scale * (lora_B @ lora_A)`` at ``scale = 1.0``, with every target
9
- required to resolve. See the LoRA section below for the layout transforms the diffusers port needs.
10
-
11
- The distilled schedule ``[1.0, 0.7, 0.4, 0.15, 0.0]`` is a ``base_schedule``: it replaces the *uniform*
12
- ``linspace(1, 0, num_inference_steps)`` grid that ``MiniMaxH3Scheduler.set_timesteps`` builds, and H3's
13
- per-modality exponential sigma shift (``shift = 12.0`` video, ``3.0`` audio — the release's
14
- ``_minimax_h3.sigma_shift_scales``) still applies on top of it. A single ``base_schedule`` serves two
15
- modalities precisely because it is pre-shift. It is injected through
16
- ``MiniMaxH3Scheduler.set_timesteps(sigmas=...)``, which takes a sigma grid verbatim, so each scheduler is
17
- handed ``BASE_SCHEDULE`` already pushed through *its own* shift. Note that ``num_inference_steps`` counts
18
- *sigma grid points* for this scheduler (the terminal ``0.0`` included), so a 4-step schedule is 5 points
19
- and the pipeline has to be asked for 5, not 4 — and because the pipeline re-runs ``set_timesteps`` itself
20
- inside the denoise step, the distilled grid is pinned onto both schedulers rather than merely set before
21
- the call.
22
-
23
- H3's 62 GiB Qwen3-VL text encoder does not fit alongside the 66 GiB transformer on a single ZeroGPU
24
- worker, so conditioning is delegated to ``multimodalart/qwen3vl-conditioner`` over the gradio API —
25
- the same split every MiniMax-H3 LoRA Space uses.
26
- """
27
-
28
  from __future__ import annotations
29
 
30
  import os
31
  import tempfile
32
  import time
33
- import traceback
34
- from functools import cache
35
 
36
  import spaces
37
- import gradio as gr
38
- import torch
39
-
40
- MODEL_REPO = os.environ.get("H3_MODEL_REPO", "MiniMaxAI/MiniMax-H3")
41
- LORA_REPO = os.environ.get("H3_LORA_REPO", "Beidouqixing/minimax-h3-4step-lora-flashgen")
42
- LORA_FILE = os.environ.get("H3_LORA_FILE", "minimax_h3_4step_lora_flashgen_v1.0_768p_bf16.safetensors")
43
-
44
- # Additional LoRAs, merged on top of the FlashGen 4-step weights.
45
- MOTION_LORA_REPO = os.environ.get(
46
- "H3_MOTION_LORA_REPO",
47
- "rzgar/minimax-h3_fl2v_8Step_motion_enhancer",
48
- )
49
- MOTION_LORA_FILE = os.environ.get(
50
- "H3_MOTION_LORA_FILE",
51
- "minimax-h3_fl2v_8Step_motion_enhancer.safetensors",
52
- )
53
- MOTION_LORA_SCALE = float(os.environ.get("H3_MOTION_LORA_SCALE", "1.0"))
54
-
55
- # Realism / anatomy LoRAs. These are folded after FlashGen and the existing
56
- # Motion Enhancer, using the same extra-LoRA merge path below.
57
- REALISM_LORA_REPO = os.environ.get(
58
- "H3_REALISM_LORA_REPO",
59
- "fal/MiniMax-H3-Realism-People-LoRA",
60
- )
61
- REALISM_LORA_FILE = os.environ.get(
62
- "H3_REALISM_LORA_FILE",
63
- "h3-realism-people-t2v-i2v-r2v.safetensors",
64
- )
65
- REALISM_LORA_SCALE = float(os.environ.get("H3_REALISM_LORA_SCALE", "0.8"))
66
- REALISM_LORA_TRIGGER = "r34l1sm"
67
 
68
- # Perfect Eyes / Skin / Hands. The original release is on Civitai, but the
69
- # weights are mirrored in a Hugging Face repository so this Space never needs
70
- # to contact Civitai at startup. The filename is pinned; no model-page/API lookup.
71
- DETAIL_LORA_REPO = os.environ.get(
72
- "H3_DETAIL_LORA_REPO",
73
- "Alex995647/loras-minimax-h3",
74
- )
75
- DETAIL_LORA_FILE = os.environ.get(
76
- "H3_DETAIL_LORA_FILE",
77
- "minimax-h3-t2va-t2v-polyhedron-perfect-eyes-perfect-skin/polyhedron_minimax_h3_image_lora_v2__perfe8ct.safetensors",
78
- )
79
- DETAIL_LORA_SCALE = float(os.environ.get("H3_DETAIL_LORA_SCALE", "0.7"))
80
 
81
- # Spatial & Physics: object collisions, gravity, stacking, falling, occlusion, and
82
- # more consistent spatial relationships. The clean pruned H3 adapter has no trigger word.
83
- SPATIAL_LORA_REPO = os.environ.get(
84
- "H3_SPATIAL_LORA_REPO",
85
- "Jojocodex/minimax-h3-spatial-physics-lora",
86
- )
87
- SPATIAL_LORA_FILE = os.environ.get(
88
- "H3_SPATIAL_LORA_FILE",
89
- "wushu_spatial_physics_clean_3000_pruned.safetensors",
90
- )
91
- SPATIAL_LORA_SCALE = float(os.environ.get("H3_SPATIAL_LORA_SCALE", "0.4"))
92
-
93
- # HMBreasts V2: H3 anatomy adapter for chest size/shape control.
94
- # The weight is pulled directly from Hugging Face; there is no Civitai/API lookup.
95
- BREASTS_LORA_REPO = os.environ.get(
96
- "H3_BREASTS_LORA_REPO",
97
- "Hearmeman/minimax-h3-loras",
98
- )
99
- BREASTS_LORA_FILE = os.environ.get(
100
- "H3_BREASTS_LORA_FILE",
101
- "HMBreastsV2.safetensors",
102
- )
103
- BREASTS_LORA_SCALE = float(os.environ.get("H3_BREASTS_LORA_SCALE", "1.0"))
104
 
105
- CONDITIONER_SPACE = os.environ.get("H3_CONDITIONER", "multimodalart/qwen3vl-conditioner")
106
- # ``pack`` places the transformer at startup (66 GB on disk via the spaces pack); VAEs move on first GPU call.
107
- PLACEMENT = os.environ.get("H3_PLACEMENT", "pack").lower()
108
- ATTENTION = os.environ.get("H3_ATTENTION", "_native_cudnn").lower()
109
- GPU_SIZE = os.environ.get("H3_GPU_SIZE", "xlarge")
110
 
111
- # The distilled schedule from the LoRA README: 5 *base* sigma points = 4 Euler steps. This is the grid
112
- # before H3's per-modality shift, i.e. the replacement for `linspace(1, 0, num_inference_steps)` — see
113
- # `shifted_schedule` below.
114
- BASE_SCHEDULE = [1.0, 0.7, 0.4, 0.15, 0.0]
115
- NUM_STEPS = len(BASE_SCHEDULE) - 1 # 4 model evaluations
116
- # `MiniMaxH3Scheduler.set_timesteps(num_inference_steps=...)` counts *sigma grid points*, terminal 0.0
117
- # included, and runs `len(sigmas) - 1` model evaluations (`timesteps = 1 - sigmas[:-1]`). So the number the
118
- # pipeline has to be given for NUM_STEPS Euler steps is NUM_STEPS + 1 == len(BASE_SCHEDULE); passing 4 there
119
- # yields a 4-point grid and only 3 steps.
120
- NUM_SIGMA_POINTS = len(BASE_SCHEDULE) # 5
121
 
122
- FPS, FRAMES_PER_CHUNK, LATENTS_PER_CHUNK = 24, 17, 5
123
- MIN_UI_DURATION, MAX_UI_DURATION = 2, 14
 
 
 
 
 
 
124
 
125
  CANVASES = {
126
- # 16:9
127
  "960x544 · 16:9 fast": (544, 960),
128
  "1024x576 · 16:9 fast": (576, 1024),
129
- "1152x640 · 16:9": (640, 1152),
130
- "1280x704 · 16:9": (704, 1280),
131
- "1344x768 · 16:9 full": (768, 1344),
132
- # 9:16
133
  "544x960 · 9:16 fast": (960, 544),
134
- "640x1152 · 9:16": (1152, 640),
135
- "768x1344 · 9:16 full": (1344, 768),
136
- # 1:1
137
  "544x544 · 1:1 fast": (544, 544),
138
- "768x768 · 1:1 full": (768, 768),
139
- # 4:3 / 3:4
140
  "768x576 · 4:3 fast": (576, 768),
141
- "1024x768 · 4:3 full": (768, 1024),
142
  "576x768 · 3:4 fast": (768, 576),
143
- "768x1024 · 3:4 full": (1024, 768),
144
- # 21:9
145
  "1152x512 · 21:9 fast": (512, 1152),
146
- "1536x672 · 21:9 full": (672, 1536),
147
  }
148
  DEFAULT_CANVAS = "960x544 · 16:9 fast"
149
 
150
- OUTPUT_DIR = os.path.join(tempfile.gettempdir(), "h3-flashgen-out")
151
- os.makedirs(OUTPUT_DIR, exist_ok=True)
152
-
153
- PIPE = None
154
- LOAD_ERROR: str | None = None
155
- LOADED_IN: float | None = None
156
-
157
- def snap_frames(seconds: float) -> int:
158
- """The frame count MiniMax-H3's video VAE can decode: the next ``17 * n + 5`` at 24 fps."""
159
- frames = max(1, round(float(seconds) * FPS))
160
- while frames % FRAMES_PER_CHUNK != LATENTS_PER_CHUNK:
161
- frames += 1
162
- return frames
163
 
164
- def lower_duration_floor(seconds: float = MIN_UI_DURATION) -> None:
165
- """Let the pipeline generate below its 5 s floor."""
166
- from diffusers.modular_pipelines.minimax_h3.modular_pipeline import MiniMaxH3ModularPipeline
167
-
168
- MiniMaxH3ModularPipeline.min_duration = property(lambda self: float(seconds))
169
-
170
- def shifted_schedule(shift: float) -> list[float]:
171
- """``BASE_SCHEDULE`` pushed through H3's exponential sigma shift ``s*σ / (1 + (s - 1)*σ)``.
172
-
173
- ``base_schedule`` is what ``merge_lora_ckpt.py`` pins into ``_minimax_h3.base_schedule``, and the name
174
- is literal: it replaces the *base* grid — ``MiniMaxH3Scheduler.set_timesteps``'s
175
- ``linspace(1, 0, num_inference_steps)``, which the merge script calls "the server's uniform schedule" —
176
- and the release's per-modality ``sigma_shift_scales`` (video ``12.0``, audio ``3.0``) still apply after
177
- it. One schedule can only serve both modalities because it is pre-shift.
178
-
179
- Feeding the base grid to the model as if it were already shifted is what produced the ghosted,
180
- superimposed-frames output: every step conditioned the transformer on a timestep far from the latent's
181
- real noise level (σ 0.7 instead of 0.966, 0.4 instead of 0.889, 0.15 instead of 0.679), so each
182
- prediction was an average over noise levels the student was never distilled at. The shift fixes 1.0 and
183
- 0.0, so the endpoints — and the 4-step count — are unchanged.
184
- """
185
- return [shift * sigma / (1 + (shift - 1) * sigma) for sigma in BASE_SCHEDULE]
186
-
187
- def pin_distilled_schedule(scheduler) -> None:
188
- """Pin the shifted ``BASE_SCHEDULE`` onto a scheduler so *every* ``set_timesteps`` call lands on it.
189
-
190
- Setting the sigmas before calling the pipeline is not enough: the denoise step contains
191
- ``MiniMaxH3SetTimestepsStep``, which calls ``scheduler.set_timesteps(num_inference_steps, device=...)``
192
- itself and so rebuilds the uniform shifted grid, throwing the distilled one away. That grid is also
193
- where the missing step went — ``num_inference_steps`` is a *grid point* count for this scheduler, so
194
- the 4 we asked for produced a 4-sigma grid, i.e. ``len(sigmas) - 1 == 3`` model evaluations.
195
-
196
- The schedule pinned here is ``BASE_SCHEDULE`` under *this* scheduler's own ``shift``, so the video
197
- scheduler (``shift = 12.0``) and the audio one (``shift = 3.0``) keep the distinct grids the two
198
- modalities are denoised on. ``set_timesteps(sigmas=...)`` takes its argument verbatim, so the shift has
199
- to be applied here rather than left to the scheduler.
200
-
201
- Pinning is the equivalent of ``merge_lora_ckpt.py``'s ``_minimax_h3.base_schedule`` in
202
- ``model_index.json``: the served checkpoint carries its distilled schedule and the caller's step count
203
- cannot silently replace it.
204
- """
205
- if getattr(scheduler, "_flashgen_pinned", False):
206
- return
207
-
208
- original = scheduler.set_timesteps
209
- pinned = shifted_schedule(scheduler.shift)
210
-
211
- def set_timesteps(num_inference_steps=None, device=None, sigmas=None):
212
- return original(device=device, sigmas=pinned if sigmas is None else sigmas)
213
-
214
- scheduler.set_timesteps = set_timesteps
215
- scheduler._flashgen_pinned = True
216
-
217
- # ── LoRA loading (weight folding) ─────────────────────────────────────────────
218
- #
219
- # This mirrors ``merge_lora_ckpt.py``, the merge script the LoRA repo itself ships, rather than a
220
- # default ``load_lora_adapter`` + ``set_adapters``. That script folds the adapter into the base
221
- # weights ahead of serving (``W' = W + scale * (lora_B @ lora_A)``, accumulated in fp32, with
222
- # ``scale = lora_alpha / rank = 1.0`` because FlashGen's alpha equals its rank of 64), pins the
223
- # distilled ``base_schedule``, and hard-fails if any LoRA target finds no base weight. We reproduce
224
- # all of that here — merge-and-unload semantics, not a runtime adapter.
225
- #
226
- # The one thing the reference script never has to do is translate names: it merges into the
227
- # *original* MiniMax-H3 partition, whose keys are exactly the LoRA's own
228
- # (``blocks.N.attn.qkv_proj.weight``). Merging into the diffusers port instead means replaying the
229
- # very same transforms ``scripts/convert_minimax_h3_to_diffusers.py`` applied to the base weights,
230
- # because a delta is only valid in the layout of the weight it is added to:
231
- #
232
- # * ``blocks.`` -> ``transformer_blocks.``, ``token_refiner.blocks.`` ->
233
- # ``token_refiner.refiner_blocks.``, ``final_layer.adaln_proj.linear`` -> ``norm_out.linear``,
234
- # ``attn.out_proj`` -> ``attn.to_out.0``, ``mlp.fc2`` -> ``ff.net.2`` (pure renames),
235
- # * ``mlp.fc1`` -> ``ff.net.0.proj`` with its two fused halves swapped, because diffusers' SwiGLU
236
- # reads ``[value; gate]`` where the checkpoint stores ``[gate; value]``,
237
- # * ``attn.qkv_proj`` -> ``to_q`` / ``to_k`` / ``to_v``, and this is the subtle one: the fused QKV
238
- # rows of the original checkpoint are **per-head interleaved**
239
- # (``[head0: q k v, head1: q k v, ...]``), so they must be de-interleaved into
240
- # ``[q_all; k_all; v_all]`` *before* being split into thirds. Splitting the raw rows into
241
- # contiguous thirds directly — the naive reading — scatters every head's q/k/v across all three
242
- # projections, which turns the adapter into structured noise on all 52 attention blocks.
243
-
244
- def _reorder_interleaved_qkv(weight, num_heads: int, head_dim: int):
245
- """De-interleave per-head fused-QKV rows into ``[q_all; k_all; v_all]``.
246
-
247
- Identical to ``reorder_interleaved_qkv`` in diffusers' MiniMax-H3 conversion script, which
248
- applies it to the base fused QKV weight before splitting it. ``lora_B`` indexes that same fused
249
- output space, so the delta needs the same reorder.
250
- """
251
  grouped = weight.reshape(num_heads, 3 * head_dim, *weight.shape[1:])
252
  query, key, value = grouped.split(head_dim, dim=1)
253
  return torch.cat(
254
  [part.reshape(num_heads * head_dim, *weight.shape[1:]) for part in (query, key, value)], dim=0
255
  )
256
 
 
257
  def _lora_target_name(source_name: str) -> str:
258
- """Map an original-checkpoint base name (without ``.lora_A/B.*``) to the diffusers parameter path."""
259
- if source_name.startswith("token_refiner.blocks."):
260
- target = source_name.replace("token_refiner.blocks.", "token_refiner.refiner_blocks.", 1)
261
- elif source_name.startswith("blocks."):
262
- target = source_name.replace("blocks.", "transformer_blocks.", 1)
263
- else:
264
- target = source_name
265
  return target.replace("final_layer.adaln_proj.linear", "norm_out.linear")
266
 
267
- def _lora_targets(name: str, b_weight, num_heads: int, head_dim: int):
268
- """Yield ``(diffusers_param_key, row_transformed_B)`` for one original LoRA base name."""
269
- target = _lora_target_name(name)
270
 
 
 
271
  if target.endswith(".attn.qkv_proj"):
272
  prefix = target.removesuffix("qkv_proj")
273
- # De-interleave the per-head rows first, then split contiguous thirds — exactly the
274
- # composition the base fused QKV weight went through during conversion.
275
  rows = _reorder_interleaved_qkv(b_weight, num_heads, head_dim)
276
  for kind, part in zip(("q", "k", "v"), rows.split(num_heads * head_dim, dim=0)):
277
  yield f"{prefix}to_{kind}.weight", part.contiguous()
278
  elif target.endswith(".mlp.fc1"):
279
- # SwiGLU gate/value swap: the checkpoint stores [gate, value]; diffusers stores [value, gate].
280
  gate, value = b_weight.chunk(2, dim=0)
281
  yield target.replace(".mlp.fc1", ".ff.net.0.proj") + ".weight", torch.cat([value, gate]).contiguous()
282
  elif target.endswith(".mlp.fc2"):
@@ -284,359 +68,108 @@ def _lora_targets(name: str, b_weight, num_heads: int, head_dim: int):
284
  elif target.endswith(".attn.out_proj"):
285
  yield target.replace(".attn.out_proj", ".attn.to_out.0") + ".weight", b_weight
286
  else:
287
- # adaln_proj.linear (block-level) and norm_out.linear: identical row layout.
288
  yield target + ".weight", b_weight
289
 
290
- def load_and_apply_lora(transformer) -> str:
291
- """Download the FlashGen LoRA, remap keys, and fold ``scale * (B @ A)`` into the base weights."""
292
- import torch
293
- from huggingface_hub import hf_hub_download
294
- from safetensors.torch import load_file
295
-
296
- lora_path = hf_hub_download(LORA_REPO, LORA_FILE)
297
- lora = load_file(lora_path)
298
-
299
- # Same pairing and validation contract as merge_lora_ckpt.py's `load_lora_pairs`.
300
- suffix_a, suffix_b = ".lora_A.default.weight", ".lora_B.default.weight"
301
- bases = sorted({key[: -len(suffix_a)] for key in lora if key.endswith(suffix_a)})
302
- unexpected = [key for key in lora if not key.endswith((suffix_a, suffix_b))]
303
- if unexpected:
304
- raise ValueError(f"{LORA_FILE} holds {len(unexpected)} non-LoRA tensors, e.g. {unexpected[:5]}")
305
- if not bases:
306
- raise ValueError(f"No lora_A/lora_B pairs found in {LORA_FILE}")
307
-
308
- ranks = set()
309
- for name in bases:
310
- if f"{name}{suffix_b}" not in lora:
311
- raise ValueError(f"LoRA is missing the lora_B twin of {name}{suffix_a}")
312
- ranks.add(lora[f"{name}{suffix_a}"].shape[0])
313
- if len(ranks) != 1:
314
- raise ValueError(f"LoRA mixes ranks {sorted(ranks)}; a DMD2 student export uses a single rank")
315
- rank = ranks.pop()
316
 
317
- num_heads = transformer.config.num_attention_heads
318
- head_dim = transformer.config.attention_head_dim
319
- # merge_lora_ckpt.py's --scale, i.e. lora_alpha / rank; 1.0 because FlashGen's alpha == rank.
320
- scale = float(os.environ.get("H3_LORA_SCALE", "1.0"))
321
-
322
- params = dict(transformer.named_parameters())
323
- folded, missed = 0, []
324
- with torch.no_grad():
325
- for name in bases:
326
- a = lora[f"{name}{suffix_a}"]
327
- b = lora[f"{name}{suffix_b}"]
328
- for key, b_part in _lora_targets(name, b, num_heads, head_dim):
329
- param = params.get(key)
330
- if param is None:
331
- missed.append(key)
332
- continue
333
- # Accumulate in fp32 so a bf16 base does not swallow the update.
334
- delta = scale * (b_part.to(torch.float32) @ a.to(torch.float32))
335
- if delta.shape != param.shape:
336
- raise ValueError(
337
- f"LoRA delta for `{key}` has shape {tuple(delta.shape)}, "
338
- f"base weight is {tuple(param.shape)}"
339
- )
340
- param.data = (param.data.float() + delta.to(param.device)).to(param.dtype)
341
- folded += 1
342
-
343
- # merge_lora_ckpt.py treats an unmatched target as fatal, and so do we: a silently skipped
344
- # target means the name mapping is wrong and the demo would serve a half-applied adapter.
345
- if missed:
346
- raise ValueError(
347
- f"{len(missed)} LoRA targets matched no transformer weight, e.g. {missed[:5]}. "
348
- "The LoRA and the diffusers transformer disagree on module naming."
349
- )
350
-
351
- return (
352
- f"FlashGen LoRA merged · {len(bases)} targets, rank {rank}, scale {scale:g} "
353
- f"-> {folded} weight deltas · {LORA_FILE}"
354
- )
355
-
356
- # ── Additional LoRAs ─────────────────────────────────────────────────────────
357
- #
358
- # These are folded after FlashGen, so the original 4-step FlashGen schedule and
359
- # all of its QKV/layout handling remain unchanged. The extra adapters use the
360
- # standard .lora_A.weight/.lora_B.weight export used by the motion-enhancer and
361
- # additional H3 checkpoints.
362
-
363
- def _apply_extra_lora(
364
- transformer,
365
- repo_id: str,
366
- filename: str,
367
- label: str,
368
- scale: float = 1.0,
369
- ) -> str:
370
  from huggingface_hub import hf_hub_download
371
  from safetensors.torch import load_file
372
 
373
- # All adapters used by this Space are pinned to repository/file paths.
374
- # There is deliberately no Civitai/API fallback here: startup must only
375
- # resolve repository files through the Hugging Face Hub.
376
- path = hf_hub_download(repo_id, filename)
377
-
378
  lora = load_file(path, device="cpu")
379
-
380
- normalized = {}
381
- for key, tensor in lora.items():
382
- k = key
383
- if k.startswith("lora_unet__"):
384
- k = k.replace("lora_unet__", "", 1)
385
- k = k.replace("_blocks_", ".blocks.")
386
- k = k.replace("_self_attn_", ".attn.")
387
- k = k.replace("_ffn_net_0_proj", ".mlp.fc1")
388
- k = k.replace("_ffn_net_2", ".mlp.fc2")
389
- k = k.replace("to_out_0", "out_proj")
390
- elif k.startswith("lora_unet_"):
391
- body = k[len("lora_unet_"):]
392
- parts = body.split(".")
393
- head = parts[0].split("_")
394
- out = []
395
- i = 0
396
- while i < len(head):
397
- if head[i] == "blocks" and i + 1 < len(head) and head[i + 1].isdigit():
398
- out += ["blocks", head[i + 1]]
399
- i += 2
400
- continue
401
- if head[i] == "self" and i + 1 < len(head) and head[i + 1] == "attn":
402
- out.append("attn")
403
- i += 2
404
- continue
405
- out.append(head[i])
406
- i += 1
407
- k = ".".join(out + parts[1:])
408
- for prefix in ("base_model.model.", "diffusion_model.", "transformer.", "pipe.dit."):
409
- if k.startswith(prefix):
410
- k = k[len(prefix):]
411
- k = k.replace(".self_attn.", ".attn.")
412
- k = k.replace(".ffn.0.", ".mlp.fc1.")
413
- k = k.replace(".ffn.2.", ".mlp.fc2.")
414
- k = k.replace(".lora_down.weight", ".lora_A.weight")
415
- k = k.replace(".lora_up.weight", ".lora_B.weight")
416
- k = k.replace(".lora_A.default.weight", ".lora_A.weight")
417
- k = k.replace(".lora_B.default.weight", ".lora_B.weight")
418
- normalized[k] = tensor
419
- lora = normalized
420
  suffix_a = ".lora_A.weight"
421
  suffix_b = ".lora_B.weight"
422
- bases = sorted(
423
- {key[: -len(suffix_a)] for key in lora if key.endswith(suffix_a)}
424
- )
425
  if not bases:
426
- sample = list(lora.keys())[:5]
427
- raise ValueError(f"{label}: no lora_A/lora_B pairs found; keys: {sample}")
428
 
429
  params = dict(transformer.named_parameters())
430
  num_heads = transformer.config.num_attention_heads
431
  head_dim = transformer.config.attention_head_dim
432
  folded = 0
433
  missed = []
434
-
435
  with torch.no_grad():
436
  for name in bases:
437
  a = lora[f"{name}{suffix_a}"]
438
  b = lora.get(f"{name}{suffix_b}")
439
  if b is None:
440
- raise ValueError(f"{label}: missing lora_B for {name}")
441
-
442
- # These older H3 LoRAs use the original checkpoint names. Reuse
443
- # the same layout transforms as the FlashGen merge path.
444
- source_name = name.removeprefix("diffusion_model.")
445
- target_name = _lora_target_name(source_name)
446
-
447
- # ComfyUI H3 LoRAs may store q/k/v as separate targets. The
448
- # diffusers H3 transformer used by this Space has separate to_q/k/v
449
- # parameters, so those can be folded directly.
450
- if target_name.endswith((".attn.q", ".attn.k", ".attn.v")):
451
- kind = target_name.rsplit(".", 1)[-1]
452
- target_parts = [(
453
- target_name.replace(f".attn.{kind}", f".attn.to_{kind}") + ".weight",
454
- b,
455
- )]
456
- elif target_name.endswith(".attn.qkv_proj"):
457
- prefix = target_name.removesuffix("qkv_proj")
458
- rows = _reorder_interleaved_qkv(b, num_heads, head_dim)
459
- targets = zip(
460
- ("q", "k", "v"),
461
- rows.split(num_heads * head_dim, dim=0),
462
- )
463
- target_parts = [
464
- (f"{prefix}to_{kind}.weight", part.contiguous())
465
- for kind, part in targets
466
- ]
467
- elif target_name.endswith(".mlp.fc1"):
468
- gate, value = b.chunk(2, dim=0)
469
- target_parts = [(
470
- target_name.replace(".mlp.fc1", ".ff.net.0.proj") + ".weight",
471
- torch.cat([value, gate]).contiguous(),
472
- )]
473
- elif target_name.endswith(".mlp.fc2"):
474
- target_parts = [(
475
- target_name.replace(".mlp.fc2", ".ff.net.2") + ".weight",
476
- b,
477
- )]
478
- elif target_name.endswith(".attn.out_proj"):
479
- target_parts = [(
480
- target_name.replace(".attn.out_proj", ".attn.to_out.0") + ".weight",
481
- b,
482
- )]
483
- else:
484
- target_parts = [(target_name + ".weight", b)]
485
-
486
- for key, b_part in target_parts:
487
  param = params.get(key)
488
  if param is None:
489
  missed.append(key)
490
  continue
491
-
492
- delta = float(scale) * (b_part.float() @ a.float())
493
  if delta.shape != param.shape:
494
- raise ValueError(
495
- f"{label}: LoRA shape mismatch for {key}: "
496
- f"{tuple(delta.shape)} vs {tuple(param.shape)}"
497
- )
498
- param.data = (
499
- param.data.float() + delta.to(param.device)
500
- ).to(param.dtype)
501
  folded += 1
502
 
503
  if missed:
504
- raise ValueError(
505
- f"{label}: {len(missed)} LoRA targets did not match the H3 transformer, "
506
- f"e.g. {missed[:5]}"
507
- )
508
 
509
- return (
510
- f"{label} merged · {len(bases)} targets, "
511
- f"{folded} weight deltas · scale {float(scale):g}"
512
- )
513
 
514
- def load_and_apply_extra_loras(transformer) -> list[str]:
515
- return [
516
- _apply_extra_lora(
517
- transformer,
518
- MOTION_LORA_REPO,
519
- MOTION_LORA_FILE,
520
- "Motion Enhancer LoRA",
521
- MOTION_LORA_SCALE,
522
- ),
523
- _apply_extra_lora(
524
- transformer,
525
- REALISM_LORA_REPO,
526
- REALISM_LORA_FILE,
527
- "H3 Realism People LoRA",
528
- REALISM_LORA_SCALE,
529
- ),
530
- _apply_extra_lora(
531
- transformer,
532
- DETAIL_LORA_REPO,
533
- DETAIL_LORA_FILE,
534
- "Perfect Eyes / Skin / Hands LoRA",
535
- DETAIL_LORA_SCALE,
536
- ),
537
- _apply_extra_lora(
538
- transformer,
539
- SPATIAL_LORA_REPO,
540
- SPATIAL_LORA_FILE,
541
- "Spatial & Physics LoRA",
542
- SPATIAL_LORA_SCALE,
543
- ),
544
- _apply_extra_lora(
545
- transformer,
546
- BREASTS_LORA_REPO,
547
- BREASTS_LORA_FILE,
548
- "HMBreasts V2 LoRA",
549
- BREASTS_LORA_SCALE,
550
- ),
551
- ]
552
 
553
- # ── Model loading ───────────────────────────────────────────────────────────
 
 
 
 
 
 
 
 
 
 
554
 
555
- def load_models() -> str | None:
556
- """Load the denoising half at startup: transformer + VAEs + schedulers, then fold the LoRA in."""
557
- global PIPE, LOAD_ERROR, LOADED_IN
558
 
559
- if PIPE is not None or LOAD_ERROR is not None:
560
- return LOAD_ERROR
561
 
562
- started = time.time()
563
- try:
564
- import torch
565
- from diffusers import ComponentsManager
566
-
567
- from h3_split_blocks import MiniMaxH3GeneratorBlocks
568
-
569
- lower_duration_floor()
570
- manager = ComponentsManager()
571
- blocks = MiniMaxH3GeneratorBlocks()
572
- print(f"[gen] loading {[c.name for c in blocks.expected_components]} from {MODEL_REPO} ...", flush=True)
573
- pipe = blocks.init_pipeline(MODEL_REPO, components_manager=manager, collection="h3")
574
- pipe.load_components(dtype=torch.bfloat16)
575
 
576
- pipe.vae.set_attention_backend("native")
577
- pipe.audio_vae.set_attention_backend("native")
578
- pipe.transformer.set_attention_backend(ATTENTION)
579
-
580
- # Fold the original FlashGen LoRA first. Its distilled 4-step schedule
581
- # remains the schedule used by the merged transformer.
582
- lora_status = load_and_apply_lora(pipe.transformer)
583
- print(f"[gen] {lora_status}", flush=True)
584
 
585
- # Then fold the requested extra LoRAs into the same transformer.
586
- # They add their deltas on top of FlashGen.
587
- for extra_status in load_and_apply_extra_loras(pipe.transformer):
588
- print(f"[gen] {extra_status}", flush=True)
589
 
590
- # ... and pin the distilled schedule the merged weights were trained for, so the pipeline's own
591
- # `set_timesteps` call cannot fall back to a uniform grid (and to NUM_STEPS - 1 evaluations).
592
- pin_distilled_schedule(pipe.scheduler)
593
- pin_distilled_schedule(pipe.audio_scheduler)
594
- print(
595
- f"[gen] distilled base_schedule {BASE_SCHEDULE} pinned -> {NUM_STEPS} steps; "
596
- f"video σ (shift {pipe.scheduler.shift:g}) "
597
- f"{[round(s, 4) for s in shifted_schedule(pipe.scheduler.shift)]}, "
598
- f"audio σ (shift {pipe.audio_scheduler.shift:g}) "
599
- f"{[round(s, 4) for s in shifted_schedule(pipe.audio_scheduler.shift)]}",
600
- flush=True,
601
- )
602
 
603
- if PLACEMENT == "pack":
604
- pipe.transformer.to("cuda")
 
 
 
605
 
606
- PIPE = pipe
607
- LOADED_IN = time.time() - started
608
- print(f"[gen] ready in {LOADED_IN:.0f}s", flush=True)
609
- except Exception as error:
610
- traceback.print_exc()
611
- LOAD_ERROR = f"**Loading failed** after {time.time() - started:.0f}s: `{type(error).__name__}: {error}`"
612
- return LOAD_ERROR
613
 
614
- # ── Remote conditioning ──────────────────────────────────────────────────────
 
615
 
616
- @cache
617
- def conditioner():
618
- from gradio_client import Client
 
 
 
 
619
 
620
- return Client(CONDITIONER_SPACE)
621
 
622
- def encode_remote(prompt, image_path, last_image_path, canvas, num_frames, ip_token=None, rewrite_prompt=False):
623
- """Call the conditioner Space to get ``prompt_embeds`` + ``text_token_tags`` + resolved geometry.
624
 
625
- ``rewrite_prompt`` is the same prompt-enhancement path ``multimodalart/minimax-h3`` uses: the conditioner
626
- Space already hosts H3's Qwen3-VL, so it first has that model *rewrite* the prompt into MiniMax-H3's
627
- structured shot description (the LLM-based rewriting step) and then encodes the rewritten text instead of
628
- the raw one. The rewritten text comes back in the plan under ``refined_prompt`` (``None`` when no rewrite
629
- happened). Encoding the enhanced prompt on the conditioner keeps the enhancement and the embeddings in
630
- sync — nothing about the FlashGen merge, the 4-step schedule, or the VAEs is involved.
631
  """
632
  from gradio_client import handle_file
633
  from safetensors import safe_open
634
 
635
  def call():
636
- return conditioner_client(ip_token).predict(
637
  prompt=prompt,
638
- image_path=handle_file(image_path) if image_path else None,
639
- last_image_path=handle_file(last_image_path) if last_image_path else None,
640
  canvas=canvas,
641
  num_frames=num_frames,
642
  rewrite_prompt=bool(rewrite_prompt),
@@ -646,381 +179,149 @@ def encode_remote(prompt, image_path, last_image_path, canvas, num_frames, ip_to
646
  try:
647
  path, plan = call()
648
  except Exception as first:
 
 
649
  print(f"[conditioner] retrying with a fresh client after: {first}", flush=True)
650
- conditioner.cache_clear()
651
  path, plan = call()
652
 
653
  with safe_open(path, framework="pt") as handle:
 
 
654
  metadata = handle.metadata()
655
- return handle.get_tensor("prompt_embeds"), handle.get_tensor("text_token_tags"), metadata, plan
656
-
657
- def conditioner_client(ip_token):
658
- if not ip_token:
659
- return conditioner()
660
- from gradio_client import Client
661
-
662
- return Client(CONDITIONER_SPACE, headers={"x-ip-token": ip_token})
663
-
664
- # ── Geometry helpers ────────────────────────────────────────────────────────
665
-
666
- def _snap_canvas(aspect: float, current_canvas: str) -> str:
667
- """The supported canvas whose aspect ratio is closest to ``aspect``.
668
-
669
- Same selection ``multimodalart/minimax-h3`` makes: among the canvases sharing a ratio keep the
670
- smallest (fastest) one, then take the ratio nearest the image's — so a 3:4 photo renders 3:4 rather
671
- than being letterboxed into the 16:9 default. A canvas the user already picked is kept when it is at
672
- least as close to the image as the best candidate, so ``1344x768`` survives a 1.75 image.
673
- """
674
- fastest: dict[float, tuple[str, tuple[int, int]]] = {}
675
- for label, (h, w) in CANVASES.items():
676
- r = w / h
677
- if r not in fastest or w * h < fastest[r][1][0] * fastest[r][1][1]:
678
- fastest[r] = (label, (h, w))
679
- ratio = min(fastest, key=lambda r: abs(r - aspect))
680
-
681
- cur_h, cur_w = CANVASES[current_canvas]
682
- if abs(cur_w / cur_h - aspect) <= abs(ratio - aspect):
683
- return current_canvas
684
- return fastest[ratio][0]
685
-
686
- def _cover_crop(image_path: str, canvas_label: str) -> str:
687
- """Resize/crop the keyframe to fill the selected canvas, with no blurred background."""
688
- from PIL import Image as _Image, ImageOps as _ImageOps
689
-
690
- h, w = CANVASES[canvas_label]
691
- with _Image.open(image_path) as source:
692
- img = _ImageOps.exif_transpose(source).convert("RGB")
693
- if img.size == (w, h):
694
- return image_path
695
-
696
- # Fill the complete canvas; if aspect ratios differ, crop excess edges.
697
- fitted = _ImageOps.fit(img, (w, h), method=_Image.Resampling.LANCZOS)
698
- fitted.save(image_path)
699
- return image_path
700
-
701
- def _fit_keyframe(image_path, current_canvas):
702
- """Fit a keyframe to the selected canvas ratio without changing the selected resolution."""
703
- return _cover_crop(image_path, current_canvas), current_canvas
704
-
705
- def _fit_keyframe_ui(image_path, current_canvas):
706
- """Crop the uploaded keyframe to the selected canvas ratio; preserve the Canvas dropdown."""
707
- path = _as_path(image_path)
708
- if not path:
709
- return gr.update(), gr.update()
710
- cropped, label = _fit_keyframe(path, current_canvas)
711
- return gr.update(value=cropped), gr.update(value=label)
712
-
713
- def _fit_last_keyframe_ui(last_image_path, image_path, current_canvas):
714
- """``last_image.upload`` handler. The first frame owns the canvas when there is one, so a last
715
- frame uploaded on top of it is only cover-cropped to the canvas already chosen."""
716
- path = _as_path(last_image_path)
717
- if not path:
718
- return gr.update(), gr.update()
719
- if _as_path(image_path):
720
- return gr.update(value=_cover_crop(path, current_canvas)), gr.update()
721
- return _fit_keyframe_ui(path, current_canvas)
722
 
723
- def _as_path(value):
724
- if isinstance(value, dict):
725
- value = value.get("path") or (value.get("url") or "").removeprefix("/gradio_api/file=")
726
- return value or None
727
 
728
- # ── GPU duration estimation ─────────────────────────────────────────────────
729
-
730
- _DUR_B, _DUR_C = 1.1745e-4, 3.8396e-9
731
- _DECODE_BASE, _DECODE_PER_DEFAULT_CANVAS, _DEFAULT_CANVAS_PIXELS = 15, 15, 960 * 544 * 124
732
- _PLACEMENT_ALLOWANCE, _PAD = 12, 10
733
-
734
- def get_duration(prompt_embeds, text_token_tags, image, last_image, height, width, num_frames, seed, *a, **k):
735
- """Estimate GPU duration for the ``_generate`` call. Steps is fixed at NUM_STEPS (4)."""
736
- height, width, num_frames = int(height), int(width), int(num_frames)
737
- steps = NUM_STEPS
738
- latent_frames = (num_frames - LATENTS_PER_CHUNK) // FRAMES_PER_CHUNK * LATENTS_PER_CHUNK + 2
739
- patches = (height // 32) * (width // 32)
740
- rows = latent_frames * patches + (int(image is not None) + int(last_image is not None)) * patches
741
- denoise = steps * (_DUR_B * rows + _DUR_C * rows**2)
742
- decode = _DECODE_BASE + _DECODE_PER_DEFAULT_CANVAS * (height * width * num_frames) / _DEFAULT_CANVAS_PIXELS
743
- return max(60, int(denoise + decode) + _PLACEMENT_ALLOWANCE + _PAD)
744
-
745
- # ── Inference ───────────────────────────────────────────────────────────────
746
-
747
- @spaces.GPU(duration=get_duration, size=GPU_SIZE)
748
- def _generate(prompt_embeds, text_token_tags, image, last_image, height, width, num_frames, seed):
749
- """Denoise + decode on GPU time. The LoRA is already folded at startup; 4 steps with custom sigmas."""
750
- import torch
751
-
752
- if PLACEMENT == "pack":
753
- PIPE.vae.to("cuda")
754
- PIPE.audio_vae.to("cuda")
755
- elif PLACEMENT == "lazy":
756
- PIPE.to("cuda")
757
-
758
- # Inject the distilled base_schedule into both schedulers, and keep it there: the pipeline calls
759
- # `set_timesteps` again inside its denoise step, so the schedule has to survive that call.
760
- pin_distilled_schedule(PIPE.scheduler)
761
- pin_distilled_schedule(PIPE.audio_scheduler)
762
- PIPE.scheduler.set_timesteps(device="cuda")
763
- PIPE.audio_scheduler.set_timesteps(device="cuda")
764
-
765
- state = PIPE(
766
- prompt_embeds=prompt_embeds.to("cuda"),
767
- text_token_tags=text_token_tags,
768
  image=image,
769
- last_image=last_image,
770
- height=height,
771
- width=width,
772
- num_frames=num_frames,
773
- # Grid points, terminal 0.0 included -> NUM_STEPS Euler steps. See NUM_SIGMA_POINTS above.
774
- num_inference_steps=NUM_SIGMA_POINTS,
775
  generator=torch.Generator("cpu").manual_seed(int(seed)),
776
  )
777
- # Read the step count back off the scheduler rather than trusting the constant: `step_index` counted
778
- # the Euler updates the denoise loop actually took, and `sigmas` is the grid it took them on.
779
- steps_taken = int(PIPE.scheduler.step_index or 0)
780
- sigmas_used = [round(float(s), 4) for s in PIPE.scheduler.sigmas.tolist()]
781
- return (
782
- state.get("videos")[0],
783
- state.get("audio")[0].cpu(),
784
- state.get("sampling_rate"),
785
- steps_taken,
786
- sigmas_used,
787
- )
788
-
789
- def _caller_ip_token() -> str | None:
790
- from gradio.context import LocalContext
791
 
792
- request = LocalContext.request.get()
793
- return request.headers.get("x-ip-token") if request is not None else None
794
 
795
  def generate(
796
- prompt,
797
- image_path=None,
798
- last_image_path=None,
799
- canvas=DEFAULT_CANVAS,
800
- duration=5,
801
- seed=42,
802
- enhance_prompt=False,
803
  progress=gr.Progress(track_tqdm=True),
804
  ):
805
- """One request: prompt (+ optional first/last frame) -> video with synchronized audio.
806
-
807
- ``enhance_prompt`` turns on the conditioner's LLM prompt rewriting (the same enhancement
808
- ``multimodalart/minimax-h3`` offers); off, the raw prompt is encoded verbatim as before. It is last and
809
- defaults to off, so a positional API client written against the previous signature is unaffected.
810
- """
811
- if LOAD_ERROR:
812
- raise gr.Error(LOAD_ERROR.replace("**", "").replace("`", ""))
813
- if PIPE is None:
814
- raise gr.Error("The model is still loading — watch the Space logs and retry shortly.")
815
  if not prompt or not prompt.strip():
816
- raise gr.Error("MiniMax-H3 always takes a prompt, keyframes or not.")
817
-
818
- from PIL import Image, ImageOps
819
 
 
 
 
 
820
  from diffusers.utils import encode_video
821
 
822
- # The user's Canvas selection is authoritative. Crop keyframes to its aspect ratio,
823
- # but never replace the selected resolution with a preset chosen from the source image.
824
- first, last = _as_path(image_path), _as_path(last_image_path)
825
- if first:
826
- first = _cover_crop(first, canvas)
827
- if last:
828
- last = _cover_crop(last, canvas)
829
-
830
- num_frames = snap_frames(duration)
831
-
832
- progress(0.0, desc="Enhancing the prompt, then encoding ..." if enhance_prompt else "Encoding prompt ...")
833
- conditioned = time.time()
834
- condition_prompt = prompt
835
- # Realism People was trained with the `r34l1sm` trigger. Keep it hidden
836
- # from the UI while ensuring the adapter receives its intended trigger.
837
- if REALISM_LORA_TRIGGER.lower() not in condition_prompt.lower():
838
- condition_prompt = f"{REALISM_LORA_TRIGGER}, {condition_prompt}"
839
-
840
- prompt_embeds, text_token_tags, metadata, plan = encode_remote(
841
- condition_prompt,
842
- first,
843
- last,
844
- canvas,
845
- num_frames,
846
- ip_token=_caller_ip_token(),
847
- rewrite_prompt=bool(enhance_prompt),
848
  )
849
- condition_seconds = time.time() - conditioned
850
- # Keep the requested Canvas dimensions even if the remote conditioner reports a fallback size.
851
- height, width = CANVASES[canvas]
852
- num_frames = int(metadata["num_frames"])
853
- # `None` whenever no rewrite happened — either none was asked for, or the conditioner declined one.
 
 
854
  enhanced = (plan or {}).get("refined_prompt") or ""
 
 
 
 
 
855
 
856
- def keyframe(path):
857
- return ImageOps.exif_transpose(Image.open(path)).convert("RGB") if path else None
858
 
859
- progress(0.1, desc=f"Generating {num_frames / FPS:.1f}s at {width}x{height} in {NUM_STEPS} steps ...")
860
- started = time.time()
861
- frames, audio, sampling_rate, steps_taken, sigmas_used = _generate(
862
- prompt_embeds,
863
- text_token_tags,
864
- keyframe(first),
865
- keyframe(last),
866
- height,
867
- width,
868
- num_frames,
869
- seed,
870
  )
871
- generate_seconds = time.time() - started
872
 
873
- path = os.path.join(OUTPUT_DIR, f"h3-flashgen-{int(time.time() * 1000)}.mp4")
874
- encode_video(frames, fps=FPS, output_path=path, audio=audio, audio_sample_rate=sampling_rate)
875
 
876
- # Verify the encoded MP4 dimensions. If the pipeline/encoder emitted another size, normalize
877
- # the final video to the selected Canvas while preserving its audio track.
878
- import json
879
- import shutil
880
- import subprocess
881
-
882
- ffprobe = shutil.which("ffprobe")
883
- ffmpeg = shutil.which("ffmpeg")
884
- actual_width, actual_height = width, height
885
- if ffprobe:
886
- probe = subprocess.run(
887
- [ffprobe, "-v", "error", "-select_streams", "v:0",
888
- "-show_entries", "stream=width,height", "-of", "json", path],
889
- capture_output=True, text=True, check=True,
890
- )
891
- streams = json.loads(probe.stdout).get("streams", [])
892
- if streams:
893
- actual_width = int(streams[0]["width"])
894
- actual_height = int(streams[0]["height"])
895
-
896
- resized = False
897
- if (actual_width, actual_height) != (width, height):
898
- if not ffmpeg:
899
- raise gr.Error(
900
- f"Выбрано {width}x{height}, но модель создала {actual_width}x{actual_height}; "
901
- "FFmpeg недоступен для исправления размера."
902
- )
903
- corrected = path[:-4] + "-canvas.mp4"
904
- subprocess.run(
905
- [ffmpeg, "-y", "-i", path,
906
- "-vf",
907
- f"scale={width}:{height}:force_original_aspect_ratio=increase:flags=lanczos,"
908
- f"crop={width}:{height},setsar=1",
909
- "-map", "0:v:0", "-map", "0:a?",
910
- "-c:v", "libx264", "-preset", "veryfast", "-crf", "18",
911
- "-c:a", "aac", "-b:a", "192k", "-movflags", "+faststart", corrected],
912
- capture_output=True, text=True, check=True,
913
- )
914
- os.replace(corrected, path)
915
- resized = True
916
- print(
917
- f"[canvas] Model/encoder output {actual_width}x{actual_height}; "
918
- f"final MP4 filled to selected Canvas (excess edges cropped) {width}x{height}.",
919
- flush=True,
920
- )
921
 
922
- # Reported, not assumed: `steps_taken` is the number of Euler updates the scheduler actually took.
923
- schedule = " → ".join(f"{s:g}" for s in sigmas_used)
924
- report = (
925
- f"{width}x{height}{' (final size corrected with FFmpeg)' if resized else ''} · "
926
- f"{num_frames} frames ({num_frames / FPS:.3f}s) · "
927
- f"{steps_taken} steps (FlashGen σ {schedule}) · "
928
- f"conditioner {condition_seconds:.0f}s{' (prompt enhanced)' if enhanced else ''} · "
929
- f"denoise + decode {generate_seconds:.0f}s · seed {int(seed)}"
930
- )
931
- print(f"[gen] {report}", flush=True)
932
- expected_sigmas = [round(s, 4) for s in shifted_schedule(PIPE.scheduler.shift)]
933
- if steps_taken != NUM_STEPS or sigmas_used != expected_sigmas:
934
- print(
935
- f"[gen] WARNING: expected {NUM_STEPS} steps on {expected_sigmas} "
936
- f"(base_schedule {BASE_SCHEDULE} at shift {PIPE.scheduler.shift:g}), "
937
- f"ran {steps_taken} on {sigmas_used}",
938
- flush=True,
939
- )
940
- return path, report, enhanced, gr.update(visible=bool(enhanced))
941
 
942
- # ── UI ──────────────────────────────────────────────────────────────────────
 
 
943
 
944
- CSS = """
945
- .main.fillable { max-width: 1100px !important; }
946
  """
947
 
948
- EXAMPLE_PROMPTS = [
949
- "Cinematic, medium wide shot. A golden retriever puppy chases autumn leaves through a sunlit park, leaves swirling around it. Warm afternoon light, shallow depth of field. The dog barks playfully and leaves rustle.",
950
- "A serene Japanese garden with a koi pond. Cherry blossom petals drift slowly on the water's surface. A small wooden bridge arcs over the pond. Gentle wind chimes tinkle in the breeze. Ambient birdsong and water sounds.",
951
- "Close-up of a chef's hands kneading dough on a flour-dusted wooden board. Flour particles float in the warm kitchen light. The dough stretches and folds rhythmically. Soft thumping sounds of dough and ambient kitchen noise.",
952
- "A neon-lit cyberpunk street in the rain. Reflected neon signs shimmer in puddles. A figure in a long coat walks away from camera. Rain patters on metal awnings, distant synth music echoes.",
953
- "A majestic eagle soars over snow-capped mountains at golden hour. The camera follows from behind, sweeping past peaks. Wind rushes past. Distant orchestral music swells.",
954
- ]
955
 
956
- with gr.Blocks(title="MiniMax-H3 4-Step FlashGen") as demo:
 
957
  with gr.Row(equal_height=True):
958
- with gr.Column():
 
959
  prompt = gr.Textbox(
960
- label="Prompt",
961
- lines=4,
962
- value=EXAMPLE_PROMPTS[0],
963
- placeholder="Describe the scene, shot style, and soundscape ...",
964
  )
965
  enhance = gr.Checkbox(
966
- label="Enhance prompt",
967
  value=False,
 
968
  )
969
- with gr.Accordion("Image-to-video (optional)", open=True):
970
- with gr.Row():
971
- image = gr.Image(label="First frame (optional)", type="filepath", height=200)
972
- last_image = gr.Image(label="Last frame (optional)", type="filepath", height=200)
973
-
974
- run = gr.Button("Generate", variant="primary", size="lg")
975
-
976
- with gr.Accordion("Advanced", open=False):
977
- canvas = gr.Dropdown(label="Canvas", choices=list(CANVASES), value=DEFAULT_CANVAS)
978
- duration = gr.Slider(
979
- label="Duration (s)",
980
- minimum=MIN_UI_DURATION,
981
- maximum=MAX_UI_DURATION,
982
- step=1,
983
- value=5,
984
- )
985
- seed = gr.Number(label="Seed", value=42, precision=0)
986
-
987
- with gr.Column():
988
- result = gr.Video(label="Video + soundtrack", autoplay=True)
989
- report = gr.Markdown()
990
- # An output, so it is revealed only for a request that actually asked for an enhancement.
991
- with gr.Accordion("Enhanced prompt", open=False, visible=False) as enhanced_panel:
992
- enhanced_box = gr.Textbox(show_label=False, lines=8, interactive=False)
993
 
994
- OUTPUTS = [result, report, enhanced_box, enhanced_panel]
 
 
 
 
995
 
996
- # An uploaded I2V frame decides the output geometry: snap Canvas to the nearest supported ratio and
997
- # cover-crop the frame to it, right away, so what is shown is what will be rendered.
998
- image.upload(_fit_keyframe_ui, [image, canvas], [image, canvas])
999
- last_image.upload(_fit_last_keyframe_ui, [last_image, image, canvas], [last_image, canvas])
1000
 
1001
- gr.Examples(
1002
- examples=[[p] for p in EXAMPLE_PROMPTS],
1003
- inputs=[prompt],
1004
  fn=generate,
1005
- outputs=OUTPUTS,
1006
- cache_examples=True,
1007
- cache_mode="lazy",
1008
- )
1009
-
1010
- run.click(
1011
- generate,
1012
- inputs=[prompt, image, last_image, canvas, duration, seed, enhance],
1013
- outputs=OUTPUTS,
1014
  api_name="generate",
1015
  )
 
1016
  prompt.submit(
1017
- generate,
1018
- inputs=[prompt, image, last_image, canvas, duration, seed, enhance],
1019
- outputs=OUTPUTS,
1020
  api_name="generate_submit",
1021
  )
1022
 
1023
- load_models()
1024
 
1025
- if __name__ == "__main__":
1026
- demo.launch(show_error=True, theme=gr.themes.Citrus(), css=CSS, allowed_paths=[OUTPUT_DIR])
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  from __future__ import annotations
2
 
3
  import os
4
  import tempfile
5
  import time
6
+ from functools import lru_cache
 
7
 
8
  import spaces
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9
 
10
+ os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
 
 
 
 
 
 
 
 
 
 
 
11
 
12
+ import gradio as gr
13
+ import torch
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
 
15
+ from h3_split_blocks import MiniMaxH3GeneratorBlocks
 
 
 
 
16
 
 
 
 
 
 
 
 
 
 
 
17
 
18
+ MODEL_REPO = "MiniMaxAI/MiniMax-H3"
19
+ LORA_REPO = "rzgar/minimax-h3_fl2v_8Step_motion_enhancer"
20
+ LORA_FILE = "minimax-h3_fl2v_8Step_motion_enhancer.safetensors"
21
+ CONDITIONER_SPACE = "multimodalart/qwen3vl-conditioner"
22
+ FPS = 24
23
+ STEPS = 8
24
+ OUTPUT_DIR = os.path.join(tempfile.gettempdir(), "minimax-h3-motion-enhancer")
25
+ os.makedirs(OUTPUT_DIR, exist_ok=True)
26
 
27
  CANVASES = {
 
28
  "960x544 · 16:9 fast": (544, 960),
29
  "1024x576 · 16:9 fast": (576, 1024),
 
 
 
 
30
  "544x960 · 9:16 fast": (960, 544),
 
 
 
31
  "544x544 · 1:1 fast": (544, 544),
 
 
32
  "768x576 · 4:3 fast": (576, 768),
 
33
  "576x768 · 3:4 fast": (768, 576),
 
 
34
  "1152x512 · 21:9 fast": (512, 1152),
 
35
  }
36
  DEFAULT_CANVAS = "960x544 · 16:9 fast"
37
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
 
39
+ def _reorder_interleaved_qkv(weight: torch.Tensor, num_heads: int, head_dim: int) -> torch.Tensor:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
  grouped = weight.reshape(num_heads, 3 * head_dim, *weight.shape[1:])
41
  query, key, value = grouped.split(head_dim, dim=1)
42
  return torch.cat(
43
  [part.reshape(num_heads * head_dim, *weight.shape[1:]) for part in (query, key, value)], dim=0
44
  )
45
 
46
+
47
  def _lora_target_name(source_name: str) -> str:
48
+ target = source_name.removeprefix("diffusion_model.")
49
+ if target.startswith("token_refiner.blocks."):
50
+ target = target.replace("token_refiner.blocks.", "token_refiner.refiner_blocks.", 1)
51
+ elif target.startswith("blocks."):
52
+ target = target.replace("blocks.", "transformer_blocks.", 1)
 
 
53
  return target.replace("final_layer.adaln_proj.linear", "norm_out.linear")
54
 
 
 
 
55
 
56
+ def _lora_targets(name: str, b_weight: torch.Tensor, num_heads: int, head_dim: int):
57
+ target = _lora_target_name(name)
58
  if target.endswith(".attn.qkv_proj"):
59
  prefix = target.removesuffix("qkv_proj")
 
 
60
  rows = _reorder_interleaved_qkv(b_weight, num_heads, head_dim)
61
  for kind, part in zip(("q", "k", "v"), rows.split(num_heads * head_dim, dim=0)):
62
  yield f"{prefix}to_{kind}.weight", part.contiguous()
63
  elif target.endswith(".mlp.fc1"):
 
64
  gate, value = b_weight.chunk(2, dim=0)
65
  yield target.replace(".mlp.fc1", ".ff.net.0.proj") + ".weight", torch.cat([value, gate]).contiguous()
66
  elif target.endswith(".mlp.fc2"):
 
68
  elif target.endswith(".attn.out_proj"):
69
  yield target.replace(".attn.out_proj", ".attn.to_out.0") + ".weight", b_weight
70
  else:
 
71
  yield target + ".weight", b_weight
72
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
73
 
74
+ def _apply_motion_lora(transformer) -> str:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
75
  from huggingface_hub import hf_hub_download
76
  from safetensors.torch import load_file
77
 
78
+ path = hf_hub_download(LORA_REPO, LORA_FILE)
 
 
 
 
79
  lora = load_file(path, device="cpu")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
  suffix_a = ".lora_A.weight"
81
  suffix_b = ".lora_B.weight"
82
+ bases = sorted({key[: -len(suffix_a)] for key in lora if key.endswith(suffix_a)})
 
 
83
  if not bases:
84
+ raise ValueError("The motion enhancer LoRA did not contain any lora_A/lora_B pairs")
 
85
 
86
  params = dict(transformer.named_parameters())
87
  num_heads = transformer.config.num_attention_heads
88
  head_dim = transformer.config.attention_head_dim
89
  folded = 0
90
  missed = []
 
91
  with torch.no_grad():
92
  for name in bases:
93
  a = lora[f"{name}{suffix_a}"]
94
  b = lora.get(f"{name}{suffix_b}")
95
  if b is None:
96
+ raise ValueError(f"Missing LoRA B matrix for {name}")
97
+ for key, b_part in _lora_targets(name, b, num_heads, head_dim):
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
98
  param = params.get(key)
99
  if param is None:
100
  missed.append(key)
101
  continue
102
+ delta = b_part.float() @ a.float()
 
103
  if delta.shape != param.shape:
104
+ raise ValueError(f"LoRA shape mismatch for {key}: {delta.shape} vs {param.shape}")
105
+ param.data = (param.data.float() + delta.to(param.device)).to(param.dtype)
 
 
 
 
 
106
  folded += 1
107
 
108
  if missed:
109
+ raise ValueError(f"{len(missed)} LoRA targets did not match the H3 transformer, e.g. {missed[:3]}")
110
+ return f"Motion enhancer loaded: {len(bases)} adapters, {folded} matrices"
 
 
111
 
 
 
 
 
112
 
113
+ def _load_pipeline():
114
+ from diffusers import ComponentsManager
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
115
 
116
+ manager = ComponentsManager()
117
+ blocks = MiniMaxH3GeneratorBlocks()
118
+ pipe = blocks.init_pipeline(MODEL_REPO, components_manager=manager, collection="h3")
119
+ pipe.load_components(dtype=torch.bfloat16)
120
+ pipe.vae.set_attention_backend("native")
121
+ pipe.audio_vae.set_attention_backend("native")
122
+ pipe.transformer.set_attention_backend(os.environ.get("H3_ATTENTION", "_native_cudnn"))
123
+ print(_apply_motion_lora(pipe.transformer), flush=True)
124
+ # Module-scope placement lets ZeroGPU pack the weights once instead of per request.
125
+ pipe.transformer.to("cuda")
126
+ return pipe
127
 
 
 
 
128
 
129
+ PIPE = _load_pipeline()
 
130
 
 
 
 
 
 
 
 
 
 
 
 
 
 
131
 
132
+ @lru_cache(maxsize=1)
133
+ def _conditioner_client():
134
+ from gradio_client import Client
 
 
 
 
 
135
 
136
+ return Client(CONDITIONER_SPACE)
 
 
 
137
 
 
 
 
 
 
 
 
 
 
 
 
 
138
 
139
+ def _snap_frames(seconds: float) -> int:
140
+ frames = max(120, round(float(seconds) * FPS))
141
+ while frames % 17 != 5:
142
+ frames += 1
143
+ return frames
144
 
 
 
 
 
 
 
 
145
 
146
+ def _cover_crop(image_path: str, canvas: str) -> str:
147
+ from PIL import Image, ImageOps
148
 
149
+ height, width = CANVASES[canvas]
150
+ with Image.open(image_path) as image:
151
+ image = ImageOps.exif_transpose(image).convert("RGB")
152
+ image = ImageOps.fit(image, (width, height), method=Image.Resampling.LANCZOS)
153
+ output = os.path.join(OUTPUT_DIR, f"frame-{time.time_ns()}.png")
154
+ image.save(output)
155
+ return output
156
 
 
157
 
158
+ def _condition(prompt: str, image_path: str, canvas: str, num_frames: int, rewrite_prompt: bool = False):
159
+ """Encode the prompt through the same Qwen3-VL conditioner path used by FlashGen.
160
 
161
+ When rewrite_prompt=True, the conditioner first rewrites the user's prompt into
162
+ MiniMax-H3's structured shot description, then encodes that rewritten prompt.
163
+ This is the "Upsample/Enhance prompt" behavior from the FlashGen Space.
 
 
 
164
  """
165
  from gradio_client import handle_file
166
  from safetensors import safe_open
167
 
168
  def call():
169
+ return _conditioner_client().predict(
170
  prompt=prompt,
171
+ image_path=handle_file(image_path),
172
+ last_image_path=None,
173
  canvas=canvas,
174
  num_frames=num_frames,
175
  rewrite_prompt=bool(rewrite_prompt),
 
179
  try:
180
  path, plan = call()
181
  except Exception as first:
182
+ # The conditioner can occasionally leave a stale Gradio client behind.
183
+ # Retry once with a fresh client, matching the FlashGen implementation.
184
  print(f"[conditioner] retrying with a fresh client after: {first}", flush=True)
185
+ _conditioner_client.cache_clear()
186
  path, plan = call()
187
 
188
  with safe_open(path, framework="pt") as handle:
189
+ embeds = handle.get_tensor("prompt_embeds")
190
+ tags = handle.get_tensor("text_token_tags")
191
  metadata = handle.metadata()
192
+ return embeds, tags, metadata, plan
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
193
 
 
 
 
 
194
 
195
+ @spaces.GPU(duration=120, size="xlarge")
196
+ def _generate_gpu(embeds, tags, image, height: int, width: int, num_frames: int, seed: int):
197
+ """Run MiniMax-H3 denoising and return CPU video frames plus soundtrack."""
198
+ PIPE.vae.to("cuda")
199
+ PIPE.audio_vae.to("cuda")
200
+ pipe_state = PIPE(
201
+ prompt_embeds=embeds.to("cuda"),
202
+ text_token_tags=tags,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
203
  image=image,
204
+ height=int(height),
205
+ width=int(width),
206
+ num_frames=int(num_frames),
207
+ num_inference_steps=STEPS + 1,
 
 
208
  generator=torch.Generator("cpu").manual_seed(int(seed)),
209
  )
210
+ frames = pipe_state.get("videos")[0]
211
+ audio = pipe_state.get("audio")[0]
212
+ if torch.is_tensor(frames):
213
+ frames = frames.cpu()
214
+ if torch.is_tensor(audio):
215
+ audio = audio.cpu()
216
+ return frames, audio, int(pipe_state.get("sampling_rate"))
 
 
 
 
 
 
 
217
 
 
 
218
 
219
  def generate(
220
+ prompt: str,
221
+ image_path: str | None,
222
+ canvas: str,
223
+ duration: float,
224
+ seed: int,
225
+ enhance_prompt: bool = False,
 
226
  progress=gr.Progress(track_tqdm=True),
227
  ):
228
+ """Animate an uploaded image, optionally using H3's Qwen3-VL prompt enhancement."""
229
+ if not image_path:
230
+ raise gr.Error("Upload a first frame before generating.")
 
 
 
 
 
 
 
231
  if not prompt or not prompt.strip():
232
+ raise gr.Error("Describe the motion you want to see.")
 
 
233
 
234
+ height, width = CANVASES[canvas]
235
+ num_frames = _snap_frames(duration)
236
+ cropped_path = _cover_crop(image_path, canvas)
237
+ from PIL import Image
238
  from diffusers.utils import encode_video
239
 
240
+ progress(
241
+ 0.05,
242
+ desc="Enhancing prompt, then encoding..." if enhance_prompt else "Encoding the first frame...",
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
243
  )
244
+ embeds, tags, metadata, plan = _condition(
245
+ prompt.strip(), cropped_path, canvas, num_frames, rewrite_prompt=bool(enhance_prompt)
246
+ )
247
+ height = int(metadata.get("height", height))
248
+ width = int(metadata.get("width", width))
249
+ num_frames = int(metadata.get("num_frames", num_frames))
250
+
251
  enhanced = (plan or {}).get("refined_prompt") or ""
252
+ progress(0.15, desc=f"Generating {num_frames / FPS:.1f}s at {width}x{height} in {STEPS} steps")
253
+ with Image.open(cropped_path) as image:
254
+ frames, audio, sampling_rate = _generate_gpu(
255
+ embeds, tags, image.convert("RGB"), height, width, num_frames, int(seed)
256
+ )
257
 
258
+ output = os.path.join(OUTPUT_DIR, f"motion-enhanced-{time.time_ns()}.mp4")
259
+ encode_video(frames, fps=FPS, output_path=output, audio=audio, audio_sample_rate=sampling_rate)
260
 
261
+ status = (
262
+ f"{width}x{height} · {num_frames / FPS:.2f}s · {STEPS} denoising steps · "
263
+ f"seed {int(seed)}"
 
 
 
 
 
 
 
 
264
  )
265
+ return output, status, enhanced, gr.update(visible=bool(enhanced))
266
 
 
 
267
 
268
+ DESCRIPTION = """
269
+ # MiniMax-H3 Motion Enhancer
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
270
 
271
+ Upload a still image and describe the movement. The Space uses the **8-step Motion Enhancer LoRA**
272
+ with MiniMax-H3 and synchronized audio.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
273
 
274
+ **Enhance / Upsample prompt** uses the same Qwen3-VL prompt-rewriting path as the MiniMax-H3 FlashGen
275
+ Space: your short instruction is first expanded into a structured shot/action/sound description and
276
+ then those rewritten embeddings are sent to H3.
277
 
278
+ Use only images you own or have permission to animate.
 
279
  """
280
 
 
 
 
 
 
 
 
281
 
282
+ with gr.Blocks(title="MiniMax-H3 Motion Enhancer") as demo:
283
+ gr.Markdown(DESCRIPTION)
284
  with gr.Row(equal_height=True):
285
+ with gr.Column(scale=1):
286
+ image = gr.Image(label="First frame", type="filepath", sources=["upload", "webcam"])
287
  prompt = gr.Textbox(
288
+ label="Motion prompt",
289
+ value="The person naturally performs the described action. Keep the same identity and appearance. Natural body motion, realistic facial expression, synchronized sound effects and speech if specified.",
290
+ lines=5,
291
+ placeholder="Describe the action, camera movement and sound...",
292
  )
293
  enhance = gr.Checkbox(
294
+ label="Enhance / Upsample prompt",
295
  value=False,
296
+ info="Rewrite the prompt with H3's Qwen3-VL into a detailed structured shot description before generation.",
297
  )
298
+ with gr.Row():
299
+ canvas = gr.Dropdown(list(CANVASES), value=DEFAULT_CANVAS, label="Output canvas")
300
+ duration = gr.Slider(5, 15, value=5, step=1, label="Duration (seconds)")
301
+ seed = gr.Number(value=42, precision=0, label="Seed")
302
+ generate_button = gr.Button("Generate motion", variant="primary", size="lg")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
303
 
304
+ with gr.Column(scale=1):
305
+ video = gr.Video(label="Generated video + soundtrack", autoplay=True)
306
+ status = gr.Markdown()
307
+ with gr.Accordion("Upsampled prompt", open=False, visible=False) as enhanced_panel:
308
+ enhanced_box = gr.Textbox(show_label=False, lines=10, interactive=False)
309
 
310
+ outputs = [video, status, enhanced_box, enhanced_panel]
 
 
 
311
 
312
+ generate_button.click(
 
 
313
  fn=generate,
314
+ inputs=[prompt, image, canvas, duration, seed, enhance],
315
+ outputs=outputs,
 
 
 
 
 
 
 
316
  api_name="generate",
317
  )
318
+
319
  prompt.submit(
320
+ fn=generate,
321
+ inputs=[prompt, image, canvas, duration, seed, enhance],
322
+ outputs=outputs,
323
  api_name="generate_submit",
324
  )
325
 
 
326
 
327
+ demo.queue(max_size=8).launch(mcp_server=True)
 
h3_split_blocks.py CHANGED
@@ -1,15 +1,7 @@
1
- """The halves of a **split** MiniMax-H3 deployment, for both of its checkpoint partitions.
2
 
3
- MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at 150 GB of storage, so `MiniMaxH3Blocks` is cut
4
- at its `text_encoder` step: the 62.14 GiB Qwen3-VL runs in the conditioner Space, everything else in a generator
5
- Space, and `prompt_embeds` + `text_token_tags` is the whole wire format between them.
6
-
7
- `resize` / `setup` run on **both** sides: they own no pretrained component, and each half needs the canvas and the
8
- prepared keyframes or normalized references. Both conditioner halves also return the resolved `height` / `width` /
9
- `num_frames`, which the generating half pins rather than re-deriving.
10
-
11
- Two things the blocks leave to the caller: a keyframe reaches them EXIF-transposed and in RGB, and the `t2va` / `fl2va`
12
- frame count is aligned to `17 * n + 5` before the call, since that arithmetic lives on the denoising side of the cut.
13
  """
14
 
15
  from diffusers.modular_pipelines.minimax_h3.before_encoder import MiniMaxH3Ref2VASetupStep
@@ -32,43 +24,26 @@ from diffusers.modular_pipelines.modular_pipeline_utils import OutputParam
32
 
33
 
34
  def _wire_outputs(num_frames: bool = True) -> list[OutputParam]:
35
- """The wire format of the split. `num_frames` is declared by the `ref2va` half alone, whose setup resolves one."""
36
  return [
37
  OutputParam.template("prompt_embeds"),
38
- OutputParam("text_token_tags", description="The per-row modality tag of every row of `prompt_embeds`."),
39
- OutputParam("height", type_hint=int, description="Resolved height of the generated video in pixels."),
40
- OutputParam("width", type_hint=int, description="Resolved width of the generated video in pixels."),
41
- *(
42
- [OutputParam("num_frames", type_hint=int, description="Resolved number of frames, of the form 17 * n + 5.")]
43
- if num_frames
44
- else []
45
- ),
46
  ]
47
 
48
 
49
  class MiniMaxH3ConditionerBlocks(SequentialPipelineBlocks):
50
- """The conditioner half of a split MiniMax-H3: the keyframes on the canvas plus the Qwen3-VL read at layer 50."""
51
-
52
  model_name = "minimax-h3"
53
  block_classes = [MiniMaxH3AutoResizeStep, MiniMaxH3TextEncoderStep]
54
  block_names = ["resize", "text_encoder"]
55
 
56
- @property
57
- def description(self):
58
- return (
59
- "The conditioner half of a split MiniMax-H3 deployment: puts the keyframes onto the target canvas and "
60
- "encodes MiniMax-H3's presentation of the request into the `prompt_embeds` / `text_token_tags` pair the "
61
- "denoising half consumes. The frame count is the caller's to align."
62
- )
63
-
64
  @property
65
  def outputs(self):
66
  return _wire_outputs(num_frames=False)
67
 
68
 
69
  class MiniMaxH3GeneratorBlocks(SequentialPipelineBlocks):
70
- """The denoising half of a split MiniMax-H3: `MiniMaxH3Blocks` with its `text_encoder` step removed."""
71
-
72
  model_name = "minimax-h3"
73
  block_classes = [
74
  MiniMaxH3AutoResizeStep,
@@ -82,9 +57,8 @@ class MiniMaxH3GeneratorBlocks(SequentialPipelineBlocks):
82
  @property
83
  def description(self):
84
  return (
85
- "The denoising half of a split MiniMax-H3 deployment: the `t2va` / `fl2va` branch of `MiniMaxH3Blocks` "
86
- "without its text-encoder step, so `prompt_embeds` and `text_token_tags` come in as inputs and the "
87
- "62.14 GiB Qwen3-VL conditioner is never loaded here."
88
  )
89
 
90
  @property
@@ -93,37 +67,16 @@ class MiniMaxH3GeneratorBlocks(SequentialPipelineBlocks):
93
 
94
 
95
  class MiniMaxH3Ref2VAConditionerBlocks(SequentialPipelineBlocks):
96
- """The conditioner half of a split `ref2va`: the resolved plan plus the Qwen3-VL read at its 50th layer.
97
-
98
- Component for component this is `MiniMaxH3ConditionerBlocks`, so one conditioner Space serves both partitions.
99
- What differs is the presentation: `ref2va` prepends a label per reference and a vision block per image and per
100
- merged video frame pair, so the references themselves have to reach this half.
101
- """
102
-
103
  model_name = "minimax-h3"
104
  block_classes = [MiniMaxH3Ref2VASetupStep, MiniMaxH3Ref2VATextEncoderStep]
105
  block_names = ["setup", "text_encoder"]
106
 
107
- @property
108
- def description(self):
109
- return (
110
- "The conditioner half of a split MiniMax-H3 `ref2va` deployment: resolves the request plan (canvas, frame "
111
- "count, references normalized onto MiniMax-H3's own rates and resolutions) and encodes MiniMax-H3's "
112
- "presentation of it into the `prompt_embeds` / `text_token_tags` pair the denoising half consumes."
113
- )
114
-
115
  @property
116
  def outputs(self):
117
  return _wire_outputs()
118
 
119
 
120
  class MiniMaxH3Ref2VAGeneratorBlocks(SequentialPipelineBlocks):
121
- """The denoising half of a split `ref2va`: the `ref2va` branch with its `text_encoder` step removed.
122
-
123
- `reference_encoder` stays here, next to the two autoencoders it runs: its output shapes are where every reference
124
- block's geometry in the packed layout comes from.
125
- """
126
-
127
  model_name = "minimax-h3"
128
  block_classes = [
129
  MiniMaxH3Ref2VASetupStep,
@@ -134,14 +87,6 @@ class MiniMaxH3Ref2VAGeneratorBlocks(SequentialPipelineBlocks):
134
  ]
135
  block_names = ["setup", "reference_encoder", "denoise", "after_denoise", "decode"]
136
 
137
- @property
138
- def description(self):
139
- return (
140
- "The denoising half of a split MiniMax-H3 `ref2va` deployment: the `ref2va` branch of `MiniMaxH3Blocks` "
141
- "without its text-encoder step, so `prompt_embeds` and `text_token_tags` come in as inputs and the "
142
- "62.14 GiB Qwen3-VL conditioner is never loaded here. The transformer is the `transformer_ref` partition."
143
- )
144
-
145
  @property
146
  def outputs(self):
147
  return _generation_outputs()
 
1
+ """The denoising half of MiniMax-H3 used by this Space.
2
 
3
+ The Qwen3-VL conditioner is served by the public
4
+ multimodalart/qwen3vl-conditioner Space so the generator can fit on ZeroGPU.
 
 
 
 
 
 
 
 
5
  """
6
 
7
  from diffusers.modular_pipelines.minimax_h3.before_encoder import MiniMaxH3Ref2VASetupStep
 
24
 
25
 
26
  def _wire_outputs(num_frames: bool = True) -> list[OutputParam]:
 
27
  return [
28
  OutputParam.template("prompt_embeds"),
29
+ OutputParam("text_token_tags", description="The per-row modality tag of every row of prompt_embeds."),
30
+ OutputParam("height", type_hint=int),
31
+ OutputParam("width", type_hint=int),
32
+ *([OutputParam("num_frames", type_hint=int)] if num_frames else []),
 
 
 
 
33
  ]
34
 
35
 
36
  class MiniMaxH3ConditionerBlocks(SequentialPipelineBlocks):
 
 
37
  model_name = "minimax-h3"
38
  block_classes = [MiniMaxH3AutoResizeStep, MiniMaxH3TextEncoderStep]
39
  block_names = ["resize", "text_encoder"]
40
 
 
 
 
 
 
 
 
 
41
  @property
42
  def outputs(self):
43
  return _wire_outputs(num_frames=False)
44
 
45
 
46
  class MiniMaxH3GeneratorBlocks(SequentialPipelineBlocks):
 
 
47
  model_name = "minimax-h3"
48
  block_classes = [
49
  MiniMaxH3AutoResizeStep,
 
57
  @property
58
  def description(self):
59
  return (
60
+ "The denoising half of MiniMax-H3 with the text encoder removed. "
61
+ "Prompt embeddings and token tags arrive from the external conditioner Space."
 
62
  )
63
 
64
  @property
 
67
 
68
 
69
  class MiniMaxH3Ref2VAConditionerBlocks(SequentialPipelineBlocks):
 
 
 
 
 
 
 
70
  model_name = "minimax-h3"
71
  block_classes = [MiniMaxH3Ref2VASetupStep, MiniMaxH3Ref2VATextEncoderStep]
72
  block_names = ["setup", "text_encoder"]
73
 
 
 
 
 
 
 
 
 
74
  @property
75
  def outputs(self):
76
  return _wire_outputs()
77
 
78
 
79
  class MiniMaxH3Ref2VAGeneratorBlocks(SequentialPipelineBlocks):
 
 
 
 
 
 
80
  model_name = "minimax-h3"
81
  block_classes = [
82
  MiniMaxH3Ref2VASetupStep,
 
87
  ]
88
  block_names = ["setup", "reference_encoder", "denoise", "after_denoise", "decode"]
89
 
 
 
 
 
 
 
 
 
90
  @property
91
  def outputs(self):
92
  return _generation_outputs()
requirements.txt CHANGED
@@ -1,6 +1,4 @@
1
- # diffusers from the MiniMax-H3 PR (https://github.com/huggingface/diffusers/pull/14371), pinned to a commit
2
- # on its minimax-h3-refactor branch. This Space subclasses its block classes in h3_split_blocks.py.
3
- # 665f578278365ea4a3318cb8c9b66ce6c01204b9 = refs/pull/14371/head at the time of this deploy
4
  --extra-index-url https://download.pytorch.org/whl/cu130
5
  diffusers @ git+https://github.com/huggingface/diffusers.git@665f578278365ea4a3318cb8c9b66ce6c01204b9
6
  torch==2.11.0
@@ -8,10 +6,8 @@ torchvision==0.26.0
8
  transformers==5.8.0
9
  accelerate==1.14.0
10
  huggingface-hub==1.24.0
11
- gradio @ https://huggingface.co/buckets/gradio/pypi-previews/resolve/a9ce60ae03123ed6374b739d862e62dcae2f451e/gradio-6.25.0-py3-none-any.whl
12
  av
13
  pillow
14
  numpy
15
  requests
16
  safetensors>=0.8.0
17
- peft
 
1
+ # MiniMax-H3 modular pipeline support is currently provided by this reviewed PR.
 
 
2
  --extra-index-url https://download.pytorch.org/whl/cu130
3
  diffusers @ git+https://github.com/huggingface/diffusers.git@665f578278365ea4a3318cb8c9b66ce6c01204b9
4
  torch==2.11.0
 
6
  transformers==5.8.0
7
  accelerate==1.14.0
8
  huggingface-hub==1.24.0
 
9
  av
10
  pillow
11
  numpy
12
  requests
13
  safetensors>=0.8.0