| # ComfyUI plugin (SM89 and SM120) |
|
|
| This repository is also a ComfyUI custom node for the native MiniMax-H3 model. |
| Its current implementation and cross-pipeline differences are described in the |
| [ComfyUI guide](comfyui/README.md#implementation-notes). |
| The node uses architecture-specific comfy-kitchen Spark-H3 CUDA kernels on |
| NVIDIA SM89 (RTX 4090) and SM120 (RTX 50-series) GPUs and keeps all packed |
| text, image/video reference, and audio conditioning exact. |
|
|
| The integration is built on ComfyUI's official MiniMax-H3 sparse-attention |
| architecture and the comfy-kitchen v0.2.36 Sol kernel stack. It installs |
| `dit/double_block` replacements, follows the native sigma-based sparse window, |
| resets state through `ON_CLEANUP`, and projects QKV in 4096-token chunks. The |
| projected BTHD tensors then use comfy-kitchen kernels for Top-K routing and |
| global reweighting. The fanout-16, 32-landmark, group-1 reblock permutation is |
| consumed directly by Sol preprocessing and its output scatter, without four |
| materialized Q/K/V/output gathers. Global reweighting uses the mean |
| of all target-video queries as one anchor per head; conditioning queries remain |
| dense. It does not replace each attention module's `forward` method. |
|
|
| ## Install |
|
|
| This release provides two Linux x86_64 CUDA 13.0 archives: |
| |
| | GPU | CUDA toolkit | Release archive | |
| | --- | --- | --- | |
| | SM120 (RTX 50 series) | 13.0 | `ComfyUI-Spark-H3-<version>-linux-x86_64-sm120-cu130.zip` | |
| | SM89 (RTX 4090) | 13.0 | `ComfyUI-Spark-H3-<version>-linux-x86_64-sm89-cu130.zip` | |
|
|
| Select the archive matching both the GPU architecture and |
| `torch.version.cuda`. CUDA 12.8 is available only through the explicit |
| `--experimental-cuda` local-wheel or source-build path and is not a release |
| asset. Each archive contains only the custom node, its reblock runtime, |
| workflows, and compatible backend wheels—not this repository's model or research |
| assets. Extract it below `custom_nodes` and run its installer with the same |
| Python interpreter that starts ComfyUI: |
|
|
| ```bash |
| cd /path/to/ComfyUI/custom_nodes |
| unzip /path/to/ComfyUI-Spark-H3-<version>-linux-x86_64-sm120-cu130.zip # SM120 example |
| /path/to/ComfyUI/.venv/bin/python ComfyUI-Spark-H3/install.py |
| ``` |
|
|
| After registry publication, ComfyUI Manager is the recommended discovery and |
| installation UI, but it remains optional: Spark-H3 neither bundles nor installs |
| Manager. The manual ZIP path is intentionally kept as an independently testable |
| baseline. |
|
|
| The cross-platform installer first accepts an existing compatible backend, |
| then selects a bundled or pinned-release wheel. It falls back to building the |
| pinned, patched `comfy-kitchen` source only when no wheel matches. Source builds |
| require Git, CMake, Ninja, a C++ compiler, and CUDA `nvcc`. |
|
|
| The backend wheel version encodes both architecture and CUDA toolchain: SM89 |
| CU130 uses `+spark.h3.sm89.cu130.1`, SM120 CU130 uses |
| `+spark.h3.sm120.cu130.1`, and experimental SM120 CU128 uses |
| `+spark.h3.sm120.cu128.1`. SM89 is compiled with CUDA architecture `89`, while |
| SM120 CU130 is compiled with `120f`. Archive names and |
| `spark_h3_build.json` also encode `linux-x86_64` or `windows-x86_64`; the |
| packager and installer reject wheels or packages for the other OS. Python |
| 3.10/3.11 also requires its own wheel or source compilation. Windows packaging |
| is prepared, but the Windows runtime has not yet been validated. Other GPU |
| architectures are unsupported. |
|
|
| Restart ComfyUI afterward; select **MiniMax H3 Spark Attention (SM89)** on an |
| RTX 4090 or **MiniMax H3 Spark Attention (SM120)** on an RTX 50-series GPU. |
| Both appear in `model_patches/attention`. See the dedicated |
| [ComfyUI installation and workflow guide](comfyui/README.md) for model files and |
| examples. |
|
|
| ## Workflow |
|
|
| Insert the node after `UNETLoader` and before `BasicGuider`: |
|
|
| ```text |
| UNETLoader -> MiniMax H3 Spark Attention (SM89 or SM120) -> BasicGuider |
| ``` |
|
|
| The [14.4 s native model example](workflows/spark_h3_vdn8_14p4s_t2va.json) |
| uses the complete three-shot VDN prompt 8, 345 frames at 1344×768, and a |
| 20-step sampler. It has one Spark patch in the model path, without a Turbo |
| LoRA or a second sparse-attention patch. It uses the same 20% Top-K ratio as |
| the LoRA examples, with four dense warmup evaluations. Bundled workflows |
| default to the official `minimax_h3_video_vae_fp16.safetensors`. |
|
|
| Two matching 14.4-second, 8-step LoRA examples use the same full prompt, |
| seed, resolution, Spark 20% Top-K ratio, and two dense warmup evaluations. The |
| DMAD and Alibaba-PAI examples retain the sampling protocol required by their |
| respective students: |
|
|
| | Workflow | LoRA loader | Sampler | |
| | --- | --- | --- | |
| | [LightX2V 768p 8-step LoRA](workflows/spark_h3_lightx2v_768p_8step_lora_14p4s_t2va.json) | `LoraLoaderModelOnly` | `res_multistep` | |
| | [Larryvrh v4 8-step LoRA](workflows/spark_h3_larryvrh_8step_lora_14p4s_t2va.json) | `LoraLoaderModelOnly` (converted weight) | `euler` | |
| | [DMAD 4-step LoRA](workflows/spark_h3_dmad_4step_lora_5p2s_t2va.json) | `LoraLoaderModelOnly` | `MiniMaxH3DMADSampler` | |
| | [Official LBH two-pass + LightX2V 4-step](workflows/spark_h3_lbh_official_lightx2v_4step_5p2s_i2va.json) | `LoraLoaderModelOnly` | two `SamplerCustomAdvanced` passes | |
| | [Alibaba-PAI PDD Acc FL2VA 8-step LoRA](workflows/spark_h3_alibaba_pai_acc_fl2va_8step_lora_5p2s_t2va.json) | `MiniMaxH3PDDAccApply` | `euler` + node-provided sigmas | |
|
|
| Larryvrh's original LoRA names its tensors differently and targets the full |
| 2688-wide AdaLN time embedding, while the pruned ComfyUI model stores an |
| 8-dimensional time-curve basis. Convert it once with |
| `tools/convert_larryvrh_lora_comfyui.py`; the result preserves the 208 backbone |
| updates and projects all 51 AdaLN updates into that curve basis. The bundled |
| workflow then follows the same low-overhead path as LightX2V: stock |
| `LoraLoaderModelOnly` plus a stock sampler, with no runtime-LoRA matmuls or |
| Larryvrh custom node. The measured projection error before the AdaLN output |
| matrix is about 0.10% mean and 0.22% worst-case for the published v4 weight. |
|
|
| ```bash |
| python tools/convert_larryvrh_lora_comfyui.py \ |
| --input /path/to/minimax_h3_turbo_v4_step600_ema.safetensors \ |
| --output /path/to/ComfyUI/models/loras/minimax_h3_turbo_v4_step600_ema_comfyui_curve_bf16.safetensors |
| ``` |
|
|
| Both the LightX2V and Larryvrh examples remain in the standalone ZIP as |
| explicitly labelled optional integrations. |
|
|
| DMAD publishes a Diffusers-format rank-128 LoRA. Convert it before selecting it |
| in `LoraLoaderModelOnly`: |
|
|
| ```bash |
| python scripts/convert_dmad_lora_to_comfyui.py \ |
| /path/to/dmad_minimax_h3_4step_lora_critic.safetensors \ |
| /path/to/ComfyUI/models/loras/dmad_minimax_h3_4step_lora_critic_comfyui_bf16.safetensors |
| ``` |
|
|
| The converter fuses Q/K/V adapters without changing their represented update |
| and swaps the Diffusers SwiGLU halves to ComfyUI order. The DMAD workflow then |
| applies `MiniMaxH3SigmaShift` with video shift 12 and audio shift 2 before |
| Spark. Use the bundled `MiniMaxH3DMADSampler`; a regular Euler or |
| `res_multistep` sampler does not implement the re-noise rule used to train the |
| released student. |
|
|
| The LBH example follows the repository's `h3_lbh` implementation and LBH's |
| published ComfyUI graph. It requires |
| `LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler` and |
| `minimax_h3_latent_upscaler_3d_conv_v1_fp16.safetensors` under |
| `ComfyUI/models/latent_upscale_models/`. The first pass consumes four |
| evaluations from an eight-point `simple` schedule at 0.2 MP. Its clean |
| `denoised_output` is separated into video and audio; only the video latent is |
| lifted. The second pass re-noises at 0.98 MP and runs three evaluations at |
| video sigmas `0.9035, 0.6316, 0.3158, 0.0`. Spark applies only to this |
| high-resolution pass, with no warmup and the Diffusers-side 10% Top-K setting. |
|
|
| Alibaba-PAI's MiniMax-H3-Acc checkpoints are also Diffusers-side weights, but |
| they are not ordinary PEFT LoRAs. Each file contains a rank-64 trunk LoRA and |
| 32 PDD video/audio output heads. Diffusers can use the official file directly |
| with [`scripts/minimax_h3_pdd.py`](scripts/minimax_h3_pdd.py): call |
| `apply_pdd_lora` on `pipeline.transformer` for FL2VA or |
| `pipeline.transformer_ref` for Ref2VA, then pass |
| `num_inference_steps=nfe + 1` to the modular pipeline. |
| The pinned source revision, local file sizes, and SHA-256 checksums are recorded |
| in the [MiniMax-H3-Acc integration note](docs/alibaba_pai_minimax_h3_acc.md). |
|
|
| For ComfyUI, install the Apache-2.0 |
| `BSAI-ComfyUI-MiniMax-H3-PDD-Acc` node pack and place the original weights in |
| `models/pdd_acc`. The node reads the official files directly, converts the |
| trunk keys in memory, handles the pruned checkpoint's AdaLN curve basis, and |
| installs the PDD head bank. Pair the FL2VA weight only with an FL2VA base and |
| the Ref2VA weight only with a Ref2VA base. Use CFG 1.0, video/audio shifts |
| 12/3, the plain `euler` sampler, and wire the Apply node's `sigmas` output to |
| `SamplerCustomAdvanced`; do not substitute `BasicScheduler`, `res_multistep`, |
| or another distillation LoRA. |
|
|
| The defaults select a 20% Top-K ratio, 20% dense warmup, and the first |
| transformer block kept dense. Set `steps` to the |
| number of model evaluations made by the sampler (normally the sampler's step |
| count). The warmup ratio is converted to an exact number of model evaluations; |
| minimum-token and dense-layer gating use the same |
| policy object as ComfyUI's official sparse node. |
| The public Spark node uses the full reweight path and a dense video tail; |
| research ablation and tail-mode controls are not exposed in the node UI. |
| The default `reblock_layout=q_reuse_k` builds the K-side landmark layout once |
| and aliases that permutation for Q, matching Spark's `q_from_k` mode while |
| avoiding the query-side M2 transform, tree plan, and duplicate index storage. |
| Select `independent` to reproduce the former separately planned Q/K layouts. |
| The default `topk_mode=topk_ratio` selects a fraction of the target video's |
| 64-token key blocks with `topk_ratio` (default 0.2). Set |
| `topk_mode=topk_blocks` to request a fixed number with `topk_blocks` (default |
| 228); the ratio field is ignored in this mode. The block count is capped at the number |
| of candidate video blocks, while conditioning/sink blocks remain exact and |
| do not consume the Top-K budget. To use a 10% ratio, set `topk_mode=topk_ratio` |
| and `topk_ratio=0.1`. |
| `warmup_mode=warmup_ratio` (default) computes dense warmup as the ceiling of |
| `steps * warmup_ratio`, where `0.2` means 20%; `warmup_mode=warmup_steps` |
| uses the fixed |
| `warmup_steps` count (default 4). Both modes cap warmup at `steps`. |
| The frontend migrates legacy workflow values such as `warmup_percent=20` to |
| `warmup_ratio=0.2` when loading the graph. |
| `strict=true` is recommended: it reports an incompatible ComfyUI build, |
| dtype, GPU, or kernel error instead of silently switching to dense attention. |
|
|
| Requirements: |
|
|
| - ComfyUI 0.38.x or 0.39.x with native MiniMax-H3 support. |
| - CUDA BF16 execution on a compute-capability 8.9 or 12.0 GPU. |
| - The ComfyUI-supported PyTorch/CUDA stack and a comfy-kitchen build containing |
| the Spark Top-K, reblock, and global-reweight extension. |
|
|
| The ComfyUI integration never calls Spark-H3's research Triton/CuTe attention |
| backends. The compatibility Sol node delegates to ComfyUI's official |
| `apply_block_sparse_attention`, while the Spark node fails explicitly unless |
| the comfy-kitchen global Spark backend is selected. The repository's original |
| PyTorch pipeline and experimental kernels remain available only through the |
| Python API. |
|
|
| The node is compatible with ComfyUI's T2VA, FL2VA, and Ref2VA packed layouts. |
| It uses `transformer_options["minimax_h3_layout"]` rather than guessing token |
| spans. Do not stack it with another node that replaces MiniMax-H3 |
| `dit/double_block` entries; choose one sparse-attention implementation per |
| workflow. |
|
|
