| # TaoMate-H3 FL2VA LoRA for ComfyUI |
|
|
| [TaoMate-H3-FL2VA](https://huggingface.co/TaoLiveAIGC/TaoMate-H3-FL2VA) by the Alibaba TaoLive AIGC Team is a |
| 3-step first/last-frame-to-audio-video LoRA for MiniMax H3 (step-300 online generator, rank 128, alpha 128), |
| released in a PEFT-style layout for TaoMate's own streaming runtime. This is the same LoRA with its keys converted |
| to ComfyUI's layout. |
|
|
| Unlike the [attention config LoRAs](attention.md), this is a **weight LoRA**: it changes the transformer's weights |
| and holds ~620M parameters. |
|
|
| | File | What | Size | |
| |---|---|---| |
| | `distill_loras/taomate_h3_fl2va_step300_bf16.safetensors` | TaoMate-H3-FL2VA step 300, full rank 128, ComfyUI keys | 1.24 GB | |
|
|
| ## Conversion |
|
|
| ComfyUI's H3 model uses the same module names and the same fused layouts as TaoMate's (`attn.qkv_proj` as q, k, v |
| concatenated; `mlp.fc1` as the fused SwiGLU input), so no weight is reordered: |
|
|
| - keys renamed `blocks.N.attn.qkv_proj.lora_a` / `lora_b` → |
| `diffusion_model.blocks.N.attn.qkv_proj.lora_down.weight` / `lora_up.weight`, plus `.alpha` = 128; |
| - fp32 rounded to bf16 (the only lossy step; rank, alpha and shapes unchanged); |
| - TaoMate's `adapter_config.json` is kept in the file's metadata. |
|
|
| Checked with ComfyUI's own LoRA loader against the H3 model: all 208 layers (50 DiT blocks and 2 token refiner |
| blocks × qkv, out, fc1, fc2) are matched with the expected shapes, and no key is left unloaded. |
|
|
| ## Use |
|
|
| Put it in `models/loras/` and load it with **Load LoRA (Model Only)** on the H3 FL2VA transformer, before any |
| attention config LoRA. Alpha equals rank, so strength 1.0 is the adapter's own scale. |
|
|
| TaoMate has not released FL2VA inference code yet. Their released T2AV code |
| ([TaoMate-H3](https://github.com/TaoLiveAIGC/TaoMate-H3), `denoise_schedule.py`, `model/pipeline.py`, |
| `model/denoise.py`) samples its 3-step LoRA like this: |
|
|
| - 3 steps at states 0, 16, 33, 49 of a 50-step shifted schedule: video shift 12 (sigmas 1.0, 0.9612, 0.8533, 0), |
| audio shift 3 (sigmas 1.0, 0.8609, 0.5926, 0); |
| - a deterministic Euler update on the flow; |
| - positive prompt only, no CFG. |
|
|
| In ComfyUI that is **ModelSamplingMiniMaxH3** (shift_video 12, shift_audio 3), **ManualSigmas** |
| `1.0, 0.9612, 0.8533, 0.0`, sampler `euler`, CFG 1. ComfyUI derives the audio sigmas from the video ones through |
| the two shifts and gives 1.0, 0.861, 0.5925, 0. Whether FL2VA uses the same schedule is not confirmed until |
| TaoMate's FL2VA code is out. |
|
|
| TaoMate's runtime generates in streamed chunks with a persistent audio-video KV cache; ComfyUI samples the whole |
| clip in one pass. |
|
|
| ## License and attribution |
|
|
| TaoMate-H3 is by the Alibaba TaoLive AIGC Team, built on MiniMax H3, and distributed under the MiniMax H3 Community |
| License Agreement. The converted file is a modified derivative (keys renamed, fp32 → bf16); see `NOTICE`. |
|
|
