| --- |
| base_model: MiniMaxAI/MiniMax-H3 |
| library_name: pytorch |
| license: other |
| license_name: minimax-h3-community |
| license_link: LICENSE |
| pipeline_tag: image-to-video |
| tags: |
| - lora |
| - audio-video-generation |
| - reference-to-video |
| - streaming |
| - safetensors |
| --- |
| |
| **Paper:** [TaoMate-R1: Temporal Role Specialization for Streaming Multi-Reference Video Generation](https://huggingface.co/papers/2611.14307) |
| **Code:** [TaoMate-H3](https://github.com/TaoLiveAIGC/TaoMate-H3) |
|
|
| # TaoMate-R1 |
|
|
| Reference-conditioned streaming audio-video generation built on |
| [MiniMax H3](https://huggingface.co/MiniMaxAI/MiniMax-H3), developed by the |
| [Alibaba TaoLive AIGC Team](https://github.com/TaoLiveAIGC). |
|
|
| The model generates audio and video jointly in three denoising steps per chunk. |
| An opening LoRA handles the first chunk; a continuation LoRA handles later chunks |
| with clean audio-video history. Both are online generator adapters, stored in |
| float32 with LoRA rank 128 and alpha 128. |
|
|
| ## Files |
|
|
| ```text |
| config.json |
| cold/config.json |
| cold/adapter_model.safetensors |
| warm/config.json |
| warm/adapter_model.safetensors |
| ``` |
|
|
| Each bank contains 416 native H3 LoRA tensors. The runtime loads these adapters |
| onto the MiniMax H3 **Ref2VA** base-model partition. |
|
|
| ## Usage |
|
|
| Install [TaoMate-H3](https://github.com/TaoLiveAIGC/TaoMate-H3) using its shared |
| installation instructions, then run the following commands from the repository |
| root. Download the base partition and adapters: |
|
|
| ```bash |
| hf download MiniMaxAI/MiniMax-H3 \ |
| --include "Ref2VA/*" --local-dir models/MiniMax-H3 |
| |
| hf download TaoLiveAIGC/TaoMate-R1 --local-dir models/TaoMate-R1 |
| ``` |
|
|
| Run the included presenter-and-product example: |
|
|
| ```bash |
| python -m taomate_h3.ref2av \ |
| --model-root models/MiniMax-H3 \ |
| --adapter models/TaoMate-R1 \ |
| --prompt-file examples/ref2av_prompt.txt \ |
| --reference-image examples/ref2av/presenter.png \ |
| --reference-image examples/ref2av/product.png \ |
| --duration 10 --resolution 768x1344 --seed 9111 \ |
| --gpus 8 --output outputs/ref2av |
| ``` |
|
|
| Repeat `--reference-image` or `--reference-video` for multiple visual references. |
| Their command-line order is preserved. Reference-video audio is ignored; output |
| audio is generated by the model. The result is `outputs/ref2av/video.mp4`. |
|
|
| [Inputs and usage](https://github.com/TaoLiveAIGC/TaoMate-H3#-ref2av-taomate-r1) |
| · [Example video](https://github.com/user-attachments/assets/8a95695b-3e6a-410f-a82d-e74169b06173) |
|
|
| ## License and acknowledgments |
|
|
| This release follows the [MiniMax H3 Community License Agreement](LICENSE). |
| We thank MiniMax, Qwen, PyTorch, and the Self Forcing authors for their models, |
| implementations, and research. |
