File size: 2,643 Bytes
a19af24 377b70c 89eecdb 377b70c a19af24 89eecdb 377b70c 89eecdb f69bf7a 89eecdb f69bf7a 89eecdb f69bf7a 89eecdb f69bf7a 89eecdb 377b70c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | ---
base_model: MiniMaxAI/MiniMax-H3
library_name: pytorch
license: other
license_name: minimax-h3-community
license_link: LICENSE
pipeline_tag: image-to-video
tags:
- lora
- audio-video-generation
- reference-to-video
- streaming
- safetensors
---
**Paper:** [TaoMate-R1: Temporal Role Specialization for Streaming Multi-Reference Video Generation](https://huggingface.co/papers/2611.14307)
**Code:** [TaoMate-H3](https://github.com/TaoLiveAIGC/TaoMate-H3)
# TaoMate-R1
Reference-conditioned streaming audio-video generation built on
[MiniMax H3](https://huggingface.co/MiniMaxAI/MiniMax-H3), developed by the
[Alibaba TaoLive AIGC Team](https://github.com/TaoLiveAIGC).
The model generates audio and video jointly in three denoising steps per chunk.
An opening LoRA handles the first chunk; a continuation LoRA handles later chunks
with clean audio-video history. Both are online generator adapters, stored in
float32 with LoRA rank 128 and alpha 128.
## Files
```text
config.json
cold/config.json
cold/adapter_model.safetensors
warm/config.json
warm/adapter_model.safetensors
```
Each bank contains 416 native H3 LoRA tensors. The runtime loads these adapters
onto the MiniMax H3 **Ref2VA** base-model partition.
## Usage
Install [TaoMate-H3](https://github.com/TaoLiveAIGC/TaoMate-H3) using its shared
installation instructions, then run the following commands from the repository
root. Download the base partition and adapters:
```bash
hf download MiniMaxAI/MiniMax-H3 \
--include "Ref2VA/*" --local-dir models/MiniMax-H3
hf download TaoLiveAIGC/TaoMate-R1 --local-dir models/TaoMate-R1
```
Run the included presenter-and-product example:
```bash
python -m taomate_h3.ref2av \
--model-root models/MiniMax-H3 \
--adapter models/TaoMate-R1 \
--prompt-file examples/ref2av_prompt.txt \
--reference-image examples/ref2av/presenter.png \
--reference-image examples/ref2av/product.png \
--duration 10 --resolution 768x1344 --seed 9111 \
--gpus 8 --output outputs/ref2av
```
Repeat `--reference-image` or `--reference-video` for multiple visual references.
Their command-line order is preserved. Reference-video audio is ignored; output
audio is generated by the model. The result is `outputs/ref2av/video.mp4`.
[Inputs and usage](https://github.com/TaoLiveAIGC/TaoMate-H3#-ref2av-taomate-r1)
· [Example video](https://github.com/user-attachments/assets/8a95695b-3e6a-410f-a82d-e74169b06173)
## License and acknowledgments
This release follows the [MiniMax H3 Community License Agreement](LICENSE).
We thank MiniMax, Qwen, PyTorch, and the Self Forcing authors for their models,
implementations, and research. |