README.md

---
license: apache-2.0
language:
- en
base_model:
- Wan-AI/Wan2.1-T2V-14B
pipeline_tag: image-to-video
tags:
- Talking Avatar
- Audio-Driven Video Generation
- Video Reference
---

<div style="display:flex;align-items:center;justify-content:space-between;gap:16px"><span style="font-size:2em;font-weight:600;line-height:1.25">TAVR</span><picture style="flex:none"><source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/HeyGenAI/TAVR/resolve/main/assets/heygen-logo-white.png"><img src="https://huggingface.co/HeyGenAI/TAVR/resolve/main/assets/heygen-logo-black.png" alt="HeyGen Research" style="height:44px;display:block"></picture></div>

<p align="center">
  <a href="https://arxiv.org/abs/2604.27918"><img src="https://img.shields.io/badge/arXiv-2604.27918-b31b1b.svg" alt="arXiv"></a>
  <a href="https://gseancdat.github.io/projects/TAVR.html"><img src="https://img.shields.io/badge/Project-Page-1f6feb.svg" alt="Project Page"></a>
  <a href="https://github.com/heygen-com/TAVR"><img src="https://img.shields.io/badge/GitHub-heygen--com%2FTAVR-181717.svg" alt="GitHub"></a>
  <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-3da639.svg" alt="License"></a>
</p>

<p align="center">
  <b>Generate Your Talking Avatar from Video Reference</b>
</p>

<p align="center">
  <b>Accepted to SIGGRAPH Asia 2026</b>
</p>

> [!IMPORTANT]
> **This is not a drop-in Wan2.1 checkpoint.** It is a TAVR transformer with 18 input channels
> and its own reference and audio conditioning, and it needs four companion models at inference
> time (Wan VAE, umT5, wav2vec 2.0, and a person detector plus a pose model) that are not bundled
> here. A stock `diffusers` Wan2.1 pipeline will not produce correct results.
>
> Run it through the released inference code: **https://github.com/heygen-com/TAVR**

## Model Description

TAVR replaces single-image avatar references with short video clips, enabling cross-scene
generation with significantly better identity preservation. A three-stage training strategy
bridges the domain gap between reference and target scenes.

## Model Details

| | |
|---|---|
| Base model | [Wan-AI/Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B) |
| Input | reference video, target still, caption, driving audio |
| Output | 896x480, 25 fps, driving audio muxed in |
| Sampler | UniPC, 24 steps, flow shift 5.0; text guidance 5.0, audio guidance 1.8 |
| Reference frames | Flexible (default 20) |
| Precision | `bfloat16` |
| Hardware | one Hopper-class CUDA GPU with at least 80 GB of memory |

## Usage

```bash
git clone https://github.com/heygen-com/TAVR
cd TAVR

python3.10 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu126

hf download HeyGenAI/TAVR --local-dir ./ckpt

PYTHONPATH=. python infer.py \
  --sample-dir samples/example1 \
  --dit-ckpt ckpt/tavr_transformer.safetensors \
  --ckpt-dir . \
  --output-dir outputs
```

`flash_attn_3` has no PyPI wheel and is the only attention backend; it and the companion models
are set up as described in the
[repository README](https://github.com/heygen-com/TAVR).

## Evaluation

Qualitative results and comparisons against baselines are on the
[project page](https://gseancdat.github.io/projects/TAVR.html).

On the cross-scene benchmark, TAVR yields the best identity similarity and an overall quality
score of 16.42 against 14.13 for the next best method.

### Cross-scene benchmark data

`benchmark_data.json` lists the 158 reference/target pairs of the cross-scene benchmark. The clips
come from the public [TalkVid](https://github.com/FreedomIntelligence/TalkVid) dataset. Each sample has:

| field | content |
|---|---|
| `reference`, `target` | `video_id`, `video_url`, `start_time` / `end_time` (seconds), `start_frame` / `end_frame` at the source `fps`, `width`, `height` |
| `target_caption` | scene caption used as the text prompt |

## Release Progress

- [x] Model weights
- [x] Inference code
- [x] Cross-scene benchmark data

## Citation

```bibtex
@inproceedings{guo2026generate,
     title={Generate Your Talking Avatar from Video Reference},
     author={Guo, Zujin and Ye, Zhenhui and Ren, Yi and Li, Yuanming and Chen, Ce and Hong, Zhibin and Loy, Chen Change},
      booktitle={SIGGRAPH Asia 2026 Conference Papers},
     year={2026}
}
```

## License

Apache License 2.0, inherited from [Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B).

### Third-party weights at inference time

<sub>Running TAVR additionally requires
<a href="https://github.com/IDEA-Research/DWPose">DWPose</a> pose weights (Apache-2.0) and
<a href="https://github.com/ultralytics/ultralytics">Ultralytics YOLO</a> weights
(<b>AGPL-3.0</b>). Neither is redistributed here, and neither is contained in these weights;
obtain them from their own channels and observe their terms.</sub>

<sub>All videos and results shown here are for research demonstration purposes only.</sub>