TAVR release

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,120 @@
+---
+license: apache-2.0
+language:
+- en
+base_model:
+- Wan-AI/Wan2.1-T2V-14B
+pipeline_tag: image-to-video
+tags:
+- Talking Avatar
+- Audio-Driven Video Generation
+- Video Reference
+---
+
+<div style="display:flex;align-items:center;justify-content:space-between;gap:16px"><span style="font-size:2em;font-weight:600;line-height:1.25">TAVR</span><picture style="flex:none"><source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/HeyGenAI/TAVR/resolve/main/assets/heygen-logo-white.png"><img src="https://huggingface.co/HeyGenAI/TAVR/resolve/main/assets/heygen-logo-black.png" alt="HeyGen Research" style="height:44px;display:block"></picture></div>
+
+<p align="center">
+  <a href="https://arxiv.org/abs/2604.27918"><img src="https://img.shields.io/badge/arXiv-2604.27918-b31b1b.svg" alt="arXiv"></a>
+  <a href="https://gseancdat.github.io/projects/TAVR.html"><img src="https://img.shields.io/badge/Project-Page-1f6feb.svg" alt="Project Page"></a>
+  <a href="https://github.com/heygen-com/TAVR"><img src="https://img.shields.io/badge/GitHub-heygen--com%2FTAVR-181717.svg" alt="GitHub"></a>
+  <a href="https://www.apache.org/licenses/LICENSE-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-3da639.svg" alt="License"></a>
+</p>
+
+<p align="center">
+  <b>Generate Your Talking Avatar from Video Reference</b>
+</p>
+
+<p align="center">
+  <b>Accepted to SIGGRAPH Asia 2026</b>
+</p>
+
+> [!IMPORTANT]
+> **This is not a drop-in Wan2.1 checkpoint.** It is a TAVR transformer with 18 input channels
+> and its own reference and audio conditioning, and it needs four companion models at inference
+> time (Wan VAE, umT5, wav2vec 2.0, and a person detector plus a pose model) that are not bundled
+> here. A stock `diffusers` Wan2.1 pipeline will not produce correct results.
+>
+> Run it through the released inference code: **https://github.com/heygen-com/TAVR**
+
+## Model Description
+
+TAVR replaces single-image avatar references with short video clips, enabling cross-scene
+generation with significantly better identity preservation. A three-stage training strategy
+bridges the domain gap between reference and target scenes.
+
+## Model Details
+
+| | |
+|---|---|
+| Base model | [Wan-AI/Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B) |
+| Input | reference video, target still, caption, driving audio |
+| Output | 896x480, 25 fps, driving audio muxed in |
+| Sampler | UniPC, 24 steps, flow shift 5.0; text guidance 5.0, audio guidance 1.8 |
+| Reference frames | Flexible (default 20) |
+| Precision | `bfloat16` |
+| Hardware | one Hopper-class CUDA GPU with at least 80 GB of memory |
+
+## Usage
+
+```bash
+git clone https://github.com/heygen-com/TAVR
+cd TAVR
+
+python3.10 -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu126
+
+hf download HeyGenAI/TAVR --local-dir ./ckpt
+
+PYTHONPATH=. python infer.py \
+  --sample-dir samples/example1 \
+  --dit-ckpt ckpt/tavr_transformer.safetensors \
+  --ckpt-dir . \
+  --output-dir outputs
+```
+
+`flash_attn_3` has no PyPI wheel and is the only attention backend; it and the companion models
+are set up as described in the
+[repository README](https://github.com/heygen-com/TAVR).
+
+## Evaluation
+
+Qualitative results and comparisons against baselines are on the
+[project page](https://gseancdat.github.io/projects/TAVR.html).
+
+On the cross-scene benchmark, TAVR yields the best identity similarity and an overall quality
+score of 16.42 against 14.13 for the next best method. The benchmark data is not part of this
+release.
+
+## Release Progress
+
+- [x] Model weights
+- [x] Inference code
+- [ ] Evaluation code
+- [ ] Cross-scene benchmark data
+
+🚧 The remaining items are being prepared for release.
+
+## Citation
+
+```bibtex
+@inproceedings{guo2026generate,
+     title={Generate Your Talking Avatar from Video Reference},
+     author={Guo, Zujin and Ye, Zhenhui and Ren, Yi and Li, Yuanming and Chen, Ce and Hong, Zhibin and Loy, Chen Change},
+      booktitle={SIGGRAPH Asia 2026 Conference Papers},
+     year={2026}
+}
+```
+
+## License
+
+Apache License 2.0, inherited from [Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B).
+
+### Third-party weights at inference time
+
+<sub>Running TAVR additionally requires
+<a href="https://github.com/IDEA-Research/DWPose">DWPose</a> pose weights (Apache-2.0) and
+<a href="https://github.com/ultralytics/ultralytics">Ultralytics YOLO</a> weights
+(<b>AGPL-3.0</b>). Neither is redistributed here, and neither is contained in these weights;
+obtain them from their own channels and observe their terms.</sub>
+
+<sub>All videos and results shown here are for research demonstration purposes only.</sub>

tavr_transformer.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3eee7781ebf6fca9d85887c2234f9e263f364f83eab399250d7786b289a6c9c
+size 37812071752