File size: 2,252 Bytes
9e1f36d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6d29bb6
9e1f36d
 
 
 
 
 
 
 
 
 
 
 
6d29bb6
9e1f36d
 
6d29bb6
 
 
 
9e1f36d
 
 
 
 
 
6d29bb6
 
9e1f36d
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
---
license: other
license_name: audiolift-component-licenses
license_link: LICENSE.md
base_model:
- FrancisRing/StableAvatar
- wlyu/OpenFaceLift
tags:
- audiolift
- multi-view
- audio-driven-animation
- gaussian-splatting
---

# AudioLift

**Audio-Driven Full-Head Gaussian Avatar via Multi-view Video Generation**

This repository provides the two fine-tuned models used by AudioLift.
The weights are publicly downloadable without a Hugging Face login.

| Model | File | Purpose |
| --- | --- | --- |
| AudioLift-mvdiffusion | `AudioLift-mvdiffusion.pt` | Generate five synchronized video views from a portrait and speech audio. |
| AudioLift-gaussianlifter | `AudioLift-gaussianlifter.pt` | Reconstruct and render a Gaussian head from the five generated views. |

The original selected checkpoint bytes are preserved. Only the public filenames
have changed. File checksums are recorded in [SHA256SUMS](SHA256SUMS).

## Usage

Use the AudioLift inference entrypoints in the
[AudioLift repository](https://github.com/ZhuoyuanJiang/AudioLift).
The entrypoints download missing models to the local `checkpoints/` directory
and verify the fine-tuned checkpoint checksums before loading them.
To download both models and their required base assets before inference, run
`python scripts/download_models.py` from an installed AudioLift environment.
See the [installation guide](https://github.com/ZhuoyuanJiang/AudioLift/blob/main/docs/installation.md)
for environment setup and validation status.

AudioLift-mvdiffusion also requires the Wan and wav2vec2 assets from
[StableAvatar](https://huggingface.co/FrancisRing/StableAvatar). These are downloaded
from the upstream model repository. AudioLift-gaussianlifter requires the camera
and model configuration files supplied with the AudioLift code.

These are inference checkpoints. Training code is not included in the public
AudioLift code release.

## Licenses and acknowledgements

See [model licenses](LICENSE.md) before use. The Gaussian reconstruction weights
retain FaceLift's Adobe Research License for noncommercial research.

Built on [StableAvatar](https://github.com/Francis-Rings/StableAvatar),
[Wan](https://github.com/Wan-Video/Wan2.1) and
[FaceLift](https://github.com/weijielyu/FaceLift).