Upload folder using huggingface_hub
Browse files- .gitattributes +5 -0
- README.md +86 -0
- SoulX-FlashHead-Lite-BF16.gguf +3 -0
- SoulX-FlashHead-Lite-Q8_0.gguf +3 -0
- demo/english.mp4 +3 -0
- demo/english.txt +1 -0
- demo/english.wav +3 -0
- demo/portrait.png +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
SoulX-FlashHead-Lite-BF16.gguf filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
SoulX-FlashHead-Lite-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
demo/english.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
demo/english.wav filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
demo/portrait.png filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,86 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: audio.cpp
|
| 4 |
+
tags:
|
| 5 |
+
- audio.cpp
|
| 6 |
+
- gguf
|
| 7 |
+
- talking-head
|
| 8 |
+
- image-to-video
|
| 9 |
+
- audio-to-video
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# SoulX-FlashHead Lite GGUF
|
| 13 |
+
|
| 14 |
+
Follow or star [audio.cpp](https://github.com/0xShug0/audio.cpp) for the C++ runtime and model integration updates.
|
| 15 |
+
|
| 16 |
+
GGUF packaging of [Soul-AILab/SoulX-FlashHead-1_3B](https://huggingface.co/Soul-AILab/SoulX-FlashHead-1_3B/tree/5fbd2e44ba08ab2deb28bd55cff2c15523e51ba5).
|
| 17 |
+
The model generates streaming talking-portrait video from a reference image and speech audio.
|
| 18 |
+
This repository packages **Lite** only, including its DiT, LTX video VAE and
|
| 19 |
+
[Wav2Vec2 speech encoder](https://huggingface.co/facebook/wav2vec2-base-960h).
|
| 20 |
+
Upstream checkpoint revision: `5fbd2e44ba08ab2deb28bd55cff2c15523e51ba5`.
|
| 21 |
+
Model architecture and weights are by Soul-AILab; the C++ integration is in audio.cpp.
|
| 22 |
+
|
| 23 |
+
> [!IMPORTANT]
|
| 24 |
+
> **cuDNN is strongly recommended for CUDA: about 2× faster warm inference, with lower peak VRAM.**
|
| 25 |
+
> **Prebuilt audio.cpp binaries do not include cuDNN support.** Build from source
|
| 26 |
+
> with cuDNN and `-DGGML_CUDA_CUDNN_CONV3D=ON`, then select
|
| 27 |
+
> `--session-option 'soulx_flashhead.exec_plan={"vae":{"impl":"cudnn"}}'`.
|
| 28 |
+
> The default GGML path remains available without cuDNN, including in prebuilts.
|
| 29 |
+
|
| 30 |
+
## English streaming demo
|
| 31 |
+
|
| 32 |
+
Generated from a single reference portrait and 42.24 seconds of English narration
|
| 33 |
+
using audio.cpp: Lite Q8, 512×512, four denoising steps, 25 FPS, Vulkan backend.
|
| 34 |
+
Audio is processed in 0.96-second chunks with continuous motion between chunks.
|
| 35 |
+
|
| 36 |
+
<video controls src="https://huggingface.co/audio-cpp/SoulX-FlashHead-GGUF/resolve/main/demo/english.mp4"></video>
|
| 37 |
+
|
| 38 |
+
Inputs: [English audio](demo/english.wav), [transcript](demo/english.txt),
|
| 39 |
+
and [portrait](demo/portrait.png). The portrait is from the
|
| 40 |
+
[upstream example](https://github.com/Soul-AILab/SoulX-FlashHead/blob/main/examples/girl.png).
|
| 41 |
+
The English narration was generated with Qwen3-TTS.
|
| 42 |
+
|
| 43 |
+
## audio.cpp CUDA performance
|
| 44 |
+
|
| 45 |
+
Measured on an RTX 5090 with the Q8 package, 512×512 output, four denoising steps,
|
| 46 |
+
and the 42.24-second English demo above (44 streaming chunks). Debug build,
|
| 47 |
+
eight CPU threads; the two modes were run sequentially.
|
| 48 |
+
|
| 49 |
+
| VAE implementation | Warm RTF ↓ | Full-run RTF ↓ | Peak VRAM |
|
| 50 |
+
|---|---:|---:|---:|
|
| 51 |
+
| cuDNN (recommended) | 0.215 | 0.430 | 5,168 MiB |
|
| 52 |
+
| GGML (default) | 0.490 | 0.706 | 9,152 MiB |
|
| 53 |
+
|
| 54 |
+
RTF is generation time divided by video duration; lower is faster. Warm RTF
|
| 55 |
+
covers the full audio-encoder, DiT, VAE, color-correction and RGB-output pipeline,
|
| 56 |
+
excluding the first two chunks and CLI artifact serialization. Full-run RTF also
|
| 57 |
+
includes initialization, cold chunks and CLI artifact serialization. Both exclude
|
| 58 |
+
model loading and MP4 encoding. Peak VRAM is sampled per-process GPU memory.
|
| 59 |
+
The cuDNN VAE uses F16 activations with F32 accumulation.
|
| 60 |
+
|
| 61 |
+
## Weights and usage
|
| 62 |
+
|
| 63 |
+
| File | Components |
|
| 64 |
+
|---|---|
|
| 65 |
+
| `SoulX-FlashHead-Lite-Q8_0.gguf` | Q8 DiT, BF16 LTX VAE, F32 Wav2Vec2, embedded configuration and model contract |
|
| 66 |
+
| `SoulX-FlashHead-Lite-BF16.gguf` | BF16 DiT and LTX VAE, F32 Wav2Vec2, embedded configuration and model contract |
|
| 67 |
+
|
| 68 |
+
```bash
|
| 69 |
+
audiocpp_cli --family soulx_flashhead --task gen \
|
| 70 |
+
--model SoulX-FlashHead-Lite-Q8_0.gguf --backend cuda \
|
| 71 |
+
--audio speech.wav --request-option reference_image_file=portrait.png \
|
| 72 |
+
--mode streaming --out-dir outputs/flashhead \
|
| 73 |
+
--log --log-file outputs/flashhead/run.log
|
| 74 |
+
```
|
| 75 |
+
|
| 76 |
+
For a cuDNN-enabled source build, add the following option to the command above:
|
| 77 |
+
|
| 78 |
+
```bash
|
| 79 |
+
--session-option 'soulx_flashhead.exec_plan={"vae":{"impl":"cudnn"}}'
|
| 80 |
+
```
|
| 81 |
+
|
| 82 |
+
Lite emits 24 RGB24 frames per chunk at 25 FPS. Streaming input is mono 16 kHz;
|
| 83 |
+
offline input is mixed down and resampled when needed. Default output is 512×512
|
| 84 |
+
with four denoising steps. Chunks retain motion continuity from the preceding output.
|
| 85 |
+
|
| 86 |
+
See the [usage and conversion documentation](https://github.com/0xShug0/audio.cpp/blob/main/docs/models/soulx_flashhead.md).
|
SoulX-FlashHead-Lite-BF16.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:989bf75eba5b152dbcaef5a00c50d0a8147bee2d0c3a2f8f2622be8b9e098a10
|
| 3 |
+
size 4269771936
|
SoulX-FlashHead-Lite-Q8_0.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5ba1f2e7026d072e62fe5078037b9c3d8570e0008ffa293fc26730edd170baad
|
| 3 |
+
size 2850631104
|
demo/english.mp4
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f249c56e73ef1863e37bce46249887adcb7aa9bef003cc8464b43f8391a4f833
|
| 3 |
+
size 3800320
|
demo/english.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
At dawn the harbor station opens its tall windows and the first clerk begins a careful report for the day. She notes the weather above the river, the slow cargo boats beyond the bridge, and the market voices arriving from the eastern road. A brass clock marks each quarter hour while porters stack wooden crates, bakers carry warm bread across the square, and a violinist practices the same bright phrase under the stone archway. By midmorning the keeper of the lighthouse sends a message about shifting currents, the museum guide unlocks a cabinet of maps, and a teacher leads a quiet line of students toward the ferry.
|
demo/english.wav
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:abf09d87fc46559436db14c1d22d1e54189b3506a691759e7ac0278f87219a5b
|
| 3 |
+
size 1351758
|
demo/portrait.png
ADDED
|
Git LFS Details
|