audio-cpp commited on
Commit
ffc8c4a
·
verified ·
1 Parent(s): dfca0ff

Upload folder using huggingface_hub

Browse files
.gitattributes CHANGED
@@ -33,3 +33,8 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ SoulX-FlashHead-Lite-BF16.gguf filter=lfs diff=lfs merge=lfs -text
37
+ SoulX-FlashHead-Lite-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
38
+ demo/english.mp4 filter=lfs diff=lfs merge=lfs -text
39
+ demo/english.wav filter=lfs diff=lfs merge=lfs -text
40
+ demo/portrait.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: audio.cpp
4
+ tags:
5
+ - audio.cpp
6
+ - gguf
7
+ - talking-head
8
+ - image-to-video
9
+ - audio-to-video
10
+ ---
11
+
12
+ # SoulX-FlashHead Lite GGUF
13
+
14
+ Follow or star [audio.cpp](https://github.com/0xShug0/audio.cpp) for the C++ runtime and model integration updates.
15
+
16
+ GGUF packaging of [Soul-AILab/SoulX-FlashHead-1_3B](https://huggingface.co/Soul-AILab/SoulX-FlashHead-1_3B/tree/5fbd2e44ba08ab2deb28bd55cff2c15523e51ba5).
17
+ The model generates streaming talking-portrait video from a reference image and speech audio.
18
+ This repository packages **Lite** only, including its DiT, LTX video VAE and
19
+ [Wav2Vec2 speech encoder](https://huggingface.co/facebook/wav2vec2-base-960h).
20
+ Upstream checkpoint revision: `5fbd2e44ba08ab2deb28bd55cff2c15523e51ba5`.
21
+ Model architecture and weights are by Soul-AILab; the C++ integration is in audio.cpp.
22
+
23
+ > [!IMPORTANT]
24
+ > **cuDNN is strongly recommended for CUDA: about 2× faster warm inference, with lower peak VRAM.**
25
+ > **Prebuilt audio.cpp binaries do not include cuDNN support.** Build from source
26
+ > with cuDNN and `-DGGML_CUDA_CUDNN_CONV3D=ON`, then select
27
+ > `--session-option 'soulx_flashhead.exec_plan={"vae":{"impl":"cudnn"}}'`.
28
+ > The default GGML path remains available without cuDNN, including in prebuilts.
29
+
30
+ ## English streaming demo
31
+
32
+ Generated from a single reference portrait and 42.24 seconds of English narration
33
+ using audio.cpp: Lite Q8, 512×512, four denoising steps, 25 FPS, Vulkan backend.
34
+ Audio is processed in 0.96-second chunks with continuous motion between chunks.
35
+
36
+ <video controls src="https://huggingface.co/audio-cpp/SoulX-FlashHead-GGUF/resolve/main/demo/english.mp4"></video>
37
+
38
+ Inputs: [English audio](demo/english.wav), [transcript](demo/english.txt),
39
+ and [portrait](demo/portrait.png). The portrait is from the
40
+ [upstream example](https://github.com/Soul-AILab/SoulX-FlashHead/blob/main/examples/girl.png).
41
+ The English narration was generated with Qwen3-TTS.
42
+
43
+ ## audio.cpp CUDA performance
44
+
45
+ Measured on an RTX 5090 with the Q8 package, 512×512 output, four denoising steps,
46
+ and the 42.24-second English demo above (44 streaming chunks). Debug build,
47
+ eight CPU threads; the two modes were run sequentially.
48
+
49
+ | VAE implementation | Warm RTF ↓ | Full-run RTF ↓ | Peak VRAM |
50
+ |---|---:|---:|---:|
51
+ | cuDNN (recommended) | 0.215 | 0.430 | 5,168 MiB |
52
+ | GGML (default) | 0.490 | 0.706 | 9,152 MiB |
53
+
54
+ RTF is generation time divided by video duration; lower is faster. Warm RTF
55
+ covers the full audio-encoder, DiT, VAE, color-correction and RGB-output pipeline,
56
+ excluding the first two chunks and CLI artifact serialization. Full-run RTF also
57
+ includes initialization, cold chunks and CLI artifact serialization. Both exclude
58
+ model loading and MP4 encoding. Peak VRAM is sampled per-process GPU memory.
59
+ The cuDNN VAE uses F16 activations with F32 accumulation.
60
+
61
+ ## Weights and usage
62
+
63
+ | File | Components |
64
+ |---|---|
65
+ | `SoulX-FlashHead-Lite-Q8_0.gguf` | Q8 DiT, BF16 LTX VAE, F32 Wav2Vec2, embedded configuration and model contract |
66
+ | `SoulX-FlashHead-Lite-BF16.gguf` | BF16 DiT and LTX VAE, F32 Wav2Vec2, embedded configuration and model contract |
67
+
68
+ ```bash
69
+ audiocpp_cli --family soulx_flashhead --task gen \
70
+ --model SoulX-FlashHead-Lite-Q8_0.gguf --backend cuda \
71
+ --audio speech.wav --request-option reference_image_file=portrait.png \
72
+ --mode streaming --out-dir outputs/flashhead \
73
+ --log --log-file outputs/flashhead/run.log
74
+ ```
75
+
76
+ For a cuDNN-enabled source build, add the following option to the command above:
77
+
78
+ ```bash
79
+ --session-option 'soulx_flashhead.exec_plan={"vae":{"impl":"cudnn"}}'
80
+ ```
81
+
82
+ Lite emits 24 RGB24 frames per chunk at 25 FPS. Streaming input is mono 16 kHz;
83
+ offline input is mixed down and resampled when needed. Default output is 512×512
84
+ with four denoising steps. Chunks retain motion continuity from the preceding output.
85
+
86
+ See the [usage and conversion documentation](https://github.com/0xShug0/audio.cpp/blob/main/docs/models/soulx_flashhead.md).
SoulX-FlashHead-Lite-BF16.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:989bf75eba5b152dbcaef5a00c50d0a8147bee2d0c3a2f8f2622be8b9e098a10
3
+ size 4269771936
SoulX-FlashHead-Lite-Q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5ba1f2e7026d072e62fe5078037b9c3d8570e0008ffa293fc26730edd170baad
3
+ size 2850631104
demo/english.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f249c56e73ef1863e37bce46249887adcb7aa9bef003cc8464b43f8391a4f833
3
+ size 3800320
demo/english.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ At dawn the harbor station opens its tall windows and the first clerk begins a careful report for the day. She notes the weather above the river, the slow cargo boats beyond the bridge, and the market voices arriving from the eastern road. A brass clock marks each quarter hour while porters stack wooden crates, bakers carry warm bread across the square, and a violinist practices the same bright phrase under the stone archway. By midmorning the keeper of the lighthouse sends a message about shifting currents, the museum guide unlocks a cabinet of maps, and a teacher leads a quiet line of students toward the ferry.
demo/english.wav ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:abf09d87fc46559436db14c1d22d1e54189b3506a691759e7ac0278f87219a5b
3
+ size 1351758
demo/portrait.png ADDED

Git LFS Details

  • SHA256: 384abd202f7dae45fdbbab3b6a7777cd7c8a64625fb19217a45d056f2162c624
  • Pointer size: 131 Bytes
  • Size of remote file: 357 kB