| --- |
| license: apache-2.0 |
| pipeline_tag: text-to-speech |
| language: |
| - zh |
| - en |
| tags: |
| - speech |
| - singing-voice-synthesis |
| - continuous-latent |
| --- |
| |
| <div align="center"> |
| <h1 style="white-space: nowrap; text-align: center; font-size: clamp(28px, 8vw, 32px); font-weight: 600; line-height: 1.25;"><span style="position: relative; display: inline-block; line-height: inherit;"><img src="https://huggingface.co/ASLP-lab/SemBridge/resolve/main/asset/sembridge-icon-transparent.png" alt="" width="64" height="27" style="position: absolute; right: 100%; top: 50%; transform: translateY(-50%); width: 2em; height: auto; max-width: none; margin: 0 0.25em 0 0;">SemBridge</span></h1> |
| <p><strong>Semantic Token Anchoring for Continuous-Latent Autoregressive Speech Generation</strong></p> |
| <p><em>Speech. Singing. One continuous voice.</em></p> |
| <p> |
| <a href="https://arxiv.org/pdf/2608.07462"><img src="https://img.shields.io/badge/arXiv-2608.07462-b31b1b.svg" alt="Paper"></a> |
| <a href="https://tiamojames.github.io/SemBridge_Demo/"><img src="https://img.shields.io/badge/Listen-Demo-15803d.svg" alt="Audio demos"></a> |
| <a href="https://huggingface.co/ASLP-lab/SemBridge"><img src="https://img.shields.io/badge/Hugging_Face-Model-f2c744.svg" alt="Model"></a> |
| <a href="https://github.com/ASLP-lab/SemBridge"><img src="https://img.shields.io/badge/GitHub-Code-181717.svg?logo=github" alt="Code"></a> |
| <a href="https://github.com/ASLP-lab/SemBridge/blob/main/LICENSE"><img src="https://img.shields.io/badge/License-Apache_2.0-2563eb.svg" alt="Code license"></a> |
| </p> |
| <p> |
| <a href="#introduction">Introduction</a> · |
| <a href="#roadmap">Roadmap</a> · |
| <a href="#installation">Installation</a> · |
| <a href="#inference">Inference</a> · |
| <a href="#community">Community</a> · |
| <a href="#citation">Citation</a> |
| </p> |
| </div> |
| |
| <a id="introduction"></a> |
|
|
| ## 📖 Introduction |
|
|
| SemBridge bridges discrete semantic supervision and continuous acoustic generation. |
| A shared semantic tokenizer anchors both acoustic latents and autoregressive language-model |
| states, improving linguistic fidelity while retaining continuous-latent generation. |
| The unified model supports **text-to-speech**, **note-conditioned singing**, and |
| **mixed speech and singing**. |
|
|
| <div align="center"> |
| <img src="https://huggingface.co/ASLP-lab/SemBridge/resolve/main/asset/sembridge_model.png" alt="SemBridge architecture: semantic anchoring for continuous-latent autoregressive generation" width="100%"> |
| </div> |
|
|
| <a id="news"></a> |
|
|
| ## 📣 News |
|
|
| - **[2026-10]**: Released the [inference code](https://github.com/ASLP-lab/SemBridge) and [model checkpoints](https://huggingface.co/ASLP-lab/SemBridge). |
| - **[2026-08]**: Released the [arXiv paper](https://arxiv.org/pdf/2608.07462). |
|
|
| <a id="roadmap"></a> |
|
|
| ## 🗺️ Roadmap |
|
|
| - [x] [arXiv paper](https://arxiv.org/abs/2608.07462) and [code repository](https://github.com/ASLP-lab/SemBridge) |
| - [x] [TTS, SVS, and unified inference code](#inference) |
| - [x] [Model checkpoints](https://huggingface.co/ASLP-lab/SemBridge) |
| - [ ] Fine-tuning code |
|
|
| <a id="installation"></a> |
|
|
| ## ⚙️ Installation |
|
|
| Use **Python 3.10**. |
|
|
| ```bash |
| conda create -n sembridge python=3.10 -y |
| conda activate sembridge |
| |
| git clone https://github.com/ASLP-lab/SemBridge.git |
| cd SemBridge |
| |
| # CUDA 12.4 example; choose a build compatible with your NVIDIA driver. |
| python -m pip install torch==2.6.0 torchaudio==2.6.0 \ |
| --index-url https://download.pytorch.org/whl/cu124 |
| python -m pip install -e . |
| ``` |
|
|
| <a id="inference"></a> |
|
|
| ## 🚀 Inference |
|
|
| ```python |
| from sembridge import SemBridge |
| from sembridge.requests import load_request |
| |
| # Automatically download and load the checkpoint. |
| model = SemBridge.from_pretrained( |
| "ASLP-lab/SemBridge", device="cuda:0", dtype="bfloat16" |
| ) |
| model.generate( |
| output="outputs/singing.wav", |
| **load_request("examples/svs/zh_r001.json"), |
| ) |
| ``` |
|
|
| | Task | Input | Default decoder | |
| | :--- | :--- | :--- | |
| | **TTS** | Text | Native causal | |
| | **SVS** | Lyrics, MIDI pitches, note durations | BigVGAN-Large | |
| | **Unified** | Ordered speech and singing segments | BigVGAN-Large | |
|
|
| <a id="examples"></a> |
|
|
| ### 🎧 Examples |
|
|
| The repository includes **10 complete examples from our [Demo page](https://tiamojames.github.io/SemBridge_Demo/)**: |
| 4 TTS, 4 SVS, and 2 Unified cases, with reference audio and prepared requests. |
| SVS and Unified include **aligned lyrics, pitches, durations, and note-continuation labels**; |
| you do not need to prepare MIDI files or write note JSON to try them. |
|
|
| ```bash |
| # Browse the available examples without loading a model |
| bash scripts/infer_examples.sh --list |
| |
| # Run all four singing examples |
| CUDA_VISIBLE_DEVICES=0 bash scripts/infer_examples.sh --task svs |
| |
| # Run one speech-singing-speech example |
| CUDA_VISIBLE_DEVICES=0 bash scripts/infer_examples.sh --example unified_zh_novel_wukong |
| |
| # Run all 10 examples; the model is loaded only once |
| CUDA_VISIBLE_DEVICES=0 bash scripts/infer_examples.sh |
| ``` |
|
|
| Audio is saved to `outputs/examples/<example_id>.wav`. Set `MODEL=/path/to/SemBridge` |
| to use local weights. See the **[example catalog](https://github.com/ASLP-lab/SemBridge/blob/main/examples/README.md)** for the |
| complete list, reference recordings, and guidance on using your own songs. |
|
|
| <a id="single-sample-inference"></a> |
|
|
| ### 🎙️ Single-Sample Inference |
|
|
| ```bash |
| # Text-to-speech |
| sembridge-infer --model ASLP-lab/SemBridge --task tts \ |
| --input examples/tts/zh_r001.json --output outputs/tts.wav \ |
| --device cuda:0 --dtype bfloat16 |
| |
| # Singing voice synthesis |
| sembridge-infer --model ASLP-lab/SemBridge --task svs \ |
| --input examples/svs/zh_r001.json --output outputs/svs.wav \ |
| --device cuda:0 --dtype bfloat16 |
| |
| # Mixed speech and singing |
| sembridge-infer --model ASLP-lab/SemBridge --task unified \ |
| --input examples/unified/zh_novel_wukong.json --output outputs/unified.wav \ |
| --device cuda:0 --dtype bfloat16 |
| ``` |
|
|
| Outputs are **mono, 44.1 kHz WAV**. `--decoder auto` uses the defaults above; |
| `--decoder native` or `--decoder bigvgan` selects a decoder explicitly for any task. |
| Prepared requests include per-example generation settings, which both entry points |
| honor. Explicit CLI options override those settings. Without request-specific limits, |
| `--max-patches 500` caps generation at 40 seconds; generation can stop earlier. |
|
|
| Add `prompt_audio` and its matching `prompt_text` for reference-conditioned generation. |
| Reference audio is automatically downmixed and resampled to 44.1 kHz. |
| For SVS, each note contains `text`, MIDI `pitch`, and `duration` **in seconds**. |
|
|
| See the **[inference guide](https://github.com/ASLP-lab/SemBridge/blob/main/docs/INFERENCE.md)** for reference voices, singing and |
| mixed-generation examples, sampling controls, and offline loading. |
|
|
| <a id="community"></a> |
|
|
| ## 🤝 Community |
|
|
| - **Windows / ComfyUI:** [ComfyUI-SemBridge](https://github.com/hiroki-abe-58/ComfyUI-SemBridge), an unofficial, community-maintained integration. |
|
|
| <a id="acknowledgements"></a> |
|
|
| ## 🙏 Acknowledgements |
|
|
| We thank the authors of the following projects for their open research and resources: |
|
|
| - **Speech generation:** [VoxCPM](https://github.com/OpenBMB/VoxCPM) and [DiTAR](https://arxiv.org/abs/2502.03930). |
| - **Audio codecs:** [DAC](https://github.com/descriptinc/descript-audio-codec), [SAC](https://github.com/Soul-AILab/SAC), and [BigVGAN](https://github.com/NVIDIA/BigVGAN). |
|
|
| <a id="citation"></a> |
|
|
| ## 📚 Citation |
|
|
| If you find SemBridge useful in your research, please cite: |
|
|
| ```bibtex |
| @article{xie2026sembridge, |
| title = {SemBridge: Semantic Token Anchoring for Continuous-Latent Autoregressive Speech Generation}, |
| author = {Hanke Xie and Haopeng Lin and Jiale Qian and Dake Guo and Yuepeng Jiang and Zhichao Wang and Wenxiao Cao and Jingbin Hu and Guobin Ma and Wenhao Li and Huakang Chen and Chengyou Wang and Ming Tao and Zhonghua Fu and Lei Xie and Xinsheng Wang}, |
| journal = {arXiv preprint arXiv:2608.07462}, |
| year = {2026}, |
| eprint = {2608.07462}, |
| archivePrefix = {arXiv}, |
| primaryClass = {eess.AS}, |
| url = {https://arxiv.org/abs/2608.07462} |
| } |
| ``` |
|
|
| <a id="license"></a> |
|
|
| ## 📄 License |
|
|
| SemBridge code is released under Apache-2.0; upstream portions retain their |
| original licenses. See [LICENSE](https://github.com/ASLP-lab/SemBridge/blob/main/LICENSE) for terms and third-party notices. |
| Model weights and third-party resources are subject to their respective licenses. |
| Use reference recordings only with permission, and disclose synthetic audio where appropriate. |
|
|
