README.md

---
license: apache-2.0
base_model:
  - FunAudioLLM/Fun-CosyVoice3-0.5B-2512
pipeline_tag: text-to-speech
language:
  - zh
  - en
  - ja
  - ko
  - de
  - es
  - fr
  - it
  - ru
tags:
  - cosyvoice
  - gguf
  - streaming-tts
  - text-to-speech
  - cpp
  - knowledge-distillation
---

# cosyvoice3_stream.cpp

Runtime assets for
[cosyvoice3_yq_cpp](https://gitee.com/yang-qi1222/cosyvoice3_yq_cpp), an
inference-only CosyVoice3 C++/GGML streaming runtime.

This repository contains one project-compressed GGUF model, the two ONNX models
needed to extract a voice prompt from reference audio, and ten small precomputed
prompt features for immediate testing. Source code is distributed separately on
Gitee.

## Files

| Path | Bytes | Purpose |
| --- | ---: | --- |
| `models/student12_mlp_inner_q8_flow4_hift_f16.gguf` | 1,637,038,400 | Student12 LLM + Flow4 + HiFT runtime model |
| `frontend/speech_tokenizer_v3.onnx` | 969,451,503 | Speech-token extraction from reference audio |
| `frontend/campplus.onnx` | 28,303,423 | Speaker embedding extraction |
| `prompts/control10_01.gguf` ... `control10_10.gguf` | 51,296-188,576 each | Ready-to-use synthetic evaluation voices |

Every binary is pinned in `SHA256SUMS`. The source repository also ships a
machine-readable manifest used by its downloader.

## Model Lineage

The base model is
[FunAudioLLM/Fun-CosyVoice3-0.5B-2512](https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512).
The published GGUF applies the following project-specific deployment changes:

- a 12-layer student speech LLM obtained through structured layer pruning and
  distillation;
- Flow inference distilled from 10 steps to 4 steps;
- GGUF conversion with inner LLM MLP tensors quantized to Q8 and Flow/HiFT kept
  in F16.

The LLM compression workflow was informed by
[SPADE](https://arxiv.org/abs/2509.20802), with additional material on the
[SPADE project page](https://mm.kaist.ac.kr/projects/SPADE/). This is an
independent CosyVoice3 engineering adaptation, not an official SPADE model or a
claim that the paper's reported metrics transfer to this model.

## Prompt Features

`control10_01` through `control10_10` are features extracted from synthetic
teacher-model outputs used in this project's five-male/five-female Control10
evaluation. No source WAV files are distributed, and these prompts are not
presented as the voice of any real person.

Use a prompt directly for the lowest startup overhead. To clone a new authorized
reference voice, use `speech_tokenizer_v3.onnx` and `campplus.onnx` with the
audio-enabled runtime to create a new `prompt_speech.gguf`.

## Download And Run

```bash
git clone https://gitee.com/yang-qi1222/cosyvoice3_yq_cpp.git
cd cosyvoice3_yq_cpp
python3 -m pip install -r requirements-tools.txt
python3 scripts/download_assets.py \
  --manifest manifests/assets.example.json \
  --asset-root assets
```

Build and start the feature-only CPU server:

```bash
scripts/build_runtime.sh \
  --mode feature \
  --backend cpu \
  --build-dir build/feature-cpu

scripts/start_server.sh \
  --build-dir build/feature-cpu \
  --backend cpu \
  --model assets/models/student12_mlp_inner_q8_flow4_hift_f16.gguf \
  --prompt-speech assets/prompts/control10_01.gguf \
  --voice control10_01 \
  --port 8080
```

CUDA architecture, thread count, chunk size, and FlashAttention settings must be
selected and validated for the deployment machine. See the source repository for
the audio-input build and complete server examples.

## Measured Baseline

The accepted project baseline covers only an NVIDIA RTX 5880 Ada Generation GPU,
one serial request at a time, 16 host threads, `chunk_tokens=75`, LLM
FlashAttention disabled, and Flow FlashAttention enabled.

| Metric | Project result |
| --- | ---: |
| Control10 median RTF | 0.104273 |
| First speech token | about 9.1 ms |
| Median first PCM | about 223.2 ms |
| Natural EOS | 10/10 |
| Resident soak | 100/100 |
| Process RSS | about 1,181 MiB |
| Device-level GPU memory used | about 2,461 MiB |

These measurements must not be extrapolated to RTX 4060, Jetson AGX, other GPU
architectures, concurrent service, or long-context synthesis.

## Limitations

- This repository distributes inference assets, not training checkpoints or
  training code.
- The reference-audio path also requires an ONNX Runtime C/C++ SDK at build time;
  the two ONNX model files do not replace that dependency.
- Voice cloning must only be performed with appropriate speaker consent and data
  rights.
- The project is community maintained and is not affiliated with or endorsed by
  the official CosyVoice or SPADE teams.

## License And Acknowledgements

Model assets are distributed under Apache-2.0. The C++ source repository uses the
MIT license; source and model licenses are separate.

This work depends on and thanks:

- [QwenAudio/CosyVoice](https://github.com/QwenAudio/CosyVoice) and the
  [CosyVoice3 paper](https://arxiv.org/abs/2505.17589);
- [Lourdle/cosyvoice.cpp](https://github.com/Lourdle/cosyvoice.cpp);
- [SPADE](https://arxiv.org/abs/2509.20802);
- [ggml](https://github.com/ggml-org/ggml),
  [llama.cpp](https://github.com/ggml-org/llama.cpp), and
  [ONNX Runtime](https://github.com/microsoft/onnxruntime).

Please cite the original CosyVoice3 and SPADE papers when this model is used in
research comparisons.