cosyvoice3_stream.cpp

Runtime assets for cosyvoice3_yq_cpp, an inference-only CosyVoice3 C++/GGML streaming runtime.

This repository contains one project-compressed GGUF model, the two ONNX models needed to extract a voice prompt from reference audio, and ten small precomputed prompt features for immediate testing. Source code is distributed separately on Gitee.

Files

Path	Bytes	Purpose
`models/student12_mlp_inner_q8_flow4_hift_f16.gguf`	1,637,038,400	Student12 LLM + Flow4 + HiFT runtime model
`frontend/speech_tokenizer_v3.onnx`	969,451,503	Speech-token extraction from reference audio
`frontend/campplus.onnx`	28,303,423	Speaker embedding extraction
`prompts/control10_01.gguf` ... `control10_10.gguf`	51,296-188,576 each	Ready-to-use synthetic evaluation voices

Every binary is pinned in SHA256SUMS. The source repository also ships a machine-readable manifest used by its downloader.

Model Lineage

The base model is FunAudioLLM/Fun-CosyVoice3-0.5B-2512. The published GGUF applies the following project-specific deployment changes:

a 12-layer student speech LLM obtained through structured layer pruning and distillation;
Flow inference distilled from 10 steps to 4 steps;
GGUF conversion with inner LLM MLP tensors quantized to Q8 and Flow/HiFT kept in F16.

The LLM compression workflow was informed by SPADE, with additional material on the SPADE project page. This is an independent CosyVoice3 engineering adaptation, not an official SPADE model or a claim that the paper's reported metrics transfer to this model.

Prompt Features

control10_01 through control10_10 are features extracted from synthetic teacher-model outputs used in this project's five-male/five-female Control10 evaluation. No source WAV files are distributed, and these prompts are not presented as the voice of any real person.

Use a prompt directly for the lowest startup overhead. To clone a new authorized reference voice, use speech_tokenizer_v3.onnx and campplus.onnx with the audio-enabled runtime to create a new prompt_speech.gguf.

Download And Run

git clone https://gitee.com/yang-qi1222/cosyvoice3_yq_cpp.git
cd cosyvoice3_yq_cpp
python3 -m pip install -r requirements-tools.txt
python3 scripts/download_assets.py \
  --manifest manifests/assets.example.json \
  --asset-root assets

Build and start the feature-only CPU server:

scripts/build_runtime.sh \
  --mode feature \
  --backend cpu \
  --build-dir build/feature-cpu

scripts/start_server.sh \
  --build-dir build/feature-cpu \
  --backend cpu \
  --model assets/models/student12_mlp_inner_q8_flow4_hift_f16.gguf \
  --prompt-speech assets/prompts/control10_01.gguf \
  --voice control10_01 \
  --port 8080

CUDA architecture, thread count, chunk size, and FlashAttention settings must be selected and validated for the deployment machine. See the source repository for the audio-input build and complete server examples.

Measured Baseline

The accepted project baseline covers only an NVIDIA RTX 5880 Ada Generation GPU, one serial request at a time, 16 host threads, chunk_tokens=75, LLM FlashAttention disabled, and Flow FlashAttention enabled.

Metric	Project result
Control10 median RTF	0.104273
First speech token	about 9.1 ms
Median first PCM	about 223.2 ms
Natural EOS	10/10
Resident soak	100/100
Process RSS	about 1,181 MiB
Device-level GPU memory used	about 2,461 MiB

These measurements must not be extrapolated to RTX 4060, Jetson AGX, other GPU architectures, concurrent service, or long-context synthesis.

Limitations

This repository distributes inference assets, not training checkpoints or training code.
The reference-audio path also requires an ONNX Runtime C/C++ SDK at build time; the two ONNX model files do not replace that dependency.
Voice cloning must only be performed with appropriate speaker consent and data rights.
The project is community maintained and is not affiliated with or endorsed by the official CosyVoice or SPADE teams.