XProger/Qwen3-TTS-12Hz-1.7B-Base-TalkerMLP-L2-L23-W4A16-RestBF16-G64-C512

Qwen3-TTS-12Hz-1.7B-Base TalkerMLP-L2-L23 W4A16 RestBF16

Experimental partially quantized version of Qwen/Qwen3-TTS-12Hz-1.7B-Base.

This model was created for vLLM-Omni inference and uses compressed-tensors packed quantized weights.

Important: this is not a fully W4A16 model.

Only the middle MLP layers of the talker decoder were quantized. The rest of the model remains in BF16 / original precision.

What is quantized

Quantized to W4A16:

• talker.model.layers.2..23.mlp.gate_proj
• talker.model.layers.2..23.mlp.up_proj
• talker.model.layers.2..23.mlp.down_proj

Left in BF16 / original precision:

• all attention layers
• decoder layers 0-1
• decoder layers 24-27
• text projection
• codec head
• code predictor
• speaker encoder
• speech tokenizer / code2wav
• embeddings
• normalization layers
• all non-talker auxiliary modules

Why partial quantization

Qwen3-TTS is very sensitive to quantization.

During testing, aggressive W4 quantization of too many layers degraded speech quality badly. The model could still produce WAV files, but the audio became meaningless or unstable.

The stable configuration found during testing was:

• quantize only the middle MLP layers
• keep all attention layers in BF16
• keep the first decoder layers in BF16
• keep the final decoder layers in BF16
• keep all codec/audio-related modules in BF16

This keeps most of the speech quality while reducing checkpoint size and VRAM usage.

Quantization details

• Base model: Qwen/Qwen3-TTS-12Hz-1.7B-Base
• Quantized area: Talker MLP layers 2-23
• Weight precision: W4
• Activation precision: A16 / BF16 runtime activations
• Group size: 64
• Calibration samples: 512
• Format: compressed-tensors
• Packed format: pack-quantized
• Runtime tested with: vLLM-Omni
• Main tested language: Russian

Size comparison

Model	Directory size	model.safetensors
BF16 base	4.3G	3.6G
Partial W4A16	3.1G	2.5G

Runtime summary

The main benefit of this model is lower VRAM usage and a smaller checkpoint.

Speed is roughly similar to BF16 on the tested setup.

Observed on the test server:

Model	Talker load VRAM	Available KV cache	TTFA after warmup
BF16	~3.63 GiB	~62k tokens	~37-44 ms
Partial W4A16	~2.49 GiB	~72k tokens	~40-47 ms

Inference with vLLM-Omni

Example server launch:

cd ~/qw3tts
source .venv-vllm/bin/activate

export MODEL="/path/to/Qwen3-TTS-12Hz-1.7B-Base-TalkerMLP-L2-L23-W4A16-RestBF16-G64-C512"
export PYTHONPATH="$PWD/vllm-omni:$PYTHONPATH"

vllm-omni serve "$MODEL" \
  --deploy-config "$PWD/vllm-omni/vllm_omni/deploy/qwen3_tts.yaml" \
  --omni \
  --port 8091 \
  --trust-remote-code \
  --enforce-eager \
  --generation-config vllm \
  --allowed-local-media-path "$PWD" \
  --stage-overrides '{"0":{"gpu_memory_utilization":0.35},"1":{"gpu_memory_utilization":0.35}}'

Example request:

curl -sS \
  -o out.wav \
  -H "Content-Type: application/json" \
  --data-binary @payload.json \
  http://127.0.0.1:8091/v1/audio/speech

Example payload.json:

{
  "model": "/path/to/Qwen3-TTS-12Hz-1.7B-Base-TalkerMLP-L2-L23-W4A16-RestBF16-G64-C512",
  "input": "Привет. Это короткий тест.",
  "task_type": "Base",
  "language": "Russian",
  "response_format": "wav",
  "ref_audio": "file:///path/to/ref.wav",
  "ref_text": "Reference audio transcript here.",
  "max_new_tokens": 96,
  "seed": 42,
  "temperature": 0.7,
  "top_p": 0.9,
  "repetition_penalty": 1.15
}

Important: for this vLLM-Omni speech endpoint, generation length is controlled by max_new_tokens, not max_tokens.

Quality note

The Russian voice cloning test samples sounded good in manual listening tests.

This is still an experimental quantized model. Please validate quality on your own voices, prompts, languages and inference settings.

Known limitations

• This is not a full W4A16 quantization.
• Only middle talker MLP layers are quantized.
• Requires vLLM-Omni support for Qwen3-TTS.
• Requires compressed-tensors support in the runtime.
• Tested mainly with Russian voice cloning.
• Quality may vary for other voices, languages and long-form generation.
• The model may be sensitive to sampling settings and max_new_tokens.

Naming

The model name describes exactly what was quantized:

TalkerMLP-L2-L23-W4A16-RestBF16-G64-C512

Meaning:

• TalkerMLP - only the talker MLP blocks were quantized
• L2-L23 - only decoder layers 2 through 23 were quantized
• W4A16 - quantized weights are 4-bit, activations remain 16-bit
• RestBF16 - the rest of the model remains BF16 / original precision
• G64 - group size 64
• C512 - 512 calibration samples

Safety and usage

Use reference voices only with proper rights and permission.

This model is intended for research and experimentation with efficient TTS inference.