ShinePixelOrg/VoxCPM2-NVFP4

VoxCPM2 NVFP4 (GPTQ)

Weight-only NVFP4 quantization of openbmb/VoxCPM2 revision 32279effe8c19989596f05d353d1447f51d9e915, produced with llm-compressor GPTQ and saved in compressed-tensors nvfp4-pack-quantized format.

These weights are a derivative of the official OpenBMB checkpoint. Use is governed by the same Apache-2.0 license as the original weights. Upstream model card: https://huggingface.co/openbmb/VoxCPM2. Project: https://github.com/OpenBMB/VoxCPM.

Only the two language-model backbones are quantized, and only their weights. Activations stay BF16. Attention v_proj and FFN down_proj stay BF16 in both language models, together with LocDiT, LocEnc, FSQ, the projection heads, the stop head and AudioVAE. This is the second revision of this repository: the first revision quantized every language-model matrix, including v_proj and down_proj, with plain min/max rounding.

Size

The comparison is the weight payload: model.safetensors plus audiovae.pth. The AudioVAE file is byte-identical in both trees.

	BF16 source	This checkpoint	Delta
model.safetensors	4.266 GiB	2.623 GiB	−1.643 GiB, −38.5%
audiovae.pth	0.351 GiB (359.5 MiB)	0.351 GiB (359.5 MiB)	0
Weight total	4.617 GiB	2.974 GiB	−1.643 GiB, 64.4% of the source

vLLM-Omni 0.30.0 reported 3.21 GiB of weight memory for this checkpoint and 4.86 GiB for the BF16 checkpoint. The engine log line to look for is ATS VoxCPM2 quant: W4A4=0 A16=180 int4=0 (see Loading).

What was quantized

Precision	Modules
NVFP4, weights only	q_proj, k_proj, o_proj, gate_proj, up_proj in BaseLM (28 layers) and ResidualLM (8 layers). 180 modules
BF16	v_proj and down_proj in both language models, token embeddings, norms, LocEnc, LocDiT, FSQ, projection heads, stop head
FP32	AudioVAE (audiovae.pth, unchanged)

Attention V and the FFN down projection are the two tensor families that community VoxCPM2 quantizations keep at higher precision: they carry the content-bearing part of the signal, and 4-bit rounding of them is what breaks long sentences. Keeping them dense is the main change from the first revision of this repository.

NVFP4 settings: 4-bit float weights, symmetric, tensor-group quantization, group size 16, per-group scales in float8_e4m3fn, plus one weight_global_scale per matrix. There is no input_global_scale, so activations are not quantized.

Method

GPTQ, block size 128, dampening 0.01, static activation ordering, Hessians offloaded to CPU.

Calibration used 48 real VoxCPM2 synthesizations (216.8 seconds of 48 kHz audio) produced by the BF16 model at temperature 0, each paired with its own transcript, covering Chinese and English statements, questions, digit strings and long passages. For every sample the waveform was encoded by the model's own AudioVAE into latent patches, embedded by LocEnc, and passed through both language models with the same text-mask / audio-mask interleaving the server uses at inference time. No random-noise calibration was used.

One honest note on the numbers: the mean relative weight error against BF16 is 14.1% for this checkpoint and 9.5% for plain min/max rounding. That is expected. GPTQ minimizes output error WX - ŴX under real activations, which accepts larger individual weight changes. Audio below is what it optimizes, and audio is better.

Audio comparison

Measured 2026-11-03. Both engines used temperature 0 and seed 42, zero-shot, no reference audio, same GPU, one request at a time. Every file is 48 kHz, 16-bit, mono, and neither side clipped. The engine is deterministic: the same BF16 model run twice scores 0.0% duration error, 1.00 contour correlation and 0.00 dB MCD, so every number below is quantization, not run-to-run noise.

MCD is mel-cepstral distortion in dB after aligning the candidate cepstral track onto the reference frame grid. Pitch-contour correlation is the Pearson correlation of 16 bins of voiced F0. Spectral similarity is the cosine of the mean log-mel spectrum.

Sentence	Duration, BF16 → NVFP4	Mean F0	F0-contour correlation	Spectral similarity	MCD
Short Chinese	3.36s → 3.36s (0%)	148 → 149 Hz	0.96	0.999	12.4 dB
Chinese question	4.32s → 4.00s (−7%)	141 → 134 Hz	0.85	0.997	20.2 dB
Numbers and time	5.60s → 5.76s (+3%)	145 → 132 Hz	0.71	0.999	16.7 dB
Long Chinese	17.44s → 17.92s (+3%)	144 → 166 Hz	−0.17	0.999	21.7 dB
Chinese story	34.24s → 35.04s (+2%)	211 → 238 Hz	−0.38	0.990	20.2 dB
English paragraph	17.12s → 17.12s (0%)	119 → 121 Hz	0.70	0.999	19.7 dB
Chinese news	15.84s → 16.96s (+7%)	144 → 131 Hz	−0.00	0.996	23.6 dB
Dialogue	17.76s → 16.96s (−5%)	268 → 273 Hz	0.84	1.000	16.1 dB
Mean	3.4% absolute drift	—	0.44	0.997	18.8 dB

Against the previous revision of this repository, which quantized all 252 language-model matrices with min/max rounding, on the same eight sentences:

	Previous revision (252 matrices, RTN)	This checkpoint (180 matrices, GPTQ)
Mean absolute duration drift	7.7%	3.4%
Mean F0-contour correlation	0.29	0.44
Mean spectral similarity	0.9952	0.9973
Mean MCD	17.96 dB	18.84 dB
Language-model weight memory	2.58 GiB	3.21 GiB

Short sentences are close to the source. Intonation on long passages is where quantization still shows: the two longest utterances keep their length and their timbre (0.990 and 0.999) but do not follow the BF16 pitch contour.

Stop behaviour and repetition

Repetition and failure to stop come from the language model, so they were checked directly. On the long Chinese sentence this checkpoint produces 109 audio tokens, the same count as BF16. Over 128 requests at concurrency 64 the mean is 59.8 tokens against 57.6 for BF16, with a maximum of 74 against 65. Across those 128 outputs the longest fixed-period repetition measured 1.08 seconds, against 1.12 seconds measured on the BF16 outputs from the same protocol. Per-utterance comparisons against BF16 score up to 1.04 seconds on the reference side, so these values are the measurement floor and not a looping defect.

Output speed

Measured 2026-11-03 on one RTX PRO 6000 Blackwell (SM 120), vLLM-Omni 0.30.0, max_num_seqs 64, max_model_len 1024, temperature 0, seed 42. Each engine ran alone on the card. Both engines used their own deploy config: 6 GiB KV cache for BF16 and 2.25 GiB for this checkpoint, 174,752 and 65,536 tokens of capacity. Peak concurrent resident demand in these runs stayed under 4,000 tokens, so neither engine was scheduling against its cache ceiling.

Single stream, long Chinese sentence, one warmup call discarded, median of three. One audio token is 1/6.25 s of 48 kHz audio.

	Wall	Audio	Tokens	Realtime	ms/token
BF16	1.219 s	17.44 s	109	14.31×	11.18
This checkpoint	1.168 s	17.44 s	109	14.93×	10.71

Concurrency 64, 128 requests over three fixed paragraphs. The first pass captures the batch-64 decode graph and is not shown; the table is the warm second pass.

	Elapsed	Audio	Realtime	Tokens	ms/token	p50	p95
BF16	11.811 s	1180.2 s	99.9×	7376 (mean 57.6)	1.60	5.41 s	6.13 s
This checkpoint	11.402 s	1225.6 s	107.5×	7660 (mean 59.8)	1.49	5.52 s	6.51 s

Read this as a memory optimization with a small speed side effect, not as a speed switch. VoxCPM2's language-model GEMMs are narrow, the DiT accounts for roughly three times the per-token FLOPs of BaseLM and stays BF16, and SM 120 has no native FP4 path for weight-only Marlin, so the gain is single-digit percent. On this card the measured gain was 4% single stream and 7% per token at concurrency 64. Take the halved weight memory and the smaller KV budget; do not plan around a large latency win.

Loading

Packed weight_packed tensors are not a drop-in for upstream VoxCPM.from_pretrained, and vLLM-Omni 0.30.0 does not install them on this architecture by itself. Serving needs a small runtime adapter that maps base_lm.* and residual_lm.* packed tensors onto the paged LM linears through MarlinNvFp4LinearKernel, leaves every other module on its dense weight, and reports what it installed.

Two log lines confirm a correct load:

ATS VoxCPM2 quant: W4A4=0 A16=180 int4=0
Model loading took 3.21 GiB memory

If the second line says 4.86 GiB and the first line is missing, the adapter did not run inside the spawned engine process. That failure is silent and sounds like short loud clicks: the language model is left as an unloaded dense shell, so generation stops after one token (0.16 s of audio per request). The usual cause is an adapter that skips patching whenever sys.argv[0] == "-c", because a multiprocessing spawn child starts as ['-c', '--multiprocessing-fork']. Skip the healthcheck on its command text (import urllib.request), which does not match the spawn child.

With the adapter in place, GET /health returns 200 and /v1/audio/speech returns 48 kHz WAV that stops inside the context limit.

Recipe

recipe.yaml in this repository records the modifier, target and ignore patterns. Quantization ran in a host environment with llmcompressor 0.10.0.1, compressed-tensors 0.14.0.1 and torch 2.11.0+cu130; serving was tested against the vLLM-Omni v0.30.0 image.