dangvansam/drama-3-nvfp4

Drama 3 NVFP4 (based on Fish Audio Drama 3)

Built with Fish Audio. This repository is a modified (Derivative Work) copy of Fish Audio Drama 3. It is not an official Fish Audio release and is not sponsored or endorsed by Fish Audio. Use is governed by the Fish Audio Community License v1.0 (see also NOTICE): research and non-commercial use are free, commercial use is limited to licensees with annual revenue below USD 500,000, offering a Hosted Service (API, SDK or TTS website/app) needs a separate license, and outputs must not be used to create, train or distill any speech or foundation model. Do not imitate real people's voices without consent.

Standalone NVFP4 checkpoint of Drama 3 (no FP8 weights inside): 24 GB instead of 37 GB.

All versions

version	experts	checkpoint (GiB)	VRAM sglang / vLLM (GiB)	TTFA p50, 1 stream, sglang / vLLM (s)	letter CER hard, sglang / vLLM	CER plain, sglang / vLLM
Drama 3 FP8 original	256	36.9	46.2 / -	0.134 / -	6.9% / -	4.1% / -
NVFP4 (this model)	256	23.8	36.3 / 30.9	0.125 / 0.141	6.6% / 7.3%	2.9% / 2.8%
192E FP8	192	29.4	40.2 / 37.5	0.129 / 0.158	7.6% / 6.7%	3.6% / 3.2%
192E NVFP4	192	19.5	31.7 / 27.2	0.124 / 0.143	6.7% / 6.2%	3.7% / 3.9%
128E FP8	128	21.9	32.2 / 29.9	0.127 / 0.154	7.8% / 8.4%	4.8% / 6.9%
128E NVFP4	128	15.3	27.0 / 23.6	0.12 / 0.146	7.5% / 7.7%	5.2% / 3.9%

Same measurement setup for every row (see Results below); the green badge is this repository. The vLLM VRAM depends on the GPU memory budget chosen for the test.

What changed

Routed experts were converted from FP8 (block 128) to NVFP4 (group 16, FP8 block scales, one FP32 global scale per expert, round to nearest, no calibration) in ModelOpt W4A16_NVFP4 layout; all other weights are unchanged bf16.

Results

Measured on one NVIDIA RTX PRO 6000 (sm_120, 96 GB) that other jobs were using at the same time, 2 voices (Northern female / Northern male), 20 hard text-normalisation sentences + 40 plain sentences per voice (120 requests, 8 concurrent). CER is computed with a Vietnamese Zipformer ASR; speaker similarity is the cosine similarity to the reference voice. VRAM is the sum over all processes of the server after the benchmark. TTFA is time to the first audio chunk (streaming), median over repeated requests at 1, 4 and 8 concurrent streams, best of 3 repeats in one server session because the GPU was shared with other jobs. With 40 hard and 80 plain sentences, differences of about 1.5 points in CER are inside the noise.

engine / model	VRAM (GiB)	TTFA p50 s, 1 / 4 / 8 streams	letter CER (hard)	CER (plain)	speaker sim (hard)	failed
sglang-omni, fishaudio/drama-3 FP8 (reference)	46.2	0.134 / 0.165 / 0.215	6.9%	4.1%	0.847	0
sglang-omni, drama-3-nvfp4	36.3	0.125 / 0.15 / 0.187	6.6%	2.9%	0.844	0
vLLM-Omni, drama-3-nvfp4	30.9	0.141 / 0.156 / 0.182	7.3%	2.8%	0.838	0

Listen

Same text and reference voice in every column. The reference column is the unmodified FP8 Drama 3.

text	fishaudio/drama-3 FP8 (reference)	this model, sglang-omni	this model, vLLM-Omni
Đại diện bảo hiểm xã hội, ủy ban nhân dân tỉnh và Sở giáo dục và đào tạo đã họp với WHO và ASEAN về công nghệ thông tin. (Northern female voice)
Cờ lên lấy lại huy hiệu, bỏ mũ phớt đặt trước ngực, khẽ vái chào, sau đó quay người đẩy cửa đi ra. (Northern male voice)
Đóng quân nhà dân không được làm phiền nhiễu nhân dân, phải giữ gìn nhà cửa sạch sẽ. (Northern male voice)
Cơm gạo mới gặt dẻo nhẹo, dùng chung mắm chưng nóng hổi, kèo nèo, dưa chuột, rau sống ngon lạ, ngon lùng. (Northern female voice)

Run

Download the repository first and run the commands from inside it: hf download dangvansam/drama-3-nvfp4 --local-dir drama-3-nvfp4 && cd drama-3-nvfp4.

Tested with sglang-omni commit 2cfb3b6 (unmodified) and vLLM 0.31.0 with vLLM-Omni git c470395bd, on an RTX PRO 6000 (sm_120). The NVFP4 checkpoint loads through the stock ModelOpt W4A16_NVFP4 path of both engines, no patch needed.

sglang-omni

# sglang-omni commit 2cfb3b6 (sglang 0.5.21, flashinfer 0.6.18)
sgl-omni serve --model-path dangvansam/drama-3-nvfp4 --host 0.0.0.0 --port 8000 --config examples/sglang_omni.yaml

Request: examples/client_sglang_omni.py.

vLLM-Omni

# vllm 0.31.0 + vllm-omni (git c470395bd), CUDA 13 toolkit for DeepGEMM; set VLLM_USE_DEEP_GEMM=0 if you only have CUDA 12.x
vllm serve dangvansam/drama-3-nvfp4 --omni --port 8091 --served-model-name fishaudio/drama-3 --deploy-config examples/vllm_omni_deploy.yaml --moe-backend marlin

Request: examples/client_vllm_omni.py. The deploy config disables MPS and caps GPU memory so the server can share a GPU; raise gpu_memory_utilization, max_num_seqs and max_model_len for more concurrent streams.

Limitations

• Only tested on one GPU model (RTX PRO 6000, sm_120) with 2 reference voices and Vietnamese text; other languages, accents and emotion tags were not evaluated.
• Small test set (120 requests); CER and similarity do not capture prosody or artefacts, so listen before deploying.
• NVFP4 here is weight-only (W4A16, Marlin kernels): weights are 4-bit, activations stay bf16. The input_scale tensors are placeholders and are ignored by the Marlin path.

Reproduce

tools/ contains the scripts used to produce this checkpoint from fishaudio/drama-3.