Reza2kn/drama-3-nvfp4

Drama 3 — NVFP4 routed experts

Built with Fish Audio.

NVFP4 derivative of Fish Audio Drama 3, with a tested, memory-bounded NVIDIA GPU runtime and English/Persian listening samples.

25.49 GB of tensor payload; about 27.42 GB including the codec and supporting files. Routed experts use NVFP4; attention, shared experts, embeddings, output head, audio decoder, and the other non-routed tensors retain their source precision. This is an experts-only quantization, not a claim that every model parameter is four-bit or that this is the smallest Drama 3 checkpoint.

Source revision: e0a9218aa48888270f06ef1ed9ea67622c0e17d2.

Quantization

• 40 layers × 256 routed experts × 3 projections = 30,720 quantized matrices.
• Packed E2M1 FP4 values, one E4M3 scale per 16 input-channel weights, and an FP32 global scale per matrix. Gate and up projections share their global scale within each expert.
• The released source already contains block-FP8 experts. Those values were dequantized and converted with round-to-nearest NVFP4; no calibration dataset, GPTQ optimization, or quantization-aware training was used.
• The supplied runtime uses native CUTLASS NVFP4 matrix multiplication and dynamic per-token FP32 activation scales, with E4M3 activation block scales. Non-routed computations retain BF16.
• The layout uses ModelOpt-style tensor names and configuration. The exporter is custom; this does not imply NVIDIA Model Optimizer calibration was performed.

FP4 payload plus its per-16-weight scales costs approximately 4.5 bits per routed-expert weight, before small global scales and serialization overhead. The full-model byte count also includes the retained higher-precision components.

Validation and listening

All work ran on a GeForce RTX 5080 Laptop GPU, 16 GB, on Linux. Validation includes:

• Complete tensor coverage: all 30,720 expert projections checked; all 805 retained tensors match the source exactly.
• Native NVFP4 GEMM and complete expert-operation tests for one-token decode and batched prefill inputs. Expert-output relative L2 errors were 0.1998 and 0.2172 in the two numerical canaries; these are not speech-quality scores.
• Real end-to-end synthesis of English, Persian, and English emotion tags. All five comparison WAVs are finite, nonempty, and have no clipped samples.
• The packaged portable launcher was then tested with its default relative checkpoint path and reproduced the approved English WAV byte-for-byte; see portable runtime smoke.
• The repository owner listened to the three NVFP4 clips and two source-reference clips and accepted them on October 15, 2026. This is a small listening review, not a broad benchmark or a formal WER evaluation.

Prompt	NVFP4	Source reference
English	5.48 s WAV	5.76 s WAV
Persian	7.06 s WAV	5.71 s WAV
English [sighs] / [laughs]	6.27 s WAV	—

The source references use the released FP8 expert weight values dequantized for BF16 expert computation, with the same backbone, audio decoder, codec, prompts, and sampling settings. They are not the upstream fused FP8-activation execution path. Samples are unconditional TTS without reference-voice cloning. See listening provenance, prompts, and human review.

Run the validated disk-backed path

The included runtime keeps a bounded 64-expert GPU cache and reads selected packed experts on demand. This makes validation possible on a 16 GB GPU, but introduces substantial Python, transfer, and disk overhead. This is not a fused-MoE throughput benchmark or a production-speed claim.

It integrates with these pinned source revisions:

• SGLang v0.5.21: e00930c5489053f26d86b179cee0d087f846acbb
• SGLang-Omni: 00ac898e38dc2e49499d35feeb1ab06c04049a96

The tested environment used PyTorch 2.13.0+cu130, Transformers 5.16.1, FlashInfer 0.6.18, CUTLASS DSL 4.6.2, TVM FFI 0.1.11, and sglang-kernel 0.4.7. The full environment snapshot records the installed versions. The tested host used Python 3.13; the upstream installation recipe targets Python 3.12. A clean installation of the recipe below has not been independently validated.

Start from a working installation of the pinned SGLang-Omni runtime with its fishaudio extra. For example, the upstream installation route is:

git clone https://github.com/sgl-project/sglang-omni.git
cd sglang-omni
git checkout 00ac898e38dc2e49499d35feeb1ab06c04049a96
uv venv --python 3.12 .venv
uv pip install --prerelease=allow -e ".[fishaudio]"

Download this repository and use the included launcher with that environment:

hf download Reza2kn/drama-3-nvfp4 --local-dir ./drama-3-nvfp4

export CUDA_HOME=/usr/local/cuda-13.3
export DRAMA3_PYTHON="$PWD/.venv/bin/python"
bash ./drama-3-nvfp4/runtime/serve.sh

Set CUDA_HOME to your installed toolkit. The validated host used a CUDA 13.3 compiler and the CUDA 13.0 PyTorch runtime. Use a matching nvcc/ptxas toolchain; compiler auto-discovery selected an incompatible pair during initial setup. Kernel compilation may take several minutes. The launcher defaults to one compiler job to limit host-memory pressure.

The server binds to 127.0.0.1:8000. Generate a WAV:

curl http://127.0.0.1:8000/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"input":"The rain has stopped, and the city sounds different now.","response_format":"wav"}' \
  --output output.wav

DRAMA3_CHECKPOINT, DRAMA3_CODEC_PATH, DRAMA3_HOST, and DRAMA3_PORT can override the default paths and endpoint. The launcher keeps reference preprocessing on CPU and the backbone and codec decoder on GPU. CUDA graphs are disabled. Its defaults allow one concurrent request, a 2,048-token context, and at most 768 generated frames per chunk.

Scope and limitations

• Tested on SM120 Blackwell hardware. Other architectures, multi-GPU configurations, and the complete fused serving path have not been validated for this derivative.
• A standard CPU-offload startup exhausted host RAM on the 32 GB validation machine. Use the included disk-backed launcher for the tested 16 GB path; weight-file size alone does not establish complete serving memory requirements.
• English, Persian, and one emotion-tagged English line were reviewed. Voice cloning, multi-speaker scenes, long-form generation, and the base model's other languages have not been evaluated here.
• No ASR/WER score, speaker-similarity benchmark, or upstream fused-FP8 quality equivalence is claimed.
• The conversion scripts under runtime/ preserve the original Stallion directory assumptions for reproducibility; adapt their ROOT paths for another conversion host. The inference launcher itself resolves paths relative to this repository or the environment overrides above.

License and attribution

Original model and codec: Fish Audio. The unchanged Fish Audio Community License v1.0 governs this derivative's weights. Read that license for the applicable use and redistribution terms.

Runtime integrates SGLang and SGLang-Omni. Their Apache-2.0 license texts are retained in runtime/THIRD_PARTY_LICENSES. See NOTICE.