ZONOS3-Mini β GPTQ-Pro INT4 / BF16 experts-only overlay
An experimental, real-weight GPTQ-Pro native INT4 overlay for Zyphra's ZONOS3-Mini TTS model, optimized for NVIDIA Ampere CUDA GPUs. This is not a standalone model. The original BF16 ZONOS3-Mini checkpoint, ZAC codec and the custom GPTQ-Pro PyTorch reference loader are required.
The model was quantized and tested on 15 November 2026, with 192 generated English/Spanish calibration requests, two separate 24-request held-out suites, 10 paired waveform syntheses per precision, and a multilingual ASR proxy check. No human MOS/listening study has been conducted; not a production audio-fidelity certification.
What's actually quantized
| Item | Measured / included |
|---|---|
| Layers processed | 11 of 11 |
| Eligible projections | 1,317 |
| Native symmetric INT4 GPTQ-Pro g64 | 1,005 projections |
| BF16-preserved projections | 312 |
| INT4 shard bytes | 323,729,736 (~309 MiB) |
| BF16 source/codec | Not included; required separately |
| Descending-activation order | Disabled (desc_act=False) |
| Audio codec | Preserved unchanged, loaded from Zyphra |
All MoE experts with insufficient activation coverage stay at exact BF16 instead of receiving potentially unreliable INT4 weights. Attention, GDN/recurrent output, and small dense/latent layers are intentionally BF16 for better fidelity. Full architectural registration with GPTQModel.load(), and SGLang/MLX/GGUF inference, are not provided by this checkpoint. The custom GPTQ-Pro reference runtime is in runtime/.
The overlay is bound to the original upstream checkpoint:
- Base model: Zyphra/ZONOS3-Mini, upstream HF revision
d2d142a86b1e6aeb4252679673c86ecd86177bf8 model.pthSHA-256:3b4ac0a6e3002bd575ad9a0761ae7524806a908f553042171dead8f3a2146caaparams.jsonSHA-256:28dc488d7142831d739f427e048c699b7ed12852d88da0a9123a137f15aa5a5b- Overlay manifest SHA-256:
4c15e6d2afdaabba43df135a03241f95c990151f7c91b2ab93cc84f9665238ba
The loader checks those source hashes and the SHA-256 of every packed safetensors shard.
Benchmarks (RTX 3090 / PyTorch reference runtime)
| Metric | Original BF16 | Experts-only GPTQ-Pro |
|---|---|---|
| Language-model allocated VRAM | 1.651 GiB | 0.858 GiB (β48.0%) |
| Full TTS peak allocated VRAM (with ZAC decoder) | 2.272 GiB | 1.479 GiB (β34.9%) |
| Mean warm synthesis, 5.6-second speech (n=9) | 0.856 s | 0.998 s (+16.5% slower) |
Both sets generated 10 matched audio samples (five English, five Spanish), with the same request and seed, 90 generated codebook rows each. First-use codec initialization was excluded from the warm timings. Results are specific to the Python reference implementation, not the fast SGLang server. This implementation is memory-efficient, not faster.
Teacher-versus-INT4 held-out logit checks:
| Corpus | NMSE | KL (nats/prediction) | Token argmax agreement |
|---|---|---|---|
| 24 standard requests | 0.004083 | 0.018779 | 91.04% |
| 24 independent, longer requests | 0.003794 | 0.024353 | 87.57% |
Both passed configured numeric gates (NMSE β€ 0.10, KL β€ 0.10). These numbers do not establish subjective voice quality. A short-clip, prefix-matched Whisper ASR proxy saw slightly worse transcription for INT4 versus BF16, especially for Spanish; see validation details.
Listen to the results
Matched audio pairs from the original BF16 checkpoint and the final experts-only GPTQ-Pro overlay. Each pair used exactly the same text and generation seed.
| Voice/language | Original BF16 | GPTQ-Pro INT4 |
|---|---|---|
| English, sample 00 | Listen | Listen |
| English, sample 02 | Listen | Listen |
| Spanish, sample 04 | Listen | Listen |
| Spanish, sample 07 | Listen | Listen |
All 20 MP3s and their text/seed-matched samples are in samples/. The upstream public voice presets were used to create these samples.
How to load it
The overlay is not auto-discovered by Transformers or standard GPTQModel. Use the custom loader from groxaxo/GPTQ-Pro with an original source checkpoint already available locally.
from pathlib import Path
import torch
from huggingface_hub import snapshot_download
from zonos3.reference.lm import Zonos3LM
from examples.zonos3_mini.overlay_runtime import load_overlay
original_bf16 = "/path/to/zonos3_mini" # must contain model.pth, params.json, tokenizer/
local_overlay = Path(snapshot_download(
repo_id="groxaxo/ZONOS3-Mini-GPTQ-Pro-INT4",
allow_patterns=["overlay/*"],
)) / "overlay/manifest.json"
model = Zonos3LM.load(original_bf16, device="cuda:0", dtype=torch.bfloat16).eval()
load_overlay(model, local_overlay, device="cuda:0", require_complete=True)
# Use normal zonos3.reference.synth.synthesize(model, request, assets=..., device="cuda:0")
# with an original Zyphra ZAC decoder and a valid GenerateRequest.
To generate a WAV with a speaker embedding, the repo provides a command-line entrypoint:
python examples/zonos3_mini/infer_published.py \
--base /path/to/ZONOS3-Mini/lm/zonos3_mini \
--overlay-repo groxaxo/ZONOS3-Mini-GPTQ-Pro-INT4 \
--asset-root /path/to/ZONOS3-Mini \
--speaker-embedding /path/to/a/ZONOS3-compatible-voice.embedding.npy \
--text "This is an INT4 demo of ZONOS3-Mini." \
--language en_us --max-tokens 120 \
--device cuda:0 --out demo.wav
The environment must provide PyTorch with a compatible CUDA toolkit and the native GPTQ-Pro CUDA kernel, the reference ZONOS3 model package and original codec files. With --asset-root, the original Zyphra autoencoder files are read from disk; without it the reference model resolves these files from Hugging Face. See runtime documentation and the GPTQ-Pro PR for full installation/architecture caveats. Because original-source loader code can evolve, use a compatible revision of Zyphra's reference source.
Scope and limitations
- Experimental native mixed-precision PyTorch overlay, not a full standalone quantized ZONOS3 checkpoint, not a vLLM/SGLang/MLX/GGUF implementation.
- The original BF16 checkpoint and original codec are mandatory, including exact weights for preserved projections. Keeping the original disk checkpoint remains essential.
- Only calibration, logit tests, basic speech generation and automated ASR proxy tests were performed. Blind human listening, MOS, speaker-identity comparisons and broader multilingual voice testing remain open.
- TTS numerical quality and runtime results do not transfer automatically to other speaker presets, languages or batch sizes.
- Ensure rights/consent before using any speaker reference or voice clone.
Provenance and licensing
Base model by Zyphra, licensed Apache-2.0. Quantization/runtime by groxaxo/GPTQ-Pro (Apache-2.0; see LICENSE). Attribution to the original model must be retained. This repo contains only derived INT4 overlay shards, custom loader code, evaluation records and synthesized reference audio examples; no original model weights or ZAC codec weights are redistributed.
Native SGLang-Omni INT4 serving (RTX 3090 tested)
This checkpoint now has actual SGLang-Omni support, using the original SGLang attention/GDN/CUDA graph/streaming-vocoder pipeline and a new native packed INT4 GPU expert kernel. It reads GPTQ-Pro g64 weights directly and keeps only the selected BF16 preserved experts; it does NOT dequantize every expert to BF16.
- GPTQ-Pro SGLang native backend and tests
- SGLang-Omni fork with opt-in model-loading hook
- Measured SGLang RTX3090 results
- Voice samples from the actual SGLang servers
Set up the same SGLang-Omni runtime validated on PyTorch 2.14/CUDA13, plus the GPTQ-Pro checkout (no need to install the incompatible GPTQ-Pro PyTorch dependency tree into SGLang), and run:
export PYTHONPATH=/path/to/GPTQ-Pro:/path/to/groxaxo-sglang-omni:/path/to/ZONOS3-Mini/model/src:$PYTHONPATH
export ZONOS3_GPTQPRO_BASE=/path/to/ZONOS3-Mini/models/lm/zonos3_mini
export ZONOS3_GPTQPRO_OVERLAY=groxaxo/ZONOS3-Mini-GPTQ-Pro-INT4
sgl-omni serve --model-path /path/to/ZONOS3-Mini/models \
--host 127.0.0.1 --port 18082 \
--tts_engine.gpu_memory_fraction 0.65 \
--tts_engine.factory.max_running_requests 2 \
--tts_engine.engine.cuda_graph_max_bs 2
Local RTX3090 benchmark: median warm RTF 0.0245 (40.8Γ real time), versus BF16 SGLang 0.0223 (44.8Γ); approximately 684 MiB lower GPU process memory. Five paired EN/ES prompts synthesized successfully; all ten WAVs matched their intended transcripts in a small Whisper ASR check. CUDA graph capture passed for 43 prefill buckets and two decode sizes (1,2).
Limits: RTX3090 TP1 only in this release; still needs the original BF16 checkpoint and codec; INT4 is ~9.8% slower than SGLang's BF16 fused MoE on the five test utterances. No subjective MOS quality certification.
Model tree for groxaxo/ZONOS3-Mini-GPTQ-Pro-INT4
Base model
Zyphra/ZONOS3-Mini