ZONOS3-Mini β€” GPTQ-Pro INT4 / BF16 experts-only overlay

An experimental, real-weight GPTQ-Pro native INT4 overlay for Zyphra's ZONOS3-Mini TTS model, optimized for NVIDIA Ampere CUDA GPUs. This is not a standalone model. The original BF16 ZONOS3-Mini checkpoint, ZAC codec and the custom GPTQ-Pro PyTorch reference loader are required.

The model was quantized and tested on 15 November 2026, with 192 generated English/Spanish calibration requests, two separate 24-request held-out suites, 10 paired waveform syntheses per precision, and a multilingual ASR proxy check. No human MOS/listening study has been conducted; not a production audio-fidelity certification.

What's actually quantized

Item Measured / included
Layers processed 11 of 11
Eligible projections 1,317
Native symmetric INT4 GPTQ-Pro g64 1,005 projections
BF16-preserved projections 312
INT4 shard bytes 323,729,736 (~309 MiB)
BF16 source/codec Not included; required separately
Descending-activation order Disabled (desc_act=False)
Audio codec Preserved unchanged, loaded from Zyphra

All MoE experts with insufficient activation coverage stay at exact BF16 instead of receiving potentially unreliable INT4 weights. Attention, GDN/recurrent output, and small dense/latent layers are intentionally BF16 for better fidelity. Full architectural registration with GPTQModel.load(), and SGLang/MLX/GGUF inference, are not provided by this checkpoint. The custom GPTQ-Pro reference runtime is in runtime/.

The overlay is bound to the original upstream checkpoint:

  • Base model: Zyphra/ZONOS3-Mini, upstream HF revision d2d142a86b1e6aeb4252679673c86ecd86177bf8
  • model.pth SHA-256: 3b4ac0a6e3002bd575ad9a0761ae7524806a908f553042171dead8f3a2146caa
  • params.json SHA-256: 28dc488d7142831d739f427e048c699b7ed12852d88da0a9123a137f15aa5a5b
  • Overlay manifest SHA-256: 4c15e6d2afdaabba43df135a03241f95c990151f7c91b2ab93cc84f9665238ba

The loader checks those source hashes and the SHA-256 of every packed safetensors shard.

Benchmarks (RTX 3090 / PyTorch reference runtime)

Metric Original BF16 Experts-only GPTQ-Pro
Language-model allocated VRAM 1.651 GiB 0.858 GiB (βˆ’48.0%)
Full TTS peak allocated VRAM (with ZAC decoder) 2.272 GiB 1.479 GiB (βˆ’34.9%)
Mean warm synthesis, 5.6-second speech (n=9) 0.856 s 0.998 s (+16.5% slower)

Both sets generated 10 matched audio samples (five English, five Spanish), with the same request and seed, 90 generated codebook rows each. First-use codec initialization was excluded from the warm timings. Results are specific to the Python reference implementation, not the fast SGLang server. This implementation is memory-efficient, not faster.

Teacher-versus-INT4 held-out logit checks:

Corpus NMSE KL (nats/prediction) Token argmax agreement
24 standard requests 0.004083 0.018779 91.04%
24 independent, longer requests 0.003794 0.024353 87.57%

Both passed configured numeric gates (NMSE ≀ 0.10, KL ≀ 0.10). These numbers do not establish subjective voice quality. A short-clip, prefix-matched Whisper ASR proxy saw slightly worse transcription for INT4 versus BF16, especially for Spanish; see validation details.

Listen to the results

Matched audio pairs from the original BF16 checkpoint and the final experts-only GPTQ-Pro overlay. Each pair used exactly the same text and generation seed.

Voice/language Original BF16 GPTQ-Pro INT4
English, sample 00 Listen Listen
English, sample 02 Listen Listen
Spanish, sample 04 Listen Listen
Spanish, sample 07 Listen Listen

All 20 MP3s and their text/seed-matched samples are in samples/. The upstream public voice presets were used to create these samples.

How to load it

The overlay is not auto-discovered by Transformers or standard GPTQModel. Use the custom loader from groxaxo/GPTQ-Pro with an original source checkpoint already available locally.

from pathlib import Path
import torch
from huggingface_hub import snapshot_download
from zonos3.reference.lm import Zonos3LM
from examples.zonos3_mini.overlay_runtime import load_overlay

original_bf16 = "/path/to/zonos3_mini"  # must contain model.pth, params.json, tokenizer/
local_overlay = Path(snapshot_download(
    repo_id="groxaxo/ZONOS3-Mini-GPTQ-Pro-INT4",
    allow_patterns=["overlay/*"],
)) / "overlay/manifest.json"

model = Zonos3LM.load(original_bf16, device="cuda:0", dtype=torch.bfloat16).eval()
load_overlay(model, local_overlay, device="cuda:0", require_complete=True)

# Use normal zonos3.reference.synth.synthesize(model, request, assets=..., device="cuda:0")
# with an original Zyphra ZAC decoder and a valid GenerateRequest.

To generate a WAV with a speaker embedding, the repo provides a command-line entrypoint:

python examples/zonos3_mini/infer_published.py \
  --base /path/to/ZONOS3-Mini/lm/zonos3_mini \
  --overlay-repo groxaxo/ZONOS3-Mini-GPTQ-Pro-INT4 \
  --asset-root /path/to/ZONOS3-Mini \
  --speaker-embedding /path/to/a/ZONOS3-compatible-voice.embedding.npy \
  --text "This is an INT4 demo of ZONOS3-Mini." \
  --language en_us --max-tokens 120 \
  --device cuda:0 --out demo.wav

The environment must provide PyTorch with a compatible CUDA toolkit and the native GPTQ-Pro CUDA kernel, the reference ZONOS3 model package and original codec files. With --asset-root, the original Zyphra autoencoder files are read from disk; without it the reference model resolves these files from Hugging Face. See runtime documentation and the GPTQ-Pro PR for full installation/architecture caveats. Because original-source loader code can evolve, use a compatible revision of Zyphra's reference source.

Scope and limitations

  • Experimental native mixed-precision PyTorch overlay, not a full standalone quantized ZONOS3 checkpoint, not a vLLM/SGLang/MLX/GGUF implementation.
  • The original BF16 checkpoint and original codec are mandatory, including exact weights for preserved projections. Keeping the original disk checkpoint remains essential.
  • Only calibration, logit tests, basic speech generation and automated ASR proxy tests were performed. Blind human listening, MOS, speaker-identity comparisons and broader multilingual voice testing remain open.
  • TTS numerical quality and runtime results do not transfer automatically to other speaker presets, languages or batch sizes.
  • Ensure rights/consent before using any speaker reference or voice clone.

Provenance and licensing

Base model by Zyphra, licensed Apache-2.0. Quantization/runtime by groxaxo/GPTQ-Pro (Apache-2.0; see LICENSE). Attribution to the original model must be retained. This repo contains only derived INT4 overlay shards, custom loader code, evaluation records and synthesized reference audio examples; no original model weights or ZAC codec weights are redistributed.


Native SGLang-Omni INT4 serving (RTX 3090 tested)

This checkpoint now has actual SGLang-Omni support, using the original SGLang attention/GDN/CUDA graph/streaming-vocoder pipeline and a new native packed INT4 GPU expert kernel. It reads GPTQ-Pro g64 weights directly and keeps only the selected BF16 preserved experts; it does NOT dequantize every expert to BF16.

Set up the same SGLang-Omni runtime validated on PyTorch 2.14/CUDA13, plus the GPTQ-Pro checkout (no need to install the incompatible GPTQ-Pro PyTorch dependency tree into SGLang), and run:

export PYTHONPATH=/path/to/GPTQ-Pro:/path/to/groxaxo-sglang-omni:/path/to/ZONOS3-Mini/model/src:$PYTHONPATH
export ZONOS3_GPTQPRO_BASE=/path/to/ZONOS3-Mini/models/lm/zonos3_mini
export ZONOS3_GPTQPRO_OVERLAY=groxaxo/ZONOS3-Mini-GPTQ-Pro-INT4
sgl-omni serve --model-path /path/to/ZONOS3-Mini/models \
  --host 127.0.0.1 --port 18082 \
  --tts_engine.gpu_memory_fraction 0.65 \
  --tts_engine.factory.max_running_requests 2 \
  --tts_engine.engine.cuda_graph_max_bs 2

Local RTX3090 benchmark: median warm RTF 0.0245 (40.8Γ— real time), versus BF16 SGLang 0.0223 (44.8Γ—); approximately 684 MiB lower GPU process memory. Five paired EN/ES prompts synthesized successfully; all ten WAVs matched their intended transcripts in a small Whisper ASR check. CUDA graph capture passed for 43 prefill buckets and two decode sizes (1,2).

Limits: RTX3090 TP1 only in this release; still needs the original BF16 checkpoint and codec; INT4 is ~9.8% slower than SGLang's BF16 fused MoE on the five test utterances. No subjective MOS quality certification.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for groxaxo/ZONOS3-Mini-GPTQ-Pro-INT4

Finetuned
(4)
this model