Gatozu35's picture
ZONOS3-Mini for native transformers (zonos3_mini)
3f582bf verified
|
Raw History Blame Contribute Delete
4.4 kB
---
license: apache-2.0
library_name: transformers
pipeline_tag: text-to-speech
base_model: Zyphra/ZONOS3-Mini
language:
- en
- fr
- es
- de
- it
- pt
- nl
- pl
- ru
- sv
- tr
- uk
- cs
- da
- fi
- el
- hu
- "no"
- ro
- sk
- bg
- zh
- ja
- ko
- ar
- he
- vi
- id
- th
- tl
- ms
- ka
- hi
- bn
- ta
- te
- mr
- gu
- kn
- ml
- pa
- or
- ur
tags:
- text-to-speech
- tts
- speech-synthesis
- multilingual
- voice-cloning
- zonos
- zonos3
- zyphra
---
# ZONOS3-Mini (native 🤗 Transformers)
[ZONOS3-Mini](https://huggingface.co/Zyphra/ZONOS3-Mini) is Zyphra's compact multilingual text-to-speech model (43
languages, about 100 million active parameters per token). This repo holds the checkpoint for the native
`transformers` port (`zonos3_mini`, no `trust_remote_code`): the language model, the MECT-B2 speaker encoder, the
SuperBPE tokenizer, the processor and the 27 release voices in `voices/`. The processor loads the
[ZAC-1](https://huggingface.co/Zyphra/ZAC-1-hf) codec as its audio tokenizer.
Model details, samples and evaluations: the [blog post](https://www.zyphra.com/our-work/zonos3-mini) and
[technical report](https://arxiv.org/abs/2611.16525).
## Usage
```python
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import AutoModelForTextToWaveform, AutoProcessor
model_id = "Zyphra/ZONOS3-Mini-transformers"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForTextToWaveform.from_pretrained(model_id, device_map="auto")
voice = load_file(hf_hub_download(model_id, "voices/Savannah.safetensors"))
conversation = [{"role": "user", "content": "Hello! This is ZONOS3-Mini running in transformers."}]
inputs = processor.apply_chat_template(
conversation,
tokenize=True,
return_dict=True,
processor_kwargs={"speaker_reference_codes": [voice["codes"]]},
).to(model.device)
audio = processor.decode(model.generate(**inputs))
processor.save_audio(audio, "zonos3_mini.wav")
```
To clone a voice from a clip, put the clip (44.1 kHz) in the user turn:
```python
conversation = [
{
"role": "user",
"content": [
{"type": "text", "text": "This voice was cloned from a short clip."},
{"type": "audio", "path": "my_voice.wav"},
],
}
]
inputs = processor.apply_chat_template(
conversation, tokenize=True, return_dict=True, processor_kwargs={"sampling_rate": 44100}
).to(model.device)
audio = processor.decode(model.generate(**inputs))
```
The clip is encoded into reference codes by ZAC-1 by default; `speaker_mode="embedding"` conditions on its speaker
embedding instead, and `speaker_guidance=True` with `guidance_scale > 1` in `generate` strengthens the voice. The
documentation covers conditions (loudness, fmax, speaking rate), batching, `torch.compile`, the pipeline and training.
Install `zyphra-textnorm` to read numbers, dates, currencies and units aloud as the model was trained; without it, the
text is used as written.
## Voices
`voices/<Name>.safetensors` holds each release voice's reference codes (`codes`, `[frames, 10]`) and speaker embedding
(`embedding`, `[192]`); `voices/voices.json` lists their language, style and gender.
## Architecture
| Component | ZONOS3-Mini |
|---|---|
| Backbone | 11 layers, dimension 1,024; latent MoE with 64 experts, top-2 routing from layer 1 |
| Sequence mixer | 9 Gated DeltaNet layers + 2 global attention layers, Full AttnRes; context 32,768 positions |
| Text | SuperBPE, 8,192 entries + 4 special tokens |
| Audio tokens | 15 Hz; 10 codebooks of 4,096 entries (1 semantic + 9 acoustic); hop 2,940 samples at 44.1 kHz |
| Speaker prompt | codec reference prefix (up to 60 s) or a 192-d MECT-B2 embedding |
| Codec | [ZAC-1](https://huggingface.co/Zyphra/ZAC-1-hf) `keep2_plain` decoder |
The language model is stored in bfloat16 (the release dtype); the speaker encoder stays in float32.
## License
Apache-2.0.
## Citation
```bibtex
@misc{zyphra2026zonos3mini,
title = {{ZONOS3-Mini Technical Report}},
author = {Gabriel Clark and Sofian Mejjoute and George Close and Beren Millidge},
year = {2026},
eprint = {2611.16525},
archivePrefix = {arXiv},
primaryClass = {cs.SD},
url = {https://arxiv.org/abs/2611.16525},
}
```