Text-to-Speech
MLX
ONNX
Portuguese
tts
pt-br
brazilian-portuguese
small-model
quantized
apple-silicon
pingo
fargan
Instructions to use semfence/pingo-ptbr with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use semfence/pingo-ptbr with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download semfence/pingo-ptbr --local-dir pingo-ptbr
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
File size: 8,472 Bytes
27d8618 99afc33 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 | # Experiments
Everything we measured on the way to 0.87 MB, in order, including what did not work. WER numbers use the MiniMax `pt` set (100 sentences per
voice, Qwen3-ASR); they move by about ±0.2 points between runs, so small differences are noise.
## 1. Quantization without retraining
Starting point: pivo-ptbr in float32 (acoustic 2.34 MB + vocoder 3.27 MB = 5.61 MB). Symmetric per-output-channel quantization, no calibration.
Size counts the weights (int4 packed two per byte, plus one scale per channel).
| Bits (acoustic / vocoder) | Weights | WER F_s4 | WER M_sm4 |
|---|---:|---:|---:|
| float32 | 5.61 MB | 0.95% | 0.95% |
| fp16 / fp16 | 2.81 MB | 1.09% | 0.81% |
| int8 / int8 | 1.44 MB | 0.76% | 0.86% |
| int8 / fp16 | 2.24 MB | 0.76% | 1.05% |
| fp16 / int8 | 2.01 MB | 0.95% | 0.76% |
| int8 / int4 | 1.04 MB | 1.14% | 0.95% |
| int4 / int8 | 1.15 MB | 1.09% | 1.24% |
| int4 / int4 | 0.75 MB | 1.28% | 1.28% |
- **WER barely moves even at 4 bits** (naive post-training quantization): the ASR is tolerant. WER does not measure timbre.
- **Listening** (author, informal): 8 bits everywhere is fine; 4 bits everywhere is audibly noisier. Everything stays intelligible with a "phone call" sound,
which comes from the 16 kHz FARGAN vocoder and is already there in float32.
- Feature error (float32 vs quantized, 20 sentences): fp16 0.0002, int8 0.0048 (mean |feature| about 1). With int8 the predicted durations change by a
frame in most sentences (rounding), but the features stay close.
## 2. A noise proxy for the vocoder
Sample-by-sample SNR is meaningless for FARGAN (autoregressive; 8 bits gives 3.5 dB). We use the **log-mel spectral distance (LSD)** between the float32 vocoder
output and the quantized one, running both on the same features.
| Vocoder scheme | Size | LSD |
|---|---:|---:|
| int8 | 0.83 MB | 0.214 |
| int6 per channel | 0.62 MB | 0.383 |
| int5 per channel | 0.52 MB | 0.467 |
| int4 per channel | 0.42 MB | 0.729 |
| int4, groups of 16 inputs | 0.51 MB | 0.474 |
| **mixed (per-layer, see below)** | **0.66 MB** | **0.28** |
## 3. Per-layer sensitivity of the vocoder
All layers int8, one layer at a time lowered; increase in LSD:
| Layer | Parameters | ΔLSD at 4 bits | ΔLSD at 6 bits |
|---|---:|---:|---:|
| skip_dense | 88,064 | 0.280 | 0.063 |
| frame_conv | 62,976 | 0.184 | 0.040 |
| gru1.input | 130,560 | 0.170 | 0.007 |
| signal_out | 5,120 | 0.170 | 0.018 |
| cond.dense1 | 2,048 | 0.153 | 0.109 |
| cond.conv | 24,576 | 0.123 | 0.018 |
| gru1.hidden | 76,800 | 0.102 | 0.003 |
| cond.dense2 | 40,960 | 0.100 | 0.001 |
| gru2.input | 92,160 | 0.093 | 0.024 |
| gru2.hidden | 49,152 | 0.049 | -0.001 |
| cond.pitch_embedding | 2,688 | 0.042 | 0.004 |
| glu1 | 25,600 | 0.040 | 0.008 |
| gru3.input | 79,872 | 0.039 | -0.001 |
| glu2 | 16,384 | 0.034 | 0.003 |
| gru3.hidden | 49,152 | 0.033 | 0.002 |
| skip_glu | 16,384 | 0.031 | 0.005 |
| frame_glu | 36,864 | 0.021 | -0.002 |
| cond_gain.w | 80 | 0.020 | 0.002 |
| pitch_gain.w | 768 | 0.018 | -0.001 |
| glu3 | 16,384 | 0.016 | 0.004 |
Almost every layer takes 6 bits at no cost; four (`cond.dense1`, `skip_dense`, `frame_conv`, `gru2.input`) do not. A greedy allocation
(additive model from this table, then a real LSD check) gave the scheme in `pingo/qpack.py`:
**0.66 MB at LSD 0.28**, against 0.62 MB at LSD 0.38 for uniform 6 bits.
## 4. Per-layer sensitivity of the acoustic model
Feature error with float32's durations forced, and log-duration error:
| Scheme | Size | Δfeature | Δlog-duration |
|---|---:|---:|---:|
| all 8 bits | 0.60 MB | 0.0048 | 0.0040 |
| all 6 bits | 0.46 MB | 0.0222 | 0.0097 |
| all 5 bits | 0.38 MB | 0.0480 | 0.0293 |
| all 4 bits | 0.31 MB | 0.0975 | 0.0681 |
| only embeddings at 4 (rest 8) | 0.59 MB | 0.0638 | 0.0394 |
| only decoder at 4 (rest 8) | 0.52 MB | 0.0529 | 0.0040 |
| only attention at 4 (rest 8) | 0.56 MB | 0.0161 | 0.0084 |
| only FFN at 4 (rest 8) | 0.52 MB | 0.0277 | 0.0176 |
| only convolutions at 4 (rest 8) | 0.53 MB | 0.0234 | 0.0151 |
| only duration predictor at 4 (rest 8) | 0.59 MB | 0.0048 | 0.0375 |
| **chosen: encoder 5, decoder 6, durations 6, embeddings 8** | **0.41 MB** | **0.0232** | **0.0127** |
The embeddings (12k weights, about 12 KB at 8 bits) and the decoder are the most sensitive, so they get more bits.
## 5. The mixed scheme on the wide model (pingo-ptbr-w96)
Acoustic 0.41 MB + vocoder 0.66 MB = **1.07 MB** of weights. Simulated quantize-dequantize: WER 0.76% / 1.19%. Loaded from the packed file
(`weights/pingo-ptbr-w96.bin`, 1.086 MB including a 14 KB header): WER 0.81% / 1.00% (float32: 0.95% / 0.95%). Listening: the author judged it
the same quality as the 8-bit-everywhere version. Reconstruction error per tensor is at most 3.3% (the 5-bit layers).
A bug found on the way: a channel with all zeros (the padding row of the embedding) had an fp16 scale of 0 and produced NaN when packing;
fixed with a floor of 1e-4 on the scale.
## 6. Narrower acoustic model (width 64) → pingo-ptbr
Width 64, FFN 128: 270,081 parameters (46% of 584,897). 40,000 steps on a rented RTX 3090 (about 37 min, ~US$ 0.14). Validation loss:
| Step | 2k | 6k | 10k | 20k | 30k | 40k |
|---|---:|---:|---:|---:|---:|---:|
| feat + 0.1 × dur | 0.2028 | 0.1841 | 0.1789 | 0.1736 | 0.1707 | **0.1699** |
pivo-ptbr's width-96 model reached 0.1682 (at 20k steps, then overfitting). The width-64 model had not overfit at 40k.
Packed with the same bit scheme: `weights/pingo-ptbr.bin`, **0.871 MB** (acoustic 0.22 MB + vocoder 0.66 MB, header 14 KB). WER from the packed file:
**F_s4 0.95%, M_sm4 1.00%**.
Training notes (details in `HOW-WE-TRAINED.md`): the Mac ran out of memory with the full set (about 19 GB of swap in use on a 16 GB machine), so we trained
on a rented GPU; the first GPU run gave NaN because of a fully masked attention row in padding frames, fixed by keeping the mask diagonal open (after the fix the
numbers match the Mac's: step 300 feature loss 0.2800, duration loss 0.1176).
## 7. English words
The phonemizer reads foreign words with Portuguese rules. `pingo/pronunciation.py` rewrites a short list before phonemizing; we checked the resulting phonemes:
| Term | Rewritten as | Note |
|---|---|---|
| Hugging Face | Ráguin Feice | "Hâguim" lost the initial H (silent in Portuguese); R gives /x/ |
| GitHub | Guit rub | "Guitrráb" produced a stray /tʃi/ |
| startup | istártape | |
| deploy | diplói | |
| pitch | pítxi | "pítchi" produced an extra /ʃ/ |
| feedback | fídbéque | |
## 8. What we did not do
- ONNX that computes in low precision: the shipped ONNX stores int8 weights but the runtime dequantizes to float32 on load.
- Quantization-aware training, fine-tuning or pruning of the vocoder, a listening test with several people.
- Compare directly with other small pt-BR voices on a shared listening panel.
## 9. Stretched starts of short utterances
Symptom: "Olá!" sounds slow. The duration predictor gives the first "o" 300 ms and then a 340 ms pause after "!". We looked at the training data
(19,024 clips) and at the three models of this family:
- **The first vowel of an utterance is long in the data**: 160 ms on average against 110 ms elsewhere (median 16 frames, 95th percentile 25), for every utterance length.
- **A "!" inside a text is followed by a long pause in the data** (387 ms, 127 cases; at the end of a text, 0). The model copies it.
- **Very short inputs are outside the training distribution**: the shortest clip has 12 phonemes (median 42) and there is a single greeting text ("Olá, meu nome é Maria.").
"Olá!" has 3 phonemes, so the prediction extrapolates, and the narrow model extrapolates more.
| First phoneme, predicted | pingo (width 64) | pivo-ptbr (width 96) | tiny-voca-ptbr |
|---|---:|---:|---:|
| "o" of "Olá!" | 300 ms | 170 ms | 224 ms |
| "o" of "Olá, meu nome é Maria." | 260 ms | 200 ms | n/a |
Mitigation in the runtime (no retraining): every phoneme is limited to 20 frames (95th percentile of vowel length in the data) and the first phoneme
of an utterance to 17 frames (its median); pauses are untouched. "Olá!" then gets a 170 ms "o". The limit is in `pingo/pingo_mlx.py`
(`PHONE_LIMIT`, `FIRST_PHONE_LIMIT`, `limit=False` turns it off) and in `web/public/worker.js`. A real fix needs short utterances and greetings in the
training data. WER on the MiniMax set with the limit: F_s4 0.86%, M_sm4 0.81% (without: 0.95% / 1.00%), i.e. no worse, within noise.
|