Text-to-Speech
MLX
ONNX
Portuguese
tts
pt-br
brazilian-portuguese
small-model
quantized
apple-silicon
pingo
fargan
Instructions to use semfence/pingo-ptbr with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use semfence/pingo-ptbr with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download semfence/pingo-ptbr --local-dir pingo-ptbr
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
|
Download docs/EXPERIMENTS.md from semfence/pingo-ptbr: direct link, hf CLI and curl.
- Browser
- Download file 8.47 kB
-
https://huggingface.co/semfence/pingo-ptbr/resolve/main/docs/EXPERIMENTS.md
- Command line
-
hf download hf://semfence/pingo-ptbr/docs/EXPERIMENTS.md
-
curl -L -o EXPERIMENTS.md https://huggingface.co/semfence/pingo-ptbr/resolve/main/docs/EXPERIMENTS.md
8.47 kB
| # Experiments | |
| Everything we measured on the way to 0.87 MB, in order, including what did not work. WER numbers use the MiniMax `pt` set (100 sentences per | |
| voice, Qwen3-ASR); they move by about ±0.2 points between runs, so small differences are noise. | |
| ## 1. Quantization without retraining | |
| Starting point: pivo-ptbr in float32 (acoustic 2.34 MB + vocoder 3.27 MB = 5.61 MB). Symmetric per-output-channel quantization, no calibration. | |
| Size counts the weights (int4 packed two per byte, plus one scale per channel). | |
| | Bits (acoustic / vocoder) | Weights | WER F_s4 | WER M_sm4 | | |
| |---|---:|---:|---:| | |
| | float32 | 5.61 MB | 0.95% | 0.95% | | |
| | fp16 / fp16 | 2.81 MB | 1.09% | 0.81% | | |
| | int8 / int8 | 1.44 MB | 0.76% | 0.86% | | |
| | int8 / fp16 | 2.24 MB | 0.76% | 1.05% | | |
| | fp16 / int8 | 2.01 MB | 0.95% | 0.76% | | |
| | int8 / int4 | 1.04 MB | 1.14% | 0.95% | | |
| | int4 / int8 | 1.15 MB | 1.09% | 1.24% | | |
| | int4 / int4 | 0.75 MB | 1.28% | 1.28% | | |
| - **WER barely moves even at 4 bits** (naive post-training quantization): the ASR is tolerant. WER does not measure timbre. | |
| - **Listening** (author, informal): 8 bits everywhere is fine; 4 bits everywhere is audibly noisier. Everything stays intelligible with a "phone call" sound, | |
| which comes from the 16 kHz FARGAN vocoder and is already there in float32. | |
| - Feature error (float32 vs quantized, 20 sentences): fp16 0.0002, int8 0.0048 (mean |feature| about 1). With int8 the predicted durations change by a | |
| frame in most sentences (rounding), but the features stay close. | |
| ## 2. A noise proxy for the vocoder | |
| Sample-by-sample SNR is meaningless for FARGAN (autoregressive; 8 bits gives 3.5 dB). We use the **log-mel spectral distance (LSD)** between the float32 vocoder | |
| output and the quantized one, running both on the same features. | |
| | Vocoder scheme | Size | LSD | | |
| |---|---:|---:| | |
| | int8 | 0.83 MB | 0.214 | | |
| | int6 per channel | 0.62 MB | 0.383 | | |
| | int5 per channel | 0.52 MB | 0.467 | | |
| | int4 per channel | 0.42 MB | 0.729 | | |
| | int4, groups of 16 inputs | 0.51 MB | 0.474 | | |
| | **mixed (per-layer, see below)** | **0.66 MB** | **0.28** | | |
| ## 3. Per-layer sensitivity of the vocoder | |
| All layers int8, one layer at a time lowered; increase in LSD: | |
| | Layer | Parameters | ΔLSD at 4 bits | ΔLSD at 6 bits | | |
| |---|---:|---:|---:| | |
| | skip_dense | 88,064 | 0.280 | 0.063 | | |
| | frame_conv | 62,976 | 0.184 | 0.040 | | |
| | gru1.input | 130,560 | 0.170 | 0.007 | | |
| | signal_out | 5,120 | 0.170 | 0.018 | | |
| | cond.dense1 | 2,048 | 0.153 | 0.109 | | |
| | cond.conv | 24,576 | 0.123 | 0.018 | | |
| | gru1.hidden | 76,800 | 0.102 | 0.003 | | |
| | cond.dense2 | 40,960 | 0.100 | 0.001 | | |
| | gru2.input | 92,160 | 0.093 | 0.024 | | |
| | gru2.hidden | 49,152 | 0.049 | -0.001 | | |
| | cond.pitch_embedding | 2,688 | 0.042 | 0.004 | | |
| | glu1 | 25,600 | 0.040 | 0.008 | | |
| | gru3.input | 79,872 | 0.039 | -0.001 | | |
| | glu2 | 16,384 | 0.034 | 0.003 | | |
| | gru3.hidden | 49,152 | 0.033 | 0.002 | | |
| | skip_glu | 16,384 | 0.031 | 0.005 | | |
| | frame_glu | 36,864 | 0.021 | -0.002 | | |
| | cond_gain.w | 80 | 0.020 | 0.002 | | |
| | pitch_gain.w | 768 | 0.018 | -0.001 | | |
| | glu3 | 16,384 | 0.016 | 0.004 | | |
| Almost every layer takes 6 bits at no cost; four (`cond.dense1`, `skip_dense`, `frame_conv`, `gru2.input`) do not. A greedy allocation | |
| (additive model from this table, then a real LSD check) gave the scheme in `pingo/qpack.py`: | |
| **0.66 MB at LSD 0.28**, against 0.62 MB at LSD 0.38 for uniform 6 bits. | |
| ## 4. Per-layer sensitivity of the acoustic model | |
| Feature error with float32's durations forced, and log-duration error: | |
| | Scheme | Size | Δfeature | Δlog-duration | | |
| |---|---:|---:|---:| | |
| | all 8 bits | 0.60 MB | 0.0048 | 0.0040 | | |
| | all 6 bits | 0.46 MB | 0.0222 | 0.0097 | | |
| | all 5 bits | 0.38 MB | 0.0480 | 0.0293 | | |
| | all 4 bits | 0.31 MB | 0.0975 | 0.0681 | | |
| | only embeddings at 4 (rest 8) | 0.59 MB | 0.0638 | 0.0394 | | |
| | only decoder at 4 (rest 8) | 0.52 MB | 0.0529 | 0.0040 | | |
| | only attention at 4 (rest 8) | 0.56 MB | 0.0161 | 0.0084 | | |
| | only FFN at 4 (rest 8) | 0.52 MB | 0.0277 | 0.0176 | | |
| | only convolutions at 4 (rest 8) | 0.53 MB | 0.0234 | 0.0151 | | |
| | only duration predictor at 4 (rest 8) | 0.59 MB | 0.0048 | 0.0375 | | |
| | **chosen: encoder 5, decoder 6, durations 6, embeddings 8** | **0.41 MB** | **0.0232** | **0.0127** | | |
| The embeddings (12k weights, about 12 KB at 8 bits) and the decoder are the most sensitive, so they get more bits. | |
| ## 5. The mixed scheme on the wide model (pingo-ptbr-w96) | |
| Acoustic 0.41 MB + vocoder 0.66 MB = **1.07 MB** of weights. Simulated quantize-dequantize: WER 0.76% / 1.19%. Loaded from the packed file | |
| (`weights/pingo-ptbr-w96.bin`, 1.086 MB including a 14 KB header): WER 0.81% / 1.00% (float32: 0.95% / 0.95%). Listening: the author judged it | |
| the same quality as the 8-bit-everywhere version. Reconstruction error per tensor is at most 3.3% (the 5-bit layers). | |
| A bug found on the way: a channel with all zeros (the padding row of the embedding) had an fp16 scale of 0 and produced NaN when packing; | |
| fixed with a floor of 1e-4 on the scale. | |
| ## 6. Narrower acoustic model (width 64) → pingo-ptbr | |
| Width 64, FFN 128: 270,081 parameters (46% of 584,897). 40,000 steps on a rented RTX 3090 (about 37 min, ~US$ 0.14). Validation loss: | |
| | Step | 2k | 6k | 10k | 20k | 30k | 40k | | |
| |---|---:|---:|---:|---:|---:|---:| | |
| | feat + 0.1 × dur | 0.2028 | 0.1841 | 0.1789 | 0.1736 | 0.1707 | **0.1699** | | |
| pivo-ptbr's width-96 model reached 0.1682 (at 20k steps, then overfitting). The width-64 model had not overfit at 40k. | |
| Packed with the same bit scheme: `weights/pingo-ptbr.bin`, **0.871 MB** (acoustic 0.22 MB + vocoder 0.66 MB, header 14 KB). WER from the packed file: | |
| **F_s4 0.95%, M_sm4 1.00%**. | |
| Training notes (details in `HOW-WE-TRAINED.md`): the Mac ran out of memory with the full set (about 19 GB of swap in use on a 16 GB machine), so we trained | |
| on a rented GPU; the first GPU run gave NaN because of a fully masked attention row in padding frames, fixed by keeping the mask diagonal open (after the fix the | |
| numbers match the Mac's: step 300 feature loss 0.2800, duration loss 0.1176). | |
| ## 7. English words | |
| The phonemizer reads foreign words with Portuguese rules. `pingo/pronunciation.py` rewrites a short list before phonemizing; we checked the resulting phonemes: | |
| | Term | Rewritten as | Note | | |
| |---|---|---| | |
| | Hugging Face | Ráguin Feice | "Hâguim" lost the initial H (silent in Portuguese); R gives /x/ | | |
| | GitHub | Guit rub | "Guitrráb" produced a stray /tʃi/ | | |
| | startup | istártape | | | |
| | deploy | diplói | | | |
| | pitch | pítxi | "pítchi" produced an extra /ʃ/ | | |
| | feedback | fídbéque | | | |
| ## 8. What we did not do | |
| - ONNX that computes in low precision: the shipped ONNX stores int8 weights but the runtime dequantizes to float32 on load. | |
| - Quantization-aware training, fine-tuning or pruning of the vocoder, a listening test with several people. | |
| - Compare directly with other small pt-BR voices on a shared listening panel. | |
| ## 9. Stretched starts of short utterances | |
| Symptom: "Olá!" sounds slow. The duration predictor gives the first "o" 300 ms and then a 340 ms pause after "!". We looked at the training data | |
| (19,024 clips) and at the three models of this family: | |
| - **The first vowel of an utterance is long in the data**: 160 ms on average against 110 ms elsewhere (median 16 frames, 95th percentile 25), for every utterance length. | |
| - **A "!" inside a text is followed by a long pause in the data** (387 ms, 127 cases; at the end of a text, 0). The model copies it. | |
| - **Very short inputs are outside the training distribution**: the shortest clip has 12 phonemes (median 42) and there is a single greeting text ("Olá, meu nome é Maria."). | |
| "Olá!" has 3 phonemes, so the prediction extrapolates, and the narrow model extrapolates more. | |
| | First phoneme, predicted | pingo (width 64) | pivo-ptbr (width 96) | tiny-voca-ptbr | | |
| |---|---:|---:|---:| | |
| | "o" of "Olá!" | 300 ms | 170 ms | 224 ms | | |
| | "o" of "Olá, meu nome é Maria." | 260 ms | 200 ms | n/a | | |
| Mitigation in the runtime (no retraining): every phoneme is limited to 20 frames (95th percentile of vowel length in the data) and the first phoneme | |
| of an utterance to 17 frames (its median); pauses are untouched. "Olá!" then gets a 170 ms "o". The limit is in `pingo/pingo_mlx.py` | |
| (`PHONE_LIMIT`, `FIRST_PHONE_LIMIT`, `limit=False` turns it off) and in `web/public/worker.js`. A real fix needs short utterances and greetings in the | |
| training data. WER on the MiniMax set with the limit: F_s4 0.86%, M_sm4 0.81% (without: 0.95% / 1.00%), i.e. no worse, within noise. | |