pingo-ptbr / docs /EXPERIMENTS.md
itxtoledo's picture
Duration limit for short utterances (fixes stretched 'Olá!')
99afc33 verified
|
Raw History Blame Contribute Delete
8.47 kB
# Experiments
Everything we measured on the way to 0.87 MB, in order, including what did not work. WER numbers use the MiniMax `pt` set (100 sentences per
voice, Qwen3-ASR); they move by about ±0.2 points between runs, so small differences are noise.
## 1. Quantization without retraining
Starting point: pivo-ptbr in float32 (acoustic 2.34 MB + vocoder 3.27 MB = 5.61 MB). Symmetric per-output-channel quantization, no calibration.
Size counts the weights (int4 packed two per byte, plus one scale per channel).
| Bits (acoustic / vocoder) | Weights | WER F_s4 | WER M_sm4 |
|---|---:|---:|---:|
| float32 | 5.61 MB | 0.95% | 0.95% |
| fp16 / fp16 | 2.81 MB | 1.09% | 0.81% |
| int8 / int8 | 1.44 MB | 0.76% | 0.86% |
| int8 / fp16 | 2.24 MB | 0.76% | 1.05% |
| fp16 / int8 | 2.01 MB | 0.95% | 0.76% |
| int8 / int4 | 1.04 MB | 1.14% | 0.95% |
| int4 / int8 | 1.15 MB | 1.09% | 1.24% |
| int4 / int4 | 0.75 MB | 1.28% | 1.28% |
- **WER barely moves even at 4 bits** (naive post-training quantization): the ASR is tolerant. WER does not measure timbre.
- **Listening** (author, informal): 8 bits everywhere is fine; 4 bits everywhere is audibly noisier. Everything stays intelligible with a "phone call" sound,
which comes from the 16 kHz FARGAN vocoder and is already there in float32.
- Feature error (float32 vs quantized, 20 sentences): fp16 0.0002, int8 0.0048 (mean |feature| about 1). With int8 the predicted durations change by a
frame in most sentences (rounding), but the features stay close.
## 2. A noise proxy for the vocoder
Sample-by-sample SNR is meaningless for FARGAN (autoregressive; 8 bits gives 3.5 dB). We use the **log-mel spectral distance (LSD)** between the float32 vocoder
output and the quantized one, running both on the same features.
| Vocoder scheme | Size | LSD |
|---|---:|---:|
| int8 | 0.83 MB | 0.214 |
| int6 per channel | 0.62 MB | 0.383 |
| int5 per channel | 0.52 MB | 0.467 |
| int4 per channel | 0.42 MB | 0.729 |
| int4, groups of 16 inputs | 0.51 MB | 0.474 |
| **mixed (per-layer, see below)** | **0.66 MB** | **0.28** |
## 3. Per-layer sensitivity of the vocoder
All layers int8, one layer at a time lowered; increase in LSD:
| Layer | Parameters | ΔLSD at 4 bits | ΔLSD at 6 bits |
|---|---:|---:|---:|
| skip_dense | 88,064 | 0.280 | 0.063 |
| frame_conv | 62,976 | 0.184 | 0.040 |
| gru1.input | 130,560 | 0.170 | 0.007 |
| signal_out | 5,120 | 0.170 | 0.018 |
| cond.dense1 | 2,048 | 0.153 | 0.109 |
| cond.conv | 24,576 | 0.123 | 0.018 |
| gru1.hidden | 76,800 | 0.102 | 0.003 |
| cond.dense2 | 40,960 | 0.100 | 0.001 |
| gru2.input | 92,160 | 0.093 | 0.024 |
| gru2.hidden | 49,152 | 0.049 | -0.001 |
| cond.pitch_embedding | 2,688 | 0.042 | 0.004 |
| glu1 | 25,600 | 0.040 | 0.008 |
| gru3.input | 79,872 | 0.039 | -0.001 |
| glu2 | 16,384 | 0.034 | 0.003 |
| gru3.hidden | 49,152 | 0.033 | 0.002 |
| skip_glu | 16,384 | 0.031 | 0.005 |
| frame_glu | 36,864 | 0.021 | -0.002 |
| cond_gain.w | 80 | 0.020 | 0.002 |
| pitch_gain.w | 768 | 0.018 | -0.001 |
| glu3 | 16,384 | 0.016 | 0.004 |
Almost every layer takes 6 bits at no cost; four (`cond.dense1`, `skip_dense`, `frame_conv`, `gru2.input`) do not. A greedy allocation
(additive model from this table, then a real LSD check) gave the scheme in `pingo/qpack.py`:
**0.66 MB at LSD 0.28**, against 0.62 MB at LSD 0.38 for uniform 6 bits.
## 4. Per-layer sensitivity of the acoustic model
Feature error with float32's durations forced, and log-duration error:
| Scheme | Size | Δfeature | Δlog-duration |
|---|---:|---:|---:|
| all 8 bits | 0.60 MB | 0.0048 | 0.0040 |
| all 6 bits | 0.46 MB | 0.0222 | 0.0097 |
| all 5 bits | 0.38 MB | 0.0480 | 0.0293 |
| all 4 bits | 0.31 MB | 0.0975 | 0.0681 |
| only embeddings at 4 (rest 8) | 0.59 MB | 0.0638 | 0.0394 |
| only decoder at 4 (rest 8) | 0.52 MB | 0.0529 | 0.0040 |
| only attention at 4 (rest 8) | 0.56 MB | 0.0161 | 0.0084 |
| only FFN at 4 (rest 8) | 0.52 MB | 0.0277 | 0.0176 |
| only convolutions at 4 (rest 8) | 0.53 MB | 0.0234 | 0.0151 |
| only duration predictor at 4 (rest 8) | 0.59 MB | 0.0048 | 0.0375 |
| **chosen: encoder 5, decoder 6, durations 6, embeddings 8** | **0.41 MB** | **0.0232** | **0.0127** |
The embeddings (12k weights, about 12 KB at 8 bits) and the decoder are the most sensitive, so they get more bits.
## 5. The mixed scheme on the wide model (pingo-ptbr-w96)
Acoustic 0.41 MB + vocoder 0.66 MB = **1.07 MB** of weights. Simulated quantize-dequantize: WER 0.76% / 1.19%. Loaded from the packed file
(`weights/pingo-ptbr-w96.bin`, 1.086 MB including a 14 KB header): WER 0.81% / 1.00% (float32: 0.95% / 0.95%). Listening: the author judged it
the same quality as the 8-bit-everywhere version. Reconstruction error per tensor is at most 3.3% (the 5-bit layers).
A bug found on the way: a channel with all zeros (the padding row of the embedding) had an fp16 scale of 0 and produced NaN when packing;
fixed with a floor of 1e-4 on the scale.
## 6. Narrower acoustic model (width 64) → pingo-ptbr
Width 64, FFN 128: 270,081 parameters (46% of 584,897). 40,000 steps on a rented RTX 3090 (about 37 min, ~US$ 0.14). Validation loss:
| Step | 2k | 6k | 10k | 20k | 30k | 40k |
|---|---:|---:|---:|---:|---:|---:|
| feat + 0.1 × dur | 0.2028 | 0.1841 | 0.1789 | 0.1736 | 0.1707 | **0.1699** |
pivo-ptbr's width-96 model reached 0.1682 (at 20k steps, then overfitting). The width-64 model had not overfit at 40k.
Packed with the same bit scheme: `weights/pingo-ptbr.bin`, **0.871 MB** (acoustic 0.22 MB + vocoder 0.66 MB, header 14 KB). WER from the packed file:
**F_s4 0.95%, M_sm4 1.00%**.
Training notes (details in `HOW-WE-TRAINED.md`): the Mac ran out of memory with the full set (about 19 GB of swap in use on a 16 GB machine), so we trained
on a rented GPU; the first GPU run gave NaN because of a fully masked attention row in padding frames, fixed by keeping the mask diagonal open (after the fix the
numbers match the Mac's: step 300 feature loss 0.2800, duration loss 0.1176).
## 7. English words
The phonemizer reads foreign words with Portuguese rules. `pingo/pronunciation.py` rewrites a short list before phonemizing; we checked the resulting phonemes:
| Term | Rewritten as | Note |
|---|---|---|
| Hugging Face | Ráguin Feice | "Hâguim" lost the initial H (silent in Portuguese); R gives /x/ |
| GitHub | Guit rub | "Guitrráb" produced a stray /tʃi/ |
| startup | istártape | |
| deploy | diplói | |
| pitch | pítxi | "pítchi" produced an extra /ʃ/ |
| feedback | fídbéque | |
## 8. What we did not do
- ONNX that computes in low precision: the shipped ONNX stores int8 weights but the runtime dequantizes to float32 on load.
- Quantization-aware training, fine-tuning or pruning of the vocoder, a listening test with several people.
- Compare directly with other small pt-BR voices on a shared listening panel.
## 9. Stretched starts of short utterances
Symptom: "Olá!" sounds slow. The duration predictor gives the first "o" 300 ms and then a 340 ms pause after "!". We looked at the training data
(19,024 clips) and at the three models of this family:
- **The first vowel of an utterance is long in the data**: 160 ms on average against 110 ms elsewhere (median 16 frames, 95th percentile 25), for every utterance length.
- **A "!" inside a text is followed by a long pause in the data** (387 ms, 127 cases; at the end of a text, 0). The model copies it.
- **Very short inputs are outside the training distribution**: the shortest clip has 12 phonemes (median 42) and there is a single greeting text ("Olá, meu nome é Maria.").
"Olá!" has 3 phonemes, so the prediction extrapolates, and the narrow model extrapolates more.
| First phoneme, predicted | pingo (width 64) | pivo-ptbr (width 96) | tiny-voca-ptbr |
|---|---:|---:|---:|
| "o" of "Olá!" | 300 ms | 170 ms | 224 ms |
| "o" of "Olá, meu nome é Maria." | 260 ms | 200 ms | n/a |
Mitigation in the runtime (no retraining): every phoneme is limited to 20 frames (95th percentile of vowel length in the data) and the first phoneme
of an utterance to 17 frames (its median); pauses are untouched. "Olá!" then gets a 170 ms "o". The limit is in `pingo/pingo_mlx.py`
(`PHONE_LIMIT`, `FIRST_PHONE_LIMIT`, `limit=False` turns it off) and in `web/public/worker.js`. A real fix needs short utterances and greetings in the
training data. WER on the MiniMax set with the limit: F_s4 0.86%, M_sm4 0.81% (without: 0.95% / 1.00%), i.e. no worse, within noise.