pingo-ptbr / docs /EXPERIMENTS.md
itxtoledo's picture
Add pingo-ptbr: 0.87 MB pt-BR TTS (MLX + ONNX + web demo)
27d8618 verified
|
Raw History Blame Contribute Delete
6.86 kB

Experiments

Everything we measured on the way to 0.87 MB, in order, including what did not work. WER numbers use the MiniMax pt set (100 sentences per voice, Qwen3-ASR); they move by about ±0.2 points between runs, so small differences are noise.

1. Quantization without retraining

Starting point: pivo-ptbr in float32 (acoustic 2.34 MB + vocoder 3.27 MB = 5.61 MB). Symmetric per-output-channel quantization, no calibration. Size counts the weights (int4 packed two per byte, plus one scale per channel).

Bits (acoustic / vocoder) Weights WER F_s4 WER M_sm4
float32 5.61 MB 0.95% 0.95%
fp16 / fp16 2.81 MB 1.09% 0.81%
int8 / int8 1.44 MB 0.76% 0.86%
int8 / fp16 2.24 MB 0.76% 1.05%
fp16 / int8 2.01 MB 0.95% 0.76%
int8 / int4 1.04 MB 1.14% 0.95%
int4 / int8 1.15 MB 1.09% 1.24%
int4 / int4 0.75 MB 1.28% 1.28%
  • WER barely moves even at 4 bits (naive post-training quantization): the ASR is tolerant. WER does not measure timbre.
  • Listening (author, informal): 8 bits everywhere is fine; 4 bits everywhere is audibly noisier. Everything stays intelligible with a "phone call" sound, which comes from the 16 kHz FARGAN vocoder and is already there in float32.
  • Feature error (float32 vs quantized, 20 sentences): fp16 0.0002, int8 0.0048 (mean |feature| about 1). With int8 the predicted durations change by a frame in most sentences (rounding), but the features stay close.

2. A noise proxy for the vocoder

Sample-by-sample SNR is meaningless for FARGAN (autoregressive; 8 bits gives 3.5 dB). We use the log-mel spectral distance (LSD) between the float32 vocoder output and the quantized one, running both on the same features.

Vocoder scheme Size LSD
int8 0.83 MB 0.214
int6 per channel 0.62 MB 0.383
int5 per channel 0.52 MB 0.467
int4 per channel 0.42 MB 0.729
int4, groups of 16 inputs 0.51 MB 0.474
mixed (per-layer, see below) 0.66 MB 0.28

3. Per-layer sensitivity of the vocoder

All layers int8, one layer at a time lowered; increase in LSD:

Layer Parameters ΔLSD at 4 bits ΔLSD at 6 bits
skip_dense 88,064 0.280 0.063
frame_conv 62,976 0.184 0.040
gru1.input 130,560 0.170 0.007
signal_out 5,120 0.170 0.018
cond.dense1 2,048 0.153 0.109
cond.conv 24,576 0.123 0.018
gru1.hidden 76,800 0.102 0.003
cond.dense2 40,960 0.100 0.001
gru2.input 92,160 0.093 0.024
gru2.hidden 49,152 0.049 -0.001
cond.pitch_embedding 2,688 0.042 0.004
glu1 25,600 0.040 0.008
gru3.input 79,872 0.039 -0.001
glu2 16,384 0.034 0.003
gru3.hidden 49,152 0.033 0.002
skip_glu 16,384 0.031 0.005
frame_glu 36,864 0.021 -0.002
cond_gain.w 80 0.020 0.002
pitch_gain.w 768 0.018 -0.001
glu3 16,384 0.016 0.004

Almost every layer takes 6 bits at no cost; four (cond.dense1, skip_dense, frame_conv, gru2.input) do not. A greedy allocation (additive model from this table, then a real LSD check) gave the scheme in pingo/qpack.py: 0.66 MB at LSD 0.28, against 0.62 MB at LSD 0.38 for uniform 6 bits.

4. Per-layer sensitivity of the acoustic model

Feature error with float32's durations forced, and log-duration error:

Scheme Size Δfeature Δlog-duration
all 8 bits 0.60 MB 0.0048 0.0040
all 6 bits 0.46 MB 0.0222 0.0097
all 5 bits 0.38 MB 0.0480 0.0293
all 4 bits 0.31 MB 0.0975 0.0681
only embeddings at 4 (rest 8) 0.59 MB 0.0638 0.0394
only decoder at 4 (rest 8) 0.52 MB 0.0529 0.0040
only attention at 4 (rest 8) 0.56 MB 0.0161 0.0084
only FFN at 4 (rest 8) 0.52 MB 0.0277 0.0176
only convolutions at 4 (rest 8) 0.53 MB 0.0234 0.0151
only duration predictor at 4 (rest 8) 0.59 MB 0.0048 0.0375
chosen: encoder 5, decoder 6, durations 6, embeddings 8 0.41 MB 0.0232 0.0127

The embeddings (12k weights, about 12 KB at 8 bits) and the decoder are the most sensitive, so they get more bits.

5. The mixed scheme on the wide model (pingo-ptbr-w96)

Acoustic 0.41 MB + vocoder 0.66 MB = 1.07 MB of weights. Simulated quantize-dequantize: WER 0.76% / 1.19%. Loaded from the packed file (weights/pingo-ptbr-w96.bin, 1.086 MB including a 14 KB header): WER 0.81% / 1.00% (float32: 0.95% / 0.95%). Listening: the author judged it the same quality as the 8-bit-everywhere version. Reconstruction error per tensor is at most 3.3% (the 5-bit layers).

A bug found on the way: a channel with all zeros (the padding row of the embedding) had an fp16 scale of 0 and produced NaN when packing; fixed with a floor of 1e-4 on the scale.

6. Narrower acoustic model (width 64) → pingo-ptbr

Width 64, FFN 128: 270,081 parameters (46% of 584,897). 40,000 steps on a rented RTX 3090 (about 37 min, ~US$ 0.14). Validation loss:

Step 2k 6k 10k 20k 30k 40k
feat + 0.1 × dur 0.2028 0.1841 0.1789 0.1736 0.1707 0.1699

pivo-ptbr's width-96 model reached 0.1682 (at 20k steps, then overfitting). The width-64 model had not overfit at 40k.

Packed with the same bit scheme: weights/pingo-ptbr.bin, 0.871 MB (acoustic 0.22 MB + vocoder 0.66 MB, header 14 KB). WER from the packed file: F_s4 0.95%, M_sm4 1.00%.

Training notes (details in HOW-WE-TRAINED.md): the Mac ran out of memory with the full set (about 19 GB of swap in use on a 16 GB machine), so we trained on a rented GPU; the first GPU run gave NaN because of a fully masked attention row in padding frames, fixed by keeping the mask diagonal open (after the fix the numbers match the Mac's: step 300 feature loss 0.2800, duration loss 0.1176).

7. English words

The phonemizer reads foreign words with Portuguese rules. pingo/pronunciation.py rewrites a short list before phonemizing; we checked the resulting phonemes:

Term Rewritten as Note
Hugging Face Ráguin Feice "Hâguim" lost the initial H (silent in Portuguese); R gives /x/
GitHub Guit rub "Guitrráb" produced a stray /tʃi/
startup istártape
deploy diplói
pitch pítxi "pítchi" produced an extra /ʃ/
feedback fídbéque

8. What we did not do

  • ONNX that computes in low precision: the shipped ONNX stores int8 weights but the runtime dequantizes to float32 on load.
  • Quantization-aware training, fine-tuning or pruning of the vocoder, a listening test with several people.
  • Compare directly with other small pt-BR voices on a shared listening panel.