# Experiments Everything we measured on the way to 0.87 MB, in order, including what did not work. WER numbers use the MiniMax `pt` set (100 sentences per voice, Qwen3-ASR); they move by about ±0.2 points between runs, so small differences are noise. ## 1. Quantization without retraining Starting point: pivo-ptbr in float32 (acoustic 2.34 MB + vocoder 3.27 MB = 5.61 MB). Symmetric per-output-channel quantization, no calibration. Size counts the weights (int4 packed two per byte, plus one scale per channel). | Bits (acoustic / vocoder) | Weights | WER F_s4 | WER M_sm4 | |---|---:|---:|---:| | float32 | 5.61 MB | 0.95% | 0.95% | | fp16 / fp16 | 2.81 MB | 1.09% | 0.81% | | int8 / int8 | 1.44 MB | 0.76% | 0.86% | | int8 / fp16 | 2.24 MB | 0.76% | 1.05% | | fp16 / int8 | 2.01 MB | 0.95% | 0.76% | | int8 / int4 | 1.04 MB | 1.14% | 0.95% | | int4 / int8 | 1.15 MB | 1.09% | 1.24% | | int4 / int4 | 0.75 MB | 1.28% | 1.28% | - **WER barely moves even at 4 bits** (naive post-training quantization): the ASR is tolerant. WER does not measure timbre. - **Listening** (author, informal): 8 bits everywhere is fine; 4 bits everywhere is audibly noisier. Everything stays intelligible with a "phone call" sound, which comes from the 16 kHz FARGAN vocoder and is already there in float32. - Feature error (float32 vs quantized, 20 sentences): fp16 0.0002, int8 0.0048 (mean |feature| about 1). With int8 the predicted durations change by a frame in most sentences (rounding), but the features stay close. ## 2. A noise proxy for the vocoder Sample-by-sample SNR is meaningless for FARGAN (autoregressive; 8 bits gives 3.5 dB). We use the **log-mel spectral distance (LSD)** between the float32 vocoder output and the quantized one, running both on the same features. | Vocoder scheme | Size | LSD | |---|---:|---:| | int8 | 0.83 MB | 0.214 | | int6 per channel | 0.62 MB | 0.383 | | int5 per channel | 0.52 MB | 0.467 | | int4 per channel | 0.42 MB | 0.729 | | int4, groups of 16 inputs | 0.51 MB | 0.474 | | **mixed (per-layer, see below)** | **0.66 MB** | **0.28** | ## 3. Per-layer sensitivity of the vocoder All layers int8, one layer at a time lowered; increase in LSD: | Layer | Parameters | ΔLSD at 4 bits | ΔLSD at 6 bits | |---|---:|---:|---:| | skip_dense | 88,064 | 0.280 | 0.063 | | frame_conv | 62,976 | 0.184 | 0.040 | | gru1.input | 130,560 | 0.170 | 0.007 | | signal_out | 5,120 | 0.170 | 0.018 | | cond.dense1 | 2,048 | 0.153 | 0.109 | | cond.conv | 24,576 | 0.123 | 0.018 | | gru1.hidden | 76,800 | 0.102 | 0.003 | | cond.dense2 | 40,960 | 0.100 | 0.001 | | gru2.input | 92,160 | 0.093 | 0.024 | | gru2.hidden | 49,152 | 0.049 | -0.001 | | cond.pitch_embedding | 2,688 | 0.042 | 0.004 | | glu1 | 25,600 | 0.040 | 0.008 | | gru3.input | 79,872 | 0.039 | -0.001 | | glu2 | 16,384 | 0.034 | 0.003 | | gru3.hidden | 49,152 | 0.033 | 0.002 | | skip_glu | 16,384 | 0.031 | 0.005 | | frame_glu | 36,864 | 0.021 | -0.002 | | cond_gain.w | 80 | 0.020 | 0.002 | | pitch_gain.w | 768 | 0.018 | -0.001 | | glu3 | 16,384 | 0.016 | 0.004 | Almost every layer takes 6 bits at no cost; four (`cond.dense1`, `skip_dense`, `frame_conv`, `gru2.input`) do not. A greedy allocation (additive model from this table, then a real LSD check) gave the scheme in `pingo/qpack.py`: **0.66 MB at LSD 0.28**, against 0.62 MB at LSD 0.38 for uniform 6 bits. ## 4. Per-layer sensitivity of the acoustic model Feature error with float32's durations forced, and log-duration error: | Scheme | Size | Δfeature | Δlog-duration | |---|---:|---:|---:| | all 8 bits | 0.60 MB | 0.0048 | 0.0040 | | all 6 bits | 0.46 MB | 0.0222 | 0.0097 | | all 5 bits | 0.38 MB | 0.0480 | 0.0293 | | all 4 bits | 0.31 MB | 0.0975 | 0.0681 | | only embeddings at 4 (rest 8) | 0.59 MB | 0.0638 | 0.0394 | | only decoder at 4 (rest 8) | 0.52 MB | 0.0529 | 0.0040 | | only attention at 4 (rest 8) | 0.56 MB | 0.0161 | 0.0084 | | only FFN at 4 (rest 8) | 0.52 MB | 0.0277 | 0.0176 | | only convolutions at 4 (rest 8) | 0.53 MB | 0.0234 | 0.0151 | | only duration predictor at 4 (rest 8) | 0.59 MB | 0.0048 | 0.0375 | | **chosen: encoder 5, decoder 6, durations 6, embeddings 8** | **0.41 MB** | **0.0232** | **0.0127** | The embeddings (12k weights, about 12 KB at 8 bits) and the decoder are the most sensitive, so they get more bits. ## 5. The mixed scheme on the wide model (pingo-ptbr-w96) Acoustic 0.41 MB + vocoder 0.66 MB = **1.07 MB** of weights. Simulated quantize-dequantize: WER 0.76% / 1.19%. Loaded from the packed file (`weights/pingo-ptbr-w96.bin`, 1.086 MB including a 14 KB header): WER 0.81% / 1.00% (float32: 0.95% / 0.95%). Listening: the author judged it the same quality as the 8-bit-everywhere version. Reconstruction error per tensor is at most 3.3% (the 5-bit layers). A bug found on the way: a channel with all zeros (the padding row of the embedding) had an fp16 scale of 0 and produced NaN when packing; fixed with a floor of 1e-4 on the scale. ## 6. Narrower acoustic model (width 64) → pingo-ptbr Width 64, FFN 128: 270,081 parameters (46% of 584,897). 40,000 steps on a rented RTX 3090 (about 37 min, ~US$ 0.14). Validation loss: | Step | 2k | 6k | 10k | 20k | 30k | 40k | |---|---:|---:|---:|---:|---:|---:| | feat + 0.1 × dur | 0.2028 | 0.1841 | 0.1789 | 0.1736 | 0.1707 | **0.1699** | pivo-ptbr's width-96 model reached 0.1682 (at 20k steps, then overfitting). The width-64 model had not overfit at 40k. Packed with the same bit scheme: `weights/pingo-ptbr.bin`, **0.871 MB** (acoustic 0.22 MB + vocoder 0.66 MB, header 14 KB). WER from the packed file: **F_s4 0.95%, M_sm4 1.00%**. Training notes (details in `HOW-WE-TRAINED.md`): the Mac ran out of memory with the full set (about 19 GB of swap in use on a 16 GB machine), so we trained on a rented GPU; the first GPU run gave NaN because of a fully masked attention row in padding frames, fixed by keeping the mask diagonal open (after the fix the numbers match the Mac's: step 300 feature loss 0.2800, duration loss 0.1176). ## 7. English words The phonemizer reads foreign words with Portuguese rules. `pingo/pronunciation.py` rewrites a short list before phonemizing; we checked the resulting phonemes: | Term | Rewritten as | Note | |---|---|---| | Hugging Face | Ráguin Feice | "Hâguim" lost the initial H (silent in Portuguese); R gives /x/ | | GitHub | Guit rub | "Guitrráb" produced a stray /tʃi/ | | startup | istártape | | | deploy | diplói | | | pitch | pítxi | "pítchi" produced an extra /ʃ/ | | feedback | fídbéque | | ## 8. What we did not do - ONNX that computes in low precision: the shipped ONNX stores int8 weights but the runtime dequantizes to float32 on load. - Quantization-aware training, fine-tuning or pruning of the vocoder, a listening test with several people. - Compare directly with other small pt-BR voices on a shared listening panel. ## 9. Stretched starts of short utterances Symptom: "Olá!" sounds slow. The duration predictor gives the first "o" 300 ms and then a 340 ms pause after "!". We looked at the training data (19,024 clips) and at the three models of this family: - **The first vowel of an utterance is long in the data**: 160 ms on average against 110 ms elsewhere (median 16 frames, 95th percentile 25), for every utterance length. - **A "!" inside a text is followed by a long pause in the data** (387 ms, 127 cases; at the end of a text, 0). The model copies it. - **Very short inputs are outside the training distribution**: the shortest clip has 12 phonemes (median 42) and there is a single greeting text ("Olá, meu nome é Maria."). "Olá!" has 3 phonemes, so the prediction extrapolates, and the narrow model extrapolates more. | First phoneme, predicted | pingo (width 64) | pivo-ptbr (width 96) | tiny-voca-ptbr | |---|---:|---:|---:| | "o" of "Olá!" | 300 ms | 170 ms | 224 ms | | "o" of "Olá, meu nome é Maria." | 260 ms | 200 ms | n/a | Mitigation in the runtime (no retraining): every phoneme is limited to 20 frames (95th percentile of vowel length in the data) and the first phoneme of an utterance to 17 frames (its median); pauses are untouched. "Olá!" then gets a 170 ms "o". The limit is in `pingo/pingo_mlx.py` (`PHONE_LIMIT`, `FIRST_PHONE_LIMIT`, `limit=False` turns it off) and in `web/public/worker.js`. A real fix needs short utterances and greetings in the training data. WER on the MiniMax set with the limit: F_s4 0.86%, M_sm4 0.81% (without: 0.95% / 1.00%), i.e. no worse, within noise.