File size: 8,472 Bytes
27d8618
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
99afc33
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
# Experiments

Everything we measured on the way to 0.87 MB, in order, including what did not work. WER numbers use the MiniMax `pt` set (100 sentences per
voice, Qwen3-ASR); they move by about ±0.2 points between runs, so small differences are noise.

## 1. Quantization without retraining

Starting point: pivo-ptbr in float32 (acoustic 2.34 MB + vocoder 3.27 MB = 5.61 MB). Symmetric per-output-channel quantization, no calibration.
Size counts the weights (int4 packed two per byte, plus one scale per channel).

| Bits (acoustic / vocoder) | Weights | WER F_s4 | WER M_sm4 |
|---|---:|---:|---:|
| float32 | 5.61 MB | 0.95% | 0.95% |
| fp16 / fp16 | 2.81 MB | 1.09% | 0.81% |
| int8 / int8 | 1.44 MB | 0.76% | 0.86% |
| int8 / fp16 | 2.24 MB | 0.76% | 1.05% |
| fp16 / int8 | 2.01 MB | 0.95% | 0.76% |
| int8 / int4 | 1.04 MB | 1.14% | 0.95% |
| int4 / int8 | 1.15 MB | 1.09% | 1.24% |
| int4 / int4 | 0.75 MB | 1.28% | 1.28% |

- **WER barely moves even at 4 bits** (naive post-training quantization): the ASR is tolerant. WER does not measure timbre.
- **Listening** (author, informal): 8 bits everywhere is fine; 4 bits everywhere is audibly noisier. Everything stays intelligible with a "phone call" sound,
  which comes from the 16 kHz FARGAN vocoder and is already there in float32.
- Feature error (float32 vs quantized, 20 sentences): fp16 0.0002, int8 0.0048 (mean |feature| about 1). With int8 the predicted durations change by a
  frame in most sentences (rounding), but the features stay close.

## 2. A noise proxy for the vocoder

Sample-by-sample SNR is meaningless for FARGAN (autoregressive; 8 bits gives 3.5 dB). We use the **log-mel spectral distance (LSD)** between the float32 vocoder
output and the quantized one, running both on the same features.

| Vocoder scheme | Size | LSD |
|---|---:|---:|
| int8 | 0.83 MB | 0.214 |
| int6 per channel | 0.62 MB | 0.383 |
| int5 per channel | 0.52 MB | 0.467 |
| int4 per channel | 0.42 MB | 0.729 |
| int4, groups of 16 inputs | 0.51 MB | 0.474 |
| **mixed (per-layer, see below)** | **0.66 MB** | **0.28** |

## 3. Per-layer sensitivity of the vocoder

All layers int8, one layer at a time lowered; increase in LSD:

| Layer | Parameters | ΔLSD at 4 bits | ΔLSD at 6 bits |
|---|---:|---:|---:|
| skip_dense | 88,064 | 0.280 | 0.063 |
| frame_conv | 62,976 | 0.184 | 0.040 |
| gru1.input | 130,560 | 0.170 | 0.007 |
| signal_out | 5,120 | 0.170 | 0.018 |
| cond.dense1 | 2,048 | 0.153 | 0.109 |
| cond.conv | 24,576 | 0.123 | 0.018 |
| gru1.hidden | 76,800 | 0.102 | 0.003 |
| cond.dense2 | 40,960 | 0.100 | 0.001 |
| gru2.input | 92,160 | 0.093 | 0.024 |
| gru2.hidden | 49,152 | 0.049 | -0.001 |
| cond.pitch_embedding | 2,688 | 0.042 | 0.004 |
| glu1 | 25,600 | 0.040 | 0.008 |
| gru3.input | 79,872 | 0.039 | -0.001 |
| glu2 | 16,384 | 0.034 | 0.003 |
| gru3.hidden | 49,152 | 0.033 | 0.002 |
| skip_glu | 16,384 | 0.031 | 0.005 |
| frame_glu | 36,864 | 0.021 | -0.002 |
| cond_gain.w | 80 | 0.020 | 0.002 |
| pitch_gain.w | 768 | 0.018 | -0.001 |
| glu3 | 16,384 | 0.016 | 0.004 |

Almost every layer takes 6 bits at no cost; four (`cond.dense1`, `skip_dense`, `frame_conv`, `gru2.input`) do not. A greedy allocation
(additive model from this table, then a real LSD check) gave the scheme in `pingo/qpack.py`:
**0.66 MB at LSD 0.28**, against 0.62 MB at LSD 0.38 for uniform 6 bits.

## 4. Per-layer sensitivity of the acoustic model

Feature error with float32's durations forced, and log-duration error:

| Scheme | Size | Δfeature | Δlog-duration |
|---|---:|---:|---:|
| all 8 bits | 0.60 MB | 0.0048 | 0.0040 |
| all 6 bits | 0.46 MB | 0.0222 | 0.0097 |
| all 5 bits | 0.38 MB | 0.0480 | 0.0293 |
| all 4 bits | 0.31 MB | 0.0975 | 0.0681 |
| only embeddings at 4 (rest 8) | 0.59 MB | 0.0638 | 0.0394 |
| only decoder at 4 (rest 8) | 0.52 MB | 0.0529 | 0.0040 |
| only attention at 4 (rest 8) | 0.56 MB | 0.0161 | 0.0084 |
| only FFN at 4 (rest 8) | 0.52 MB | 0.0277 | 0.0176 |
| only convolutions at 4 (rest 8) | 0.53 MB | 0.0234 | 0.0151 |
| only duration predictor at 4 (rest 8) | 0.59 MB | 0.0048 | 0.0375 |
| **chosen: encoder 5, decoder 6, durations 6, embeddings 8** | **0.41 MB** | **0.0232** | **0.0127** |

The embeddings (12k weights, about 12 KB at 8 bits) and the decoder are the most sensitive, so they get more bits.

## 5. The mixed scheme on the wide model (pingo-ptbr-w96)

Acoustic 0.41 MB + vocoder 0.66 MB = **1.07 MB** of weights. Simulated quantize-dequantize: WER 0.76% / 1.19%. Loaded from the packed file
(`weights/pingo-ptbr-w96.bin`, 1.086 MB including a 14 KB header): WER 0.81% / 1.00% (float32: 0.95% / 0.95%). Listening: the author judged it
the same quality as the 8-bit-everywhere version. Reconstruction error per tensor is at most 3.3% (the 5-bit layers).

A bug found on the way: a channel with all zeros (the padding row of the embedding) had an fp16 scale of 0 and produced NaN when packing;
fixed with a floor of 1e-4 on the scale.

## 6. Narrower acoustic model (width 64) → pingo-ptbr

Width 64, FFN 128: 270,081 parameters (46% of 584,897). 40,000 steps on a rented RTX 3090 (about 37 min, ~US$ 0.14). Validation loss:

| Step | 2k | 6k | 10k | 20k | 30k | 40k |
|---|---:|---:|---:|---:|---:|---:|
| feat + 0.1 × dur | 0.2028 | 0.1841 | 0.1789 | 0.1736 | 0.1707 | **0.1699** |

pivo-ptbr's width-96 model reached 0.1682 (at 20k steps, then overfitting). The width-64 model had not overfit at 40k.

Packed with the same bit scheme: `weights/pingo-ptbr.bin`, **0.871 MB** (acoustic 0.22 MB + vocoder 0.66 MB, header 14 KB). WER from the packed file:
**F_s4 0.95%, M_sm4 1.00%**.

Training notes (details in `HOW-WE-TRAINED.md`): the Mac ran out of memory with the full set (about 19 GB of swap in use on a 16 GB machine), so we trained
on a rented GPU; the first GPU run gave NaN because of a fully masked attention row in padding frames, fixed by keeping the mask diagonal open (after the fix the
numbers match the Mac's: step 300 feature loss 0.2800, duration loss 0.1176).

## 7. English words

The phonemizer reads foreign words with Portuguese rules. `pingo/pronunciation.py` rewrites a short list before phonemizing; we checked the resulting phonemes:

| Term | Rewritten as | Note |
|---|---|---|
| Hugging Face | Ráguin Feice | "Hâguim" lost the initial H (silent in Portuguese); R gives /x/ |
| GitHub | Guit rub | "Guitrráb" produced a stray /tʃi/ |
| startup | istártape | |
| deploy | diplói | |
| pitch | pítxi | "pítchi" produced an extra /ʃ/ |
| feedback | fídbéque | |

## 8. What we did not do

- ONNX that computes in low precision: the shipped ONNX stores int8 weights but the runtime dequantizes to float32 on load.
- Quantization-aware training, fine-tuning or pruning of the vocoder, a listening test with several people.
- Compare directly with other small pt-BR voices on a shared listening panel.

## 9. Stretched starts of short utterances

Symptom: "Olá!" sounds slow. The duration predictor gives the first "o" 300 ms and then a 340 ms pause after "!". We looked at the training data
(19,024 clips) and at the three models of this family:

- **The first vowel of an utterance is long in the data**: 160 ms on average against 110 ms elsewhere (median 16 frames, 95th percentile 25), for every utterance length.
- **A "!" inside a text is followed by a long pause in the data** (387 ms, 127 cases; at the end of a text, 0). The model copies it.
- **Very short inputs are outside the training distribution**: the shortest clip has 12 phonemes (median 42) and there is a single greeting text ("Olá, meu nome é Maria.").
  "Olá!" has 3 phonemes, so the prediction extrapolates, and the narrow model extrapolates more.

| First phoneme, predicted | pingo (width 64) | pivo-ptbr (width 96) | tiny-voca-ptbr |
|---|---:|---:|---:|
| "o" of "Olá!" | 300 ms | 170 ms | 224 ms |
| "o" of "Olá, meu nome é Maria." | 260 ms | 200 ms | n/a |

Mitigation in the runtime (no retraining): every phoneme is limited to 20 frames (95th percentile of vowel length in the data) and the first phoneme
of an utterance to 17 frames (its median); pauses are untouched. "Olá!" then gets a 170 ms "o". The limit is in `pingo/pingo_mlx.py`
(`PHONE_LIMIT`, `FIRST_PHONE_LIMIT`, `limit=False` turns it off) and in `web/public/worker.js`. A real fix needs short utterances and greetings in the
training data. WER on the MiniMax set with the limit: F_s4 0.86%, M_sm4 0.81% (without: 0.95% / 1.00%), i.e. no worse, within noise.