zeechimp commited on
Commit
5691579
Β·
verified Β·
1 Parent(s): 15b9f68

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +299 -0
README.md ADDED
@@ -0,0 +1,299 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: whisper-decoder
4
+ tags:
5
+ - speech
6
+ - whisper
7
+ - formants
8
+ - mfcc
9
+ - keyword-spotting
10
+ - numpy
11
+ - cpu
12
+ pipeline_tag: audio-classification
13
+ language:
14
+ - en
15
+ ---
16
+
17
+ # whisper-decoder
18
+
19
+ **A classifier trained only on whispered speech identifies the
20
+ same words when they are voiced at 83.2% accuracy β€” 8.3Γ— chance.**
21
+
22
+ Whispering removes the pitch source. The vocal folds stop vibrating
23
+ and the only sound that leaves the mouth is turbulent noise shaped
24
+ by the vocal tract. The formants survive. The word survives. Only
25
+ the voicing is gone.
26
+
27
+ A classifier trained on whispered digits and evaluated on voiced
28
+ digits recovers the word from the formants alone. This is what the
29
+ experiment measures: how much of a word's identity lives in the
30
+ formant structure, independent of the pitch that normal speech
31
+ provides.
32
+
33
+ ## The claim in one sentence
34
+
35
+ Formant structure is sufficient to identify a closed-vocabulary
36
+ word even when the voicing source that produced the word is
37
+ completely different.
38
+
39
+ ## What it produces
40
+
41
+ A closed-vocabulary classifier over ten English digits
42
+ (`zero` through `nine`). Given a 1.2-second utterance, it returns
43
+ the predicted word and a probability distribution over the
44
+ vocabulary.
45
+
46
+ Internally it computes 49 features:
47
+
48
+ | feature family | count | purpose |
49
+ |---|---|---|
50
+ | MFCC mean + std | 26 | spectral envelope, formants |
51
+ | low-band mel mean + std | 20 | F1 and F2 explicitly, 200–2500 Hz |
52
+ | harmonicity mean + std | 2 | voicing detector (near 0 for whisper) |
53
+ | trimmed duration | 1 | word length after silence removal |
54
+
55
+ ## Install
56
+
57
+ ```bash
58
+ pip install numpy
59
+ ```
60
+
61
+ No other dependencies for the classifier. `matplotlib` is optional
62
+ for the figures. No downloads. No model weights. The classifier is
63
+ 5,610 parameters and trains from scratch in 1.6 seconds.
64
+
65
+ ## Usage
66
+
67
+ ### Run the demo
68
+
69
+ ```bash
70
+ python whisper_decoder.py
71
+ ```
72
+
73
+ Prints self-test results, generates training and test data,
74
+ trains the classifier, and evaluates on three corpora. Writes one
75
+ figure to `whisper_figures/`.
76
+
77
+ ### Programmatic
78
+
79
+ ```python
80
+ from whisper_decoder import (
81
+ MLP, train_mlp, extract_features,
82
+ synth_word, generate_corpus, FEATURE_DIM,
83
+ )
84
+
85
+ # Generate a training corpus (whisper mode, 200 samples per word)
86
+ train = generate_corpus('whisper', 200, 40, seed=0,
87
+ reverb_train_frac=0.25)
88
+
89
+ # Standardize
90
+ mu = train.X_train.mean(axis=0)
91
+ sigma = train.X_train.std(axis=0) + 1e-9
92
+ X_tr = (train.X_train - mu) / sigma
93
+
94
+ # Train
95
+ model = MLP(FEATURE_DIM, 64, 32, 10, seed=0)
96
+ train_mlp(model, X_tr, train.y_train, epochs=200, batch=64)
97
+
98
+ # Classify a new utterance
99
+ sig = synth_word('seven', 'whisper', seed=42)
100
+ feats = extract_features(sig)
101
+ probs = model.predict_proba(((feats - mu) / sigma)[None, :])[0]
102
+ predicted = train.word_index_inv[int(probs.argmax())]
103
+ print(predicted, probs.max())
104
+ ```
105
+
106
+ ### Self-test
107
+
108
+ ```bash
109
+ python whisper_decoder.py --self-test
110
+ ```
111
+
112
+ 11 checks covering the synthesizer, harmonicity, feature
113
+ extraction, and MFCC reproducibility.
114
+
115
+ ## Benchmarks
116
+
117
+ Trained on 2,000 synthetic whispered utterances (200 per word,
118
+ 25% with reverb augmentation). Evaluated on 400 held-out
119
+ utterances.
120
+
121
+ ### Primary results
122
+
123
+ | corpus | accuracy | chance | ratio |
124
+ |---|---|---|---|
125
+ | **voiced speech (domain shift)** | **83.2%** | 10.0% | **8.3Γ—** |
126
+ | whisper (in-distribution) | 100.0% | 10.0% | 10.0Γ— |
127
+ | reverberant whisper | 100.0% | 10.0% | 10.0Γ— |
128
+
129
+ **The voiced result is the finding.** The classifier was never
130
+ trained on voiced speech. It has never seen a pulse train. The
131
+ same 49 features β€” MFCC, low-band mel, harmonicity, duration β€”
132
+ still identify the word.
133
+
134
+ ### What the confusion matrix looks like
135
+
136
+ Perfect diagonal on the whispered test set. Every digit is
137
+ classified correctly, 40 out of 40.
138
+
139
+ This is a synthetic dataset, not real whispered speech. The
140
+ 100% is an upper bound on the synthetic task, not a prediction
141
+ of real-world accuracy.
142
+
143
+ ### The reverb row needs a caveat
144
+
145
+ The reverb augmentation during training and the reverb applied
146
+ during testing both use `rt60_s=0.35`. The training and test
147
+ distributions are the same. The 100% measures that the classifier
148
+ handles reverb *from the family it was trained on*. It does not
149
+ measure out-of-distribution robustness.
150
+
151
+ An example script (`examples/reverb_ood.py`) tests the honest
152
+ version: train on `rt60=0.35`, evaluate on `rt60=0.7`. Run it to
153
+ get the real number.
154
+
155
+ ## How it works
156
+
157
+ ### Synthesis
158
+
159
+ The source-filter model of speech production, in its simplest
160
+ form. Each word is a sequence of phonemes drawn from a small
161
+ inventory. Each phoneme has a formant target: a list of
162
+ `(frequency_hz, bandwidth_hz, amplitude)` triples. The source is
163
+ either white noise (whisper) or a pulse train at F0 (voiced).
164
+ The formant bank is a sum of Gaussian resonances applied to the
165
+ source spectrum.
166
+
167
+ Same phonemes, same formants, different source. The only thing
168
+ that changes between a whispered utterance and a voiced utterance
169
+ of the same word is the excitation.
170
+
171
+ ### Features
172
+
173
+ - **MFCC**: 13 coefficients, mean and std over frames
174
+ - **Low-band mel**: 10 bands over 200–2500 Hz, mean and std
175
+ - **Harmonicity**: per-frame peak of the normalized autocorrelation
176
+ in the pitch-lag range (80–300 Hz). Near zero for whisper, 0.5–0.9
177
+ for voiced.
178
+ - **Trimmed duration**: length of the non-silent portion
179
+
180
+ The low-band range is deliberately extended to 2500 Hz because F2
181
+ for front vowels (`i`, `e`) sits at 2000–2500 Hz. A narrower band
182
+ loses `three`, `zero`, and `five`.
183
+
184
+ ### Why it works on voiced speech
185
+
186
+ The formants in the synthesized voiced speech occupy the same
187
+ frequencies as the formants in the synthesized whispered speech.
188
+ The classifier learned to weight MFCC and low-band mel, which are
189
+ formant-driven. Harmonicity is a near-zero feature in the
190
+ whisper training set and stays near zero in the voiced test set
191
+ for unvoiced consonants but rises above 0.5 for vowels. The
192
+ classifier is not thrown off because the other features carry
193
+ the word identity.
194
+
195
+ ## Applications
196
+
197
+ - **Assistive speech interfaces for whispered input.** A user who
198
+ has lost their voice to illness, surgery, or fatigue can whisper
199
+ and still be understood by a device.
200
+ - **Security and anonymity.** Whispering is used when the speaker
201
+ does not want to be heard. A decoder that works on whispers is
202
+ a decoder that works on material recorded from a distance with
203
+ a directional microphone.
204
+ - **Speech reconstruction from damaged recordings.** A recording
205
+ that has been aggressively denoised, high-pass filtered, or
206
+ compressed may still carry enough formant structure. The
207
+ classifier works on the formants, not on the pitch that is the
208
+ first thing to be lost.
209
+ - **Cross-modal keyword spotting.** A wake-word detector that
210
+ triggers whether the wake word is whispered or spoken. Most
211
+ wake-word systems are trained on voiced speech and fail on
212
+ whispers.
213
+
214
+ ## When to use it
215
+
216
+ - When the vocabulary is small and fixed.
217
+ - When utterances are roughly the same length.
218
+ - When the source-filter assumption holds β€” a single speaker,
219
+ moderate background noise, no heavy reverb.
220
+
221
+ ## When *not* to use it
222
+
223
+ - **Open-vocabulary transcription.** This is not a speech
224
+ recognizer. It is a 10-way classifier.
225
+ - **Real whispered speech at scale.** The training data is
226
+ synthetic. Real whispers have coarticulation, breath noise,
227
+ and speaker-specific formant variation that the simulator
228
+ does not model.
229
+ - **Speaker-independent deployment.** The formant targets are
230
+ averaged across speakers. An individual speaker's formants
231
+ will sit near but not on these targets. A real deployment
232
+ needs per-speaker calibration or a real training set.
233
+ - **Continuous speech.** The classifier assumes the entire
234
+ clip contains one word. It does not detect boundaries.
235
+
236
+ ## Honest limitations
237
+
238
+ - **The training data is synthetic.** Every number in the
239
+ benchmark table comes from a simulator, not from a recording.
240
+ Real whispered speech will have different MFCC means and
241
+ different harmonicity values. The accuracy numbers here are
242
+ upper bounds on the synthetic task.
243
+ - **The reverb test is not out-of-distribution.** See the
244
+ caveat above.
245
+ - **The confusion matrix is perfect.** On the synthetic test
246
+ set the classifier does not confuse any pair of digits. This
247
+ is because the formant targets are fixed and the noise is
248
+ white β€” a real dataset will have overlap.
249
+ - **Coarticulation is not modeled.** In real speech, adjacent
250
+ phonemes shift each other's formants. The synthetic words
251
+ concatenate independent phonemes without cross-influence.
252
+ - **Duration is a feature.** For real speech, duration varies
253
+ by speaker and by utterance. A synthetic-trained classifier
254
+ that has learned "seven is 0.51 seconds long" may not
255
+ generalize.
256
+ - **The vocabulary is ten digits.** Extending to 100 words would
257
+ require more phonemes and probably more features.
258
+
259
+ ## What is not novel
260
+
261
+ - The source-filter model of speech (Fant, 1960).
262
+ - MFCC features (Davis & Mermelstein, 1980).
263
+ - Whispered speech recognition as a task (Ito et al., 2005;
264
+ Zhang & Hansen, 2008).
265
+ - The observation that formants survive whispering (well
266
+ established in phonetics).
267
+
268
+ The contribution here is the *controlled pair* β€” same word,
269
+ same formants, two sources β€” and the demonstration that a
270
+ classifier trained on one still works on the other. It is a
271
+ clean measurement of how much information the formants carry.
272
+
273
+ ## Version history
274
+
275
+ | version | change |
276
+ |---|---|
277
+ | 0.1.0 | synthesis, features, MLP, 80% whisper accuracy |
278
+ | 0.2.0 | trimmed features, extended low band, reverb augmentation; 100% whisper, 83.2% voiced |
279
+
280
+ ## Reference
281
+
282
+ Part of a series of small tools built in one session:
283
+
284
+ | tool | reads | answers |
285
+ |---|---|---|
286
+ | `ir-source-localizer` | impulse responses | where is the source? |
287
+ | `acoustic-lidar` | impulse responses + known source | where are the reflectors? |
288
+ | `echo-lineage` | quiet frames | where was it recorded? |
289
+ | `noise-color` | the noise floor | what kind of noise is this? |
290
+ | **`whisper-decoder`** | **a whispered word** | **which word is it?** |
291
+
292
+ The tools share one principle: a signal carries information in
293
+ the parts that the standard pipeline throws away. For
294
+ `whisper-decoder`, the throwaway is the source. The formants
295
+ are the signal.
296
+
297
+ ## License
298
+
299
+ Apache-2.0