wav2vec2-base-phones-en

This model hears English phones and stands in for facebook/wav2vec2-lv-60-espeak-cv-ft in OpenPronounce, where it scores how a learner reads a sentence aloud. It is a wav2vec2-base (95 M parameters, a third of the large model) distilled from two large models, and it gives the library the two things it reads: phone posteriors over the 392 espeak tokens of the large model, and a stand-in for that model's layer 20, which feeds the word model.

pip install 'openpronounce>=1.1'
export OPENPRONOUNCE_PHONEME_MODEL=OpenPronounce/wav2vec2-base-phones-en
openpronounce recording.wav "the sentence that was read" --fast

From version 1.1, OpenPronounce then loads the word model and the scorer trained on this model's output (word_mdd_en_student.json, scorer_en_student.json, shipped in the package). Transcription still runs on facebook/wav2vec2-large-960h: free speech and the full mode load it as before, and only --fast (or transcription=False) gets by with this model alone.

Files

model.safetensors and config.json hold a plain Wav2Vec2ForCTC with the phone head; vocab.json maps its ids to phones. hidden_projection.safetensors is a linear map from 768 to 1024 dimensions applied to hidden_states[10], and the openpronounce block of config.json names it. char_head.safetensors is a CTC head over the 32 characters of wav2vec2-large-960h, which nothing uses yet: its greedy transcripts miss more words than the large model's (word error rate 0.54 against 0.48 on 200 test utterances).

onnx/model.onnx (fp32, 382 MB) and onnx/model_int8.onnx (dynamic int8 on the matrix products, 123 MB) run the three heads at once, in ONNX Runtime or in a browser. Their input input_values is the 16 kHz waveform of shape (1, samples), normalized to zero mean and unit variance by the caller. Their outputs are logits (1, frames, 392), hidden20 (1, frames, 1024) and asr_logits (1, frames, 32), one frame per 20 ms.

Training

No label was used. Both teachers listened to the same crops as the student, which learned to match the phone posteriors of facebook/wav2vec2-lv-60-espeak-cv-ft, the character posteriors of facebook/wav2vec2-large-960h (weight 0.5), and the layer 20 of the first teacher through its projection (L1 plus cosine distance). Training ran 21,193 steps on batches of 96 seconds of audio cut into crops of at most 8 seconds, with AdamW, a learning rate of 1e-4 (1e-3 for the heads) on a cosine schedule and the convolutional encoder frozen: an hour and forty minutes on one RTX 5070 Ti.

The audio adds up to 32 hours:

  • 27,065 English clips of Common Voice 27 (29.5 h, CC0), with native and non-native accents;
  • the 2,498 recordings of the speechocean762 train split (2.6 h), without their ratings. speechocean762 is published under CC BY 4.0 by Junbo Zhang et al., "speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment", Interspeech 2021.

The teachers and facebook/wav2vec2-base are under the Apache 2.0 licence.

Results

Measured on the 2,500 recordings of the speechocean762 test split, whose speakers are absent from the train split, through the library (benchmarks/word_model.py and benchmarks/scorer.py). The word AUC says how well the word model finds the words the raters scored below 5 out of 10, the Pearson correlations how closely the scores follow theirs.

phone model word AUC total accuracy fluency prosody
wav2vec2-lv-60-espeak-cv-ft, shipped models 0.928 0.790 0.743 0.827 0.808
this model, its own models 0.938 0.802 0.766 0.837 0.821
this model in int8 ONNX, same models 0.939 0.798 0.759 0.837 0.820

On a laptop CPU (Ryzen 7 PRO 7840U), assess() without transcription takes 2.8 s on an 18-second recording with eight threads, against 6.2 s with the large model, and 4.0 s against 10.9 s on a single thread.

Limits

It is meant for English read aloud. The phone vocabulary is multilingual, but the model only heard English, and nothing was measured in another language: OpenPronounce's models for French, Spanish and the others were trained on the large model's output.

Free speech keeps the scorer trained on the large model's output. With this model it follows the raters a little less closely on the same test split, prompts hidden (Pearson 0.699 with the total, against 0.708 published for the large model).

The model does better than its teacher on speechocean762, and I doubt it would elsewhere. It heard the audio of the train split, the same population as the test split (Mandarin-speaking learners, many of them children, recorded for the same corpus), and its scorers were fitted on its own output. My hypothesis is that this adaptation explains the gain. It has not been tested on other learners, so expect the large model's results at best.

Licence

AGPL-3.0-or-later, like OpenPronounce. A commercial licence covers closed products: see LICENSING.md and the prices.

Downloads last month
86
Safetensors
Model size
94.7M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for OpenPronounce/wav2vec2-base-phones-en

Quantized
(7)
this model

Dataset used to train OpenPronounce/wav2vec2-base-phones-en