--- language: - en license: cc-by-nc-4.0 library_name: onnx pipeline_tag: automatic-speech-recognition base_model: nvidia/parakeet-tdt-0.6b-v2 base_model_relation: finetune datasets: - openslr/librispeech_asr - mispeech/speechocean762 tags: - pronunciation-assessment - goodness-of-pronunciation - phoneme-recognition - mispronunciation-detection - arpabet - ctc - parakeet - fastconformer - tdt - onnx - uint8 --- # lapwing: pronunciation assessment and transcription, based on Parakeet TDT 0.6B v2

lapwing: a pixel-art lapwing flying over a forest pond

This model adds a phone-recognition head to NVIDIA's [parakeet-tdt-0.6b-v2](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) for **pronunciation assessment of English learners**. One encoder pass over a 16 kHz mono recording gives: 1. **Phone posteriors**: `log_probs [T, 40]` at 20 ms frames (CTC: blank + 39 ARPAbet phones, no stress), for goodness-of-pronunciation (GOP) scoring; 2. **Encoder outputs** for the stock TDT decoder and joiner, which give a cased, punctuated transcript for "did they read the right sentence" checks only. It is the accurate option. Its sibling [GersonHMG/lapwing-small](https://huggingface.co/GersonHMG/lapwing-small) is four times smaller. (Internal name: parakeet.) | | | |---|---| | Parameters | 628M: encoder + phone head 619.0M, decoder 7.2M, joiner 1.7M | | Changes to the base | encoder frozen (unchanged); new phone head | | Phone head | softmax layer mix of 25 hidden states → LayerNorm → ConvTranspose ×4 (80 → 20 ms) → BiLSTM 2 × 320 → Linear 640→40 (10.1M) | | Runtime size | 685 MB (uint8 encoder + int8 decoder/joiner) | | Latency, 5 s clip, 1 CPU thread | 0.24 s | | Max input | no fixed limit; evaluated up to 20 s (memory grows with length²) | ## Files | File | What it is | |---|---| | `model.uint8.onnx` | samples `[1,N]` → `outputs [1,1024,T80]`, `encoded_lengths [1]`, `log_probs [1,T,40]` | | `decoder.int8.onnx` | stock TDT prediction network (2-layer LSTM, 640) | | `joiner.int8.onnx` | stock TDT joint (1,024 tokens + blank + 5 durations = 1,030 logits) | | `tokens.txt` | phone table | | `text-tokens.txt` | SentencePiece pieces | | `tdt_meta.json` | durations, blank id, LSTM dims | | `gate.json` | gate floor and its calibration statistics | | `model.safetensors` | the fp32 weights of all three graphs, for reference and parameter counting. Tensor names are the ONNX initializer names, prefixed `model/`, `decoder/` or `joiner/`. The runtime does not need this file. | | `SHA256SUMS` | checksums | Phone ids: ` 0, AA 1, AE 2, AH 3, AO 4, AW 5, AY 6, B 7, CH 8, D 9, DH 10, EH 11, ER 12, EY 13, F 14, G 15, HH 16, IH 17, IY 18, JH 19, K 20, L 21, M 22, N 23, NG 24, OW 25, OY 26, P 27, R 28, S 29, SH 30, T 31, TH 32, UH 33, UW 34, V 35, W 36, Y 37, Z 38, ZH 39`. ## Architecture ``` samples [1,N] ─► NeMo log-mel (in graph): pre-emphasis 0.97, n_fft 512, 128 mel, per-feature normalisation ─► FastConformer encoder: subsampling ×8 (80 ms frames), 24 conformer layers, d 1024 ├─ 25 hidden states ─► softmax-weighted sum ─► LayerNorm ─► ConvTranspose ×4 (20 ms) │ ─► BiLSTM 2×320 ─► Linear ─► log_probs [1,T,40] └─ last layer ─► outputs [1,1024,T80], encoded_lengths (same as sherpa-onnx's export) ``` The upsampling exists because at 80 ms there are only about 1.3 encoder frames per phone, too few for CTC on short utterances. ## Usage notes - Input: float32 `[1, N]`, 16 kHz mono, range [−1, 1]. Mel features are computed inside the graph. - Requires onnxruntime ≥ 1.20. Tested with 1.23 and 1.28, CPU. - Text decoding: NeMo greedy TDT (`max_symbols` 10, durations `[0,1,2,3,4]`, blank 1024). The decoder takes `(targets int32 [1,1], target_length int32 [1], h, c)` with zero states `[2,1,640]`. A blank never stays on the same frame. - Gate: reject an attempt when the CTC log-likelihood of the target phones per frame, `L(c)/T`, is below **−0.489** (`gate.json`). It was fitted to accept 2.3% of wrong-sentence attempts. - **GOP colour thresholds are not calibrated.** Only the gate and word ranking (which word is worst) were validated. ## Results Speaker-disjoint split; the numbers are on the held-out half. "L2-B" is an L2-ARCTIC sentence group no model trained on. | Metric | lapwing | |---|---| | Phone error rate, L2-ARCTIC B / all | 16.6% / 15.4% | | Mispronunciation detection F1 (P / R) | 0.541 (0.58 / 0.51) | | GOP phone AUC, L2-B | 0.854 | | Correlation with expert scores (speechocean762), phone / word | 0.553 / 0.508 | | Word ranking AUC, speechocean762 / L2-B | 0.902 / 0.796 | | Phone error rate, native read speech (LibriSpeech) | 1.7% | | Text WER LibriSpeech / speechocean762 / L2-ARCTIC | 1.8% / 22.2% / 5.8% | The uint8 encoder costs about 0.5 phone-error points against int8; its transcripts are the same or better. ## Training data - LibriSpeech train-clean-100 (CC BY 4.0) - L2-ARCTIC (CC BY-NC 4.0): Arabic, Mandarin, Hindi, Korean, Spanish and Vietnamese first languages - speechocean762 (CC BY 4.0): Mandarin first language, partly children Only the phone head was trained. Read speech only: expect weaker results on spontaneous speech, other accents, noise and far-field audio. ## Licence and attribution - **Non-commercial: CC BY-NC 4.0.** The phone head was trained on L2-ARCTIC, which is CC BY-NC 4.0. Get legal advice before any commercial use. - Built on "Parakeet TDT 0.6B v2" by NVIDIA ([nvidia/parakeet-tdt-0.6b-v2](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2), CC BY 4.0).