---
language:
- en
license: cc-by-nc-4.0
library_name: onnx
pipeline_tag: automatic-speech-recognition
base_model: nvidia/parakeet-tdt-0.6b-v2
base_model_relation: finetune
datasets:
- openslr/librispeech_asr
- mispeech/speechocean762
tags:
- pronunciation-assessment
- goodness-of-pronunciation
- phoneme-recognition
- mispronunciation-detection
- arpabet
- ctc
- parakeet
- fastconformer
- tdt
- onnx
- uint8
---
# lapwing: pronunciation assessment and transcription, based on Parakeet TDT 0.6B v2

This model adds a phone-recognition head to NVIDIA's
[parakeet-tdt-0.6b-v2](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) for **pronunciation assessment of
English learners**. One encoder pass over a 16 kHz mono recording gives:
1. **Phone posteriors**: `log_probs [T, 40]` at 20 ms frames (CTC: blank + 39 ARPAbet phones, no stress), for
goodness-of-pronunciation (GOP) scoring;
2. **Encoder outputs** for the stock TDT decoder and joiner, which give a cased, punctuated transcript for "did they
read the right sentence" checks only.
It is the accurate option. Its sibling [GersonHMG/lapwing-small](https://huggingface.co/GersonHMG/lapwing-small) is
four times smaller. (Internal name: parakeet.)
| | |
|---|---|
| Parameters | 628M: encoder + phone head 619.0M, decoder 7.2M, joiner 1.7M |
| Changes to the base | encoder frozen (unchanged); new phone head |
| Phone head | softmax layer mix of 25 hidden states → LayerNorm → ConvTranspose ×4 (80 → 20 ms) → BiLSTM 2 × 320 → Linear 640→40 (10.1M) |
| Runtime size | 685 MB (uint8 encoder + int8 decoder/joiner) |
| Latency, 5 s clip, 1 CPU thread | 0.24 s |
| Max input | no fixed limit; evaluated up to 20 s (memory grows with length²) |
## Files
| File | What it is |
|---|---|
| `model.uint8.onnx` | samples `[1,N]` → `outputs [1,1024,T80]`, `encoded_lengths [1]`, `log_probs [1,T,40]` |
| `decoder.int8.onnx` | stock TDT prediction network (2-layer LSTM, 640) |
| `joiner.int8.onnx` | stock TDT joint (1,024 tokens + blank + 5 durations = 1,030 logits) |
| `tokens.txt` | phone table |
| `text-tokens.txt` | SentencePiece pieces |
| `tdt_meta.json` | durations, blank id, LSTM dims |
| `gate.json` | gate floor and its calibration statistics |
| `model.safetensors` | the fp32 weights of all three graphs, for reference and parameter counting. Tensor names are the ONNX initializer names, prefixed `model/`, `decoder/` or `joiner/`. The runtime does not need this file. |
| `SHA256SUMS` | checksums |
Phone ids: ` 0, AA 1, AE 2, AH 3, AO 4, AW 5, AY 6, B 7, CH 8, D 9, DH 10, EH 11, ER 12, EY 13, F 14, G 15,
HH 16, IH 17, IY 18, JH 19, K 20, L 21, M 22, N 23, NG 24, OW 25, OY 26, P 27, R 28, S 29, SH 30, T 31, TH 32,
UH 33, UW 34, V 35, W 36, Y 37, Z 38, ZH 39`.
## Architecture
```
samples [1,N] ─► NeMo log-mel (in graph): pre-emphasis 0.97, n_fft 512, 128 mel, per-feature normalisation
─► FastConformer encoder: subsampling ×8 (80 ms frames), 24 conformer layers, d 1024
├─ 25 hidden states ─► softmax-weighted sum ─► LayerNorm ─► ConvTranspose ×4 (20 ms)
│ ─► BiLSTM 2×320 ─► Linear ─► log_probs [1,T,40]
└─ last layer ─► outputs [1,1024,T80], encoded_lengths (same as sherpa-onnx's export)
```
The upsampling exists because at 80 ms there are only about 1.3 encoder frames per phone, too few for CTC on short
utterances.
## Usage notes
- Input: float32 `[1, N]`, 16 kHz mono, range [−1, 1]. Mel features are computed inside the graph.
- Requires onnxruntime ≥ 1.20. Tested with 1.23 and 1.28, CPU.
- Text decoding: NeMo greedy TDT (`max_symbols` 10, durations `[0,1,2,3,4]`, blank 1024). The decoder takes
`(targets int32 [1,1], target_length int32 [1], h, c)` with zero states `[2,1,640]`. A blank never stays on the
same frame.
- Gate: reject an attempt when the CTC log-likelihood of the target phones per frame, `L(c)/T`, is below
**−0.489** (`gate.json`). It was fitted to accept 2.3% of wrong-sentence attempts.
- **GOP colour thresholds are not calibrated.** Only the gate and word ranking (which word is worst) were validated.
## Results
Speaker-disjoint split; the numbers are on the held-out half. "L2-B" is an L2-ARCTIC sentence group no model trained
on.
| Metric | lapwing |
|---|---|
| Phone error rate, L2-ARCTIC B / all | 16.6% / 15.4% |
| Mispronunciation detection F1 (P / R) | 0.541 (0.58 / 0.51) |
| GOP phone AUC, L2-B | 0.854 |
| Correlation with expert scores (speechocean762), phone / word | 0.553 / 0.508 |
| Word ranking AUC, speechocean762 / L2-B | 0.902 / 0.796 |
| Phone error rate, native read speech (LibriSpeech) | 1.7% |
| Text WER LibriSpeech / speechocean762 / L2-ARCTIC | 1.8% / 22.2% / 5.8% |
The uint8 encoder costs about 0.5 phone-error points against int8; its transcripts are the same or better.
## Training data
- LibriSpeech train-clean-100 (CC BY 4.0)
- L2-ARCTIC (CC BY-NC 4.0): Arabic, Mandarin, Hindi, Korean, Spanish and Vietnamese first languages
- speechocean762 (CC BY 4.0): Mandarin first language, partly children
Only the phone head was trained. Read speech only: expect weaker results on spontaneous speech, other accents, noise
and far-field audio.
## Licence and attribution
- **Non-commercial: CC BY-NC 4.0.** The phone head was trained on L2-ARCTIC, which is CC BY-NC 4.0. Get legal
advice before any commercial use.
- Built on "Parakeet TDT 0.6B v2" by NVIDIA ([nvidia/parakeet-tdt-0.6b-v2](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2), CC BY 4.0).