lapwing: pronunciation assessment and transcription, based on Parakeet TDT 0.6B v2

This model adds a phone-recognition head to NVIDIA's parakeet-tdt-0.6b-v2 for pronunciation assessment of English learners. One encoder pass over a 16 kHz mono recording gives:

1. Phone posteriors: log_probs [T, 40] at 20 ms frames (CTC: blank + 39 ARPAbet phones, no stress), for goodness-of-pronunciation (GOP) scoring;
2. Encoder outputs for the stock TDT decoder and joiner, which give a cased, punctuated transcript for "did they read the right sentence" checks only.

It is the accurate option. Its sibling GersonHMG/lapwing-small is four times smaller. (Internal name: parakeet.)

Parameters	628M: encoder + phone head 619.0M, decoder 7.2M, joiner 1.7M
Changes to the base	encoder frozen (unchanged); new phone head
Phone head	softmax layer mix of 25 hidden states → LayerNorm → ConvTranspose ×4 (80 → 20 ms) → BiLSTM 2 × 320 → Linear 640→40 (10.1M)
Runtime size	685 MB (uint8 encoder + int8 decoder/joiner)
Latency, 5 s clip, 1 CPU thread	0.24 s
Max input	no fixed limit; evaluated up to 20 s (memory grows with length²)

Files

File	What it is
model.uint8.onnx	samples [1,N] → outputs [1,1024,T80], encoded_lengths [1], log_probs [1,T,40]
decoder.int8.onnx	stock TDT prediction network (2-layer LSTM, 640)
joiner.int8.onnx	stock TDT joint (1,024 tokens + blank + 5 durations = 1,030 logits)
tokens.txt	phone table
text-tokens.txt	SentencePiece pieces
tdt_meta.json	durations, blank id, LSTM dims
gate.json	gate floor and its calibration statistics
model.safetensors	the fp32 weights of all three graphs, for reference and parameter counting. Tensor names are the ONNX initializer names, prefixed model/, decoder/ or joiner/. The runtime does not need this file.
SHA256SUMS	checksums

Phone ids: <blk> 0, AA 1, AE 2, AH 3, AO 4, AW 5, AY 6, B 7, CH 8, D 9, DH 10, EH 11, ER 12, EY 13, F 14, G 15, HH 16, IH 17, IY 18, JH 19, K 20, L 21, M 22, N 23, NG 24, OW 25, OY 26, P 27, R 28, S 29, SH 30, T 31, TH 32, UH 33, UW 34, V 35, W 36, Y 37, Z 38, ZH 39.

Architecture

samples [1,N] ─► NeMo log-mel (in graph): pre-emphasis 0.97, n_fft 512, 128 mel, per-feature normalisation
              ─► FastConformer encoder: subsampling ×8 (80 ms frames), 24 conformer layers, d 1024
                   ├─ 25 hidden states ─► softmax-weighted sum ─► LayerNorm ─► ConvTranspose ×4 (20 ms)
                   │       ─► BiLSTM 2×320 ─► Linear ─► log_probs [1,T,40]
                   └─ last layer ─► outputs [1,1024,T80], encoded_lengths   (same as sherpa-onnx's export)

The upsampling exists because at 80 ms there are only about 1.3 encoder frames per phone, too few for CTC on short utterances.

Usage notes

• Input: float32 [1, N], 16 kHz mono, range [−1, 1]. Mel features are computed inside the graph.
• Requires onnxruntime ≥ 1.20. Tested with 1.23 and 1.28, CPU.
• Text decoding: NeMo greedy TDT (max_symbols 10, durations [0,1,2,3,4], blank 1024). The decoder takes (targets int32 [1,1], target_length int32 [1], h, c) with zero states [2,1,640]. A blank never stays on the same frame.
• Gate: reject an attempt when the CTC log-likelihood of the target phones per frame, L(c)/T, is below −0.489 (gate.json). It was fitted to accept 2.3% of wrong-sentence attempts.
• GOP colour thresholds are not calibrated. Only the gate and word ranking (which word is worst) were validated.

Results

Speaker-disjoint split; the numbers are on the held-out half. "L2-B" is an L2-ARCTIC sentence group no model trained on.

Metric	lapwing
Phone error rate, L2-ARCTIC B / all	16.6% / 15.4%
Mispronunciation detection F1 (P / R)	0.541 (0.58 / 0.51)
GOP phone AUC, L2-B	0.854
Correlation with expert scores (speechocean762), phone / word	0.553 / 0.508
Word ranking AUC, speechocean762 / L2-B	0.902 / 0.796
Phone error rate, native read speech (LibriSpeech)	1.7%
Text WER LibriSpeech / speechocean762 / L2-ARCTIC	1.8% / 22.2% / 5.8%

The uint8 encoder costs about 0.5 phone-error points against int8; its transcripts are the same or better.

Training data

• LibriSpeech train-clean-100 (CC BY 4.0)
• L2-ARCTIC (CC BY-NC 4.0): Arabic, Mandarin, Hindi, Korean, Spanish and Vietnamese first languages
• speechocean762 (CC BY 4.0): Mandarin first language, partly children

Only the phone head was trained. Read speech only: expect weaker results on spontaneous speech, other accents, noise and far-field audio.

Licence and attribution

• Non-commercial: CC BY-NC 4.0. The phone head was trained on L2-ARCTIC, which is CC BY-NC 4.0. Get legal advice before any commercial use.
• Built on "Parakeet TDT 0.6B v2" by NVIDIA (nvidia/parakeet-tdt-0.6b-v2, CC BY 4.0).