lapwing / README.md
GersonHMG's picture
Add model image
9d25ab1 verified
|
Raw History Blame Contribute Delete
5.7 kB
metadata
language:
  - en
license: cc-by-nc-4.0
library_name: onnx
pipeline_tag: automatic-speech-recognition
base_model: nvidia/parakeet-tdt-0.6b-v2
base_model_relation: finetune
datasets:
  - openslr/librispeech_asr
  - mispeech/speechocean762
tags:
  - pronunciation-assessment
  - goodness-of-pronunciation
  - phoneme-recognition
  - mispronunciation-detection
  - arpabet
  - ctc
  - parakeet
  - fastconformer
  - tdt
  - onnx
  - uint8

lapwing: pronunciation assessment and transcription, based on Parakeet TDT 0.6B v2

lapwing: a pixel-art lapwing flying over a forest pond

This model adds a phone-recognition head to NVIDIA's parakeet-tdt-0.6b-v2 for pronunciation assessment of English learners. One encoder pass over a 16 kHz mono recording gives:

  1. Phone posteriors: log_probs [T, 40] at 20 ms frames (CTC: blank + 39 ARPAbet phones, no stress), for goodness-of-pronunciation (GOP) scoring;
  2. Encoder outputs for the stock TDT decoder and joiner, which give a cased, punctuated transcript for "did they read the right sentence" checks only.

It is the accurate option. Its sibling GersonHMG/lapwing-small is four times smaller. (Internal name: parakeet.)

Parameters 628M: encoder + phone head 619.0M, decoder 7.2M, joiner 1.7M
Changes to the base encoder frozen (unchanged); new phone head
Phone head softmax layer mix of 25 hidden states → LayerNorm → ConvTranspose ×4 (80 → 20 ms) → BiLSTM 2 × 320 → Linear 640→40 (10.1M)
Runtime size 685 MB (uint8 encoder + int8 decoder/joiner)
Latency, 5 s clip, 1 CPU thread 0.24 s
Max input no fixed limit; evaluated up to 20 s (memory grows with length²)

Files

File What it is
model.uint8.onnx samples [1,N] → outputs [1,1024,T80], encoded_lengths [1], log_probs [1,T,40]
decoder.int8.onnx stock TDT prediction network (2-layer LSTM, 640)
joiner.int8.onnx stock TDT joint (1,024 tokens + blank + 5 durations = 1,030 logits)
tokens.txt phone table
text-tokens.txt SentencePiece pieces
tdt_meta.json durations, blank id, LSTM dims
gate.json gate floor and its calibration statistics
model.safetensors the fp32 weights of all three graphs, for reference and parameter counting. Tensor names are the ONNX initializer names, prefixed model/, decoder/ or joiner/. The runtime does not need this file.
SHA256SUMS checksums

Phone ids: <blk> 0, AA 1, AE 2, AH 3, AO 4, AW 5, AY 6, B 7, CH 8, D 9, DH 10, EH 11, ER 12, EY 13, F 14, G 15, HH 16, IH 17, IY 18, JH 19, K 20, L 21, M 22, N 23, NG 24, OW 25, OY 26, P 27, R 28, S 29, SH 30, T 31, TH 32, UH 33, UW 34, V 35, W 36, Y 37, Z 38, ZH 39.

Architecture

samples [1,N] ─► NeMo log-mel (in graph): pre-emphasis 0.97, n_fft 512, 128 mel, per-feature normalisation
              ─► FastConformer encoder: subsampling ×8 (80 ms frames), 24 conformer layers, d 1024
                   ├─ 25 hidden states ─► softmax-weighted sum ─► LayerNorm ─► ConvTranspose ×4 (20 ms)
                   │       ─► BiLSTM 2×320 ─► Linear ─► log_probs [1,T,40]
                   └─ last layer ─► outputs [1,1024,T80], encoded_lengths   (same as sherpa-onnx's export)

The upsampling exists because at 80 ms there are only about 1.3 encoder frames per phone, too few for CTC on short utterances.

Usage notes

  • Input: float32 [1, N], 16 kHz mono, range [−1, 1]. Mel features are computed inside the graph.
  • Requires onnxruntime ≥ 1.20. Tested with 1.23 and 1.28, CPU.
  • Text decoding: NeMo greedy TDT (max_symbols 10, durations [0,1,2,3,4], blank 1024). The decoder takes (targets int32 [1,1], target_length int32 [1], h, c) with zero states [2,1,640]. A blank never stays on the same frame.
  • Gate: reject an attempt when the CTC log-likelihood of the target phones per frame, L(c)/T, is below −0.489 (gate.json). It was fitted to accept 2.3% of wrong-sentence attempts.
  • GOP colour thresholds are not calibrated. Only the gate and word ranking (which word is worst) were validated.

Results

Speaker-disjoint split; the numbers are on the held-out half. "L2-B" is an L2-ARCTIC sentence group no model trained on.

Metric lapwing
Phone error rate, L2-ARCTIC B / all 16.6% / 15.4%
Mispronunciation detection F1 (P / R) 0.541 (0.58 / 0.51)
GOP phone AUC, L2-B 0.854
Correlation with expert scores (speechocean762), phone / word 0.553 / 0.508
Word ranking AUC, speechocean762 / L2-B 0.902 / 0.796
Phone error rate, native read speech (LibriSpeech) 1.7%
Text WER LibriSpeech / speechocean762 / L2-ARCTIC 1.8% / 22.2% / 5.8%

The uint8 encoder costs about 0.5 phone-error points against int8; its transcripts are the same or better.

Training data

  • LibriSpeech train-clean-100 (CC BY 4.0)
  • L2-ARCTIC (CC BY-NC 4.0): Arabic, Mandarin, Hindi, Korean, Spanish and Vietnamese first languages
  • speechocean762 (CC BY 4.0): Mandarin first language, partly children

Only the phone head was trained. Read speech only: expect weaker results on spontaneous speech, other accents, noise and far-field audio.

Licence and attribution

  • Non-commercial: CC BY-NC 4.0. The phone head was trained on L2-ARCTIC, which is CC BY-NC 4.0. Get legal advice before any commercial use.
  • Built on "Parakeet TDT 0.6B v2" by NVIDIA (nvidia/parakeet-tdt-0.6b-v2, CC BY 4.0).