Add base-short runtime models, weights and model card

README.md

@@ -0,0 +1,116 @@
+---
+language:
+- en
+license: cc-by-nc-4.0
+library_name: onnx
+pipeline_tag: automatic-speech-recognition
+base_model: openai/whisper-base
+base_model_relation: finetune
+datasets:
+- openslr/librispeech_asr
+- mispeech/speechocean762
+tags:
+- pronunciation-assessment
+- goodness-of-pronunciation
+- phoneme-recognition
+- mispronunciation-detection
+- arpabet
+- ctc
+- whisper
+- onnx
+- int8
+---
+
+# base-short: pronunciation assessment from whisper-base
+
+This is a fine-tune of [openai/whisper-base](https://huggingface.co/openai/whisper-base) for **pronunciation
+assessment of English learners**. One encoder pass over a 16 kHz mono recording gives:
+
+1. **Phone posteriors**: `log_probs [T, 40]` at 20 ms frames (CTC: blank + 39 ARPAbet phones, no stress), for
+   goodness-of-pronunciation (GOP) scoring;
+2. **Cross-attention keys/values** for the stock whisper-base decoder, which gives a transcript for "did they read
+   the right sentence" checks only.
+
+It is the small option. Its sibling
+[GersonHMG/pronunciation-parakeet](https://huggingface.co/GersonHMG/pronunciation-parakeet) is more accurate but
+four times larger.
+
+| | |
+|---|---|
+| Parameters | 77.2M: encoder + phone head 28.4M, decoder 48.9M |
+| Changes to the base | LoRA (rank 16, α 32) on q/v of the top 3 of 6 encoder layers, merged; new phone head |
+| Phone head | softmax layer mix of 7 hidden states → LayerNorm → BiLSTM 2 × 320 → Linear 640→40 (4.7M) |
+| Runtime size | 168 MB (int8 ONNX) |
+| Latency, 5 s clip, 1 CPU thread | 0.32 s |
+| Max input | 30 s |
+
+## Files
+
+| File | What it is |
+|---|---|
+| `model.int8.onnx` | samples `[1,N]` → `log_probs [1,T,40]`, `n_layer_cross_k/v [6,1,1500,512]` |
+| `decoder.int8.onnx` | stock whisper-base decoder (sherpa-onnx export format, self-attention KV cache) |
+| `tokens.txt` | phone table |
+| `whisper-tokens.txt` | Whisper vocabulary (base64 bytes → id) |
+| `whisper_meta.json` | special token ids, decoder dims |
+| `gate.json` | gate floor and its calibration statistics |
+| `model.safetensors` | the fp32 weights of both graphs, for reference and parameter counting. Tensor names are the ONNX initializer names, prefixed `model/` or `decoder/`. The runtime does not need this file. |
+| `SHA256SUMS` | checksums |
+
+Phone ids: `<blk> 0, AA 1, AE 2, AH 3, AO 4, AW 5, AY 6, B 7, CH 8, D 9, DH 10, EH 11, ER 12, EY 13, F 14, G 15,
+HH 16, IH 17, IY 18, JH 19, K 20, L 21, M 22, N 23, NG 24, OW 25, OY 26, P 27, R 28, S 29, SH 30, T 31, TH 32,
+UH 33, UW 34, V 35, W 36, Y 37, Z 38, ZH 39`.
+
+## Architecture
+
+```
+samples [1,N] ─► loudness normalisation ─► Whisper log-mel, 80 bins, at the real length (no 30 s padding)
+              ─► whisper-base encoder (2 conv + 6 transformer layers, d 512, 20 ms frames)
+                   ├─ 7 hidden states ─► softmax-weighted sum ─► LayerNorm ─► BiLSTM 2×320 ─► Linear ─► log_probs
+                   └─ last state + constant 30 s "silence tail" to 1500 frames ─► cross k/v for the decoder
+```
+
+Whisper's decoder was trained only on 30 s windows, so the graph appends the encoder output of silence after the
+real frames. Without it the decoder never stops and repeats the sentence.
+
+## Usage notes
+
+- Input: float32 `[1, N]`, 16 kHz mono, range [−1, 1]. Loudness and mel features are computed inside the graph.
+- Requires onnxruntime ≥ 1.20. Tested with 1.23 and 1.28, CPU.
+- Text decoding: greedy Whisper with prompt `[50258, 50259, 50359, 50363]`. Suppress `no_timestamps`, `sot`,
+  `no_speech`, `translate` (and `eot` and token 220 at step 0). Stop at `<|endoftext|>`, when the output starts
+  repeating a block, or at `max(8, 6·seconds) + 8` tokens.
+- Gate: reject an attempt when the CTC log-likelihood of the target phones per frame, `L(c)/T`, is below
+  **−0.374** (`gate.json`). It was fitted to accept 2.3% of wrong-sentence attempts.
+- **GOP colour thresholds are not calibrated.** Only the gate and word ranking (which word is worst) were validated.
+
+## Results
+
+Speaker-disjoint split; the numbers are on the held-out half. "L2-B" is an L2-ARCTIC sentence group no model trained
+on.
+
+| Metric | base-short |
+|---|---|
+| Phone error rate, L2-ARCTIC B / all | 20.7% / 19.1% |
+| Mispronunciation detection F1 (P / R) | 0.505 (0.49 / 0.52) |
+| GOP phone AUC, L2-B | 0.826 |
+| Correlation with expert scores (speechocean762), phone / word | 0.493 / 0.410 |
+| Word ranking AUC, speechocean762 / L2-B | 0.912 / 0.764 |
+| Phone error rate, native read speech (LibriSpeech) | 4.9% |
+| Text WER LibriSpeech / speechocean762 / L2-ARCTIC | 6.2% / 35.0% / 16.2% |
+
+The int8 ONNX file scores within about 1 point of the PyTorch model on phone error rate and WER.
+
+## Training data
+
+- LibriSpeech train-clean-100 (CC BY 4.0)
+- L2-ARCTIC (CC BY-NC 4.0): Arabic, Mandarin, Hindi, Korean, Spanish and Vietnamese first languages
+- speechocean762 (CC BY 4.0): Mandarin first language, partly children
+
+Read speech only. Expect weaker results on spontaneous speech, other accents, noise and far-field audio.
+
+## Licence and attribution
+
+- **Non-commercial: CC BY-NC 4.0.** The phone head and LoRA were trained on L2-ARCTIC, which is CC BY-NC 4.0.
+  Get legal advice before any commercial use.
+- Built on OpenAI's [whisper-base](https://huggingface.co/openai/whisper-base) (MIT).

SHA256SUMS

@@ -0,0 +1,7 @@
+231a81a6646b0971fa6f2376cb375e38ef047c350f912a5d4c91837f39e06c84 *decoder.int8.onnx
+54e89942b1bdbf22984ebfb5089ff48bb179eaada020cbb8e7630e93eefdf498 *gate.json
+a8fbb75f2dfc0a8578c6cfdee1ce99286ba21ab100a646440ffcc40e076ef5a3 *model.int8.onnx
+9c3c66aefd4df716adedff0d08ce9967a3469e72e4e2f00d4ff3a49dd3c983ac *tokens.txt
+b34b360dbb493e781e479794586d661700670d65564001f23024971d1f2fa126 *whisper-tokens.txt
+20342b8dc31a77da5ff398ab47f78819758c9c798763ad3cedb1d4119ff2864a *whisper_meta.json
+52326ef644a2a9a083652d4f69ca0fdf82c4b89c5757e6bcb179c91a3bdaef0b *model.safetensors

decoder.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:231a81a6646b0971fa6f2376cb375e38ef047c350f912a5d4c91837f39e06c84
+size 130659024

gate.json

@@ -0,0 +1,28 @@
+{
+ "name": "bs3c",
+ "floor": -0.37440954125091597,
+ "wrong_accepted_target": 0.023,
+ "wrong_accepted": 0.02315886984715146,
+ "genuine_refused_so762": 0.011111111111111112,
+ "genuine_q01_q05_median_so762": [
+  -0.39143950194824345,
+  -0.20738314454544318,
+  -0.07671881881977066
+ ],
+ "wrong_q50_q95_q99_so762": [
+  -0.9105667563230115,
+  -0.4072960283186184,
+  -0.28927623283639387
+ ],
+ "genuine_refused_l2arctic": 0.008898776418242492,
+ "genuine_q01_q05_median_l2arctic": [
+  -0.36474409736742813,
+  -0.25954746352806635,
+  -0.08572272415999528
+ ],
+ "wrong_q50_q95_q99_l2arctic": [
+  -1.5960055396670387,
+  -0.9300887775372075,
+  -0.718967869081176
+ ]
+}

model.int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a8fbb75f2dfc0a8578c6cfdee1ce99286ba21ab100a646440ffcc40e076ef5a3
+size 37736054

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52326ef644a2a9a083652d4f69ca0fdf82c4b89c5757e6bcb179c91a3bdaef0b
+size 308899204

tokens.txt

@@ -0,0 +1,40 @@
+<blk> 0
+AA 1
+AE 2
+AH 3
+AO 4
+AW 5
+AY 6
+B 7
+CH 8
+D 9
+DH 10
+EH 11
+ER 12
+EY 13
+F 14
+G 15
+HH 16
+IH 17
+IY 18
+JH 19
+K 20
+L 21
+M 22
+N 23
+NG 24
+OW 25
+OY 26
+P 27
+R 28
+S 29
+SH 30
+T 31
+TH 32
+UH 33
+UW 34
+V 35
+W 36
+Y 37
+Z 38
+ZH 39

whisper_meta.json

@@ -0,0 +1,15 @@
+{
+ "n_text_layer": 6,
+ "n_text_state": 512,
+ "n_text_ctx": 448,
+ "n_text_head": 8,
+ "sot": 50258,
+ "eot": 50257,
+ "blank_id": 220,
+ "no_speech": 50362,
+ "transcribe": 50359,
+ "translate": 50358,
+ "no_timestamps": 50363,
+ "n_vocab": 51865,
+ "en": 50259
+}