henryNdubuaku commited on
Commit
ddf86ff
·
verified ·
1 Parent(s): 32c2de9

Three-model comparison, plainer footnotes

Browse files
Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -39,15 +39,15 @@ Whistle is a log-mel front end and a convolutional stem feeding an audio encoder
39
 
40
  ## Benchmarks
41
 
42
- Word error rate on the full test splits, scored with the Whisper normalizers; Whisper and Moonshine figures are the ones their authors published.
43
 
44
  ![Whistle against Whisper and Moonshine](assets/whistle-benchmarks.svg)
45
 
46
- - **Speed**: the latency and the decode rate are for 10 s of audio on an Apple M4 Pro, from the default settings of each model's official engine: Whistle's C++ engine at 5 beams, `openai-whisper`, and `moonshine-voice` non-streaming over whole audio. **Time to first token** is audio in to first token, and **decode** is tokens divided by the wall time after it, so encoder cost is not counted twice.
47
  - **Precision**: Whistle 2 to 4 bit, Whisper fp32 in memory on the CPU, Moonshine int8.
48
- - **WER**: Whistle measured over 86,174 utterances; Whisper and Moonshine as published (Whisper paper Tables 9, 10 and 13, Moonshine paper Table 3). The Whisper rows are the multilingual checkpoints, not `tiny.en` or `base.en`.
49
- - **A dash** means that model's authors never published that benchmark. Moonshine is English only; Whisper reports no SPGISpeech, Earnings-22 or AMI cleaned.
50
- - **AMI** includes empty references, per the Open ASR Leaderboard convention; Whisper's figure is AMI-IHM. **TED-LIUM** excludes the corpus's `ignore_time_segment_in_scoring` regions, as its scoring protocol specifies.
51
  - **FLEURS and MLS** are averaged over Whistle's seven languages (MLS over six, no English), the same set for every model.
52
  - **No test audio appears in Whistle's training or validation data**, verified by comparing audio checksums and speaker IDs across every reported test set.
53
 
 
39
 
40
  ## Benchmarks
41
 
42
+ Word error rate on the full test splits, scored with the Whisper normalizers. Whisper and Moonshine figures are the ones their authors published.
43
 
44
  ![Whistle against Whisper and Moonshine](assets/whistle-benchmarks.svg)
45
 
46
+ - **Speed**: 10 s of audio on an Apple M4 Pro, each model on its official runtime at its defaults: Whistle's C++ engine at 5 beams, `openai-whisper`, and `moonshine-voice` non-streaming over whole audio. Time to first token is audio in to first token. Decode is tokens divided by the wall time after it, so the encoder is not counted twice.
47
  - **Precision**: Whistle 2 to 4 bit, Whisper fp32 in memory on the CPU, Moonshine int8.
48
+ - **WER**: Whistle measured over 86,174 utterances. Whisper and Moonshine as published (Whisper paper Tables 9, 10 and 13, Moonshine paper Table 3). The Whisper row is the multilingual checkpoint, not `base.en`.
49
+ - **A missing bar** means that model's authors never published that benchmark. Moonshine is English only; Whisper reports no SPGISpeech, Earnings-22 or AMI cleaned.
50
+ - **AMI** includes empty references, per the Open ASR Leaderboard convention. Whisper's figure is AMI-IHM. **TED-LIUM** excludes the corpus's `ignore_time_segment_in_scoring` regions, as its scoring protocol specifies.
51
  - **FLEURS and MLS** are averaged over Whistle's seven languages (MLS over six, no English), the same set for every model.
52
  - **No test audio appears in Whistle's training or validation data**, verified by comparing audio checksums and speaker IDs across every reported test set.
53