Three-model comparison, plainer footnotes
Browse files
README.md
CHANGED
|
@@ -39,15 +39,15 @@ Whistle is a log-mel front end and a convolutional stem feeding an audio encoder
|
|
| 39 |
|
| 40 |
## Benchmarks
|
| 41 |
|
| 42 |
-
Word error rate on the full test splits, scored with the Whisper normalizers
|
| 43 |
|
| 44 |

|
| 45 |
|
| 46 |
-
- **Speed**:
|
| 47 |
- **Precision**: Whistle 2 to 4 bit, Whisper fp32 in memory on the CPU, Moonshine int8.
|
| 48 |
-
- **WER**: Whistle measured over 86,174 utterances
|
| 49 |
-
- **A
|
| 50 |
-
- **AMI** includes empty references, per the Open ASR Leaderboard convention
|
| 51 |
- **FLEURS and MLS** are averaged over Whistle's seven languages (MLS over six, no English), the same set for every model.
|
| 52 |
- **No test audio appears in Whistle's training or validation data**, verified by comparing audio checksums and speaker IDs across every reported test set.
|
| 53 |
|
|
|
|
| 39 |
|
| 40 |
## Benchmarks
|
| 41 |
|
| 42 |
+
Word error rate on the full test splits, scored with the Whisper normalizers. Whisper and Moonshine figures are the ones their authors published.
|
| 43 |
|
| 44 |

|
| 45 |
|
| 46 |
+
- **Speed**: 10 s of audio on an Apple M4 Pro, each model on its official runtime at its defaults: Whistle's C++ engine at 5 beams, `openai-whisper`, and `moonshine-voice` non-streaming over whole audio. Time to first token is audio in to first token. Decode is tokens divided by the wall time after it, so the encoder is not counted twice.
|
| 47 |
- **Precision**: Whistle 2 to 4 bit, Whisper fp32 in memory on the CPU, Moonshine int8.
|
| 48 |
+
- **WER**: Whistle measured over 86,174 utterances. Whisper and Moonshine as published (Whisper paper Tables 9, 10 and 13, Moonshine paper Table 3). The Whisper row is the multilingual checkpoint, not `base.en`.
|
| 49 |
+
- **A missing bar** means that model's authors never published that benchmark. Moonshine is English only; Whisper reports no SPGISpeech, Earnings-22 or AMI cleaned.
|
| 50 |
+
- **AMI** includes empty references, per the Open ASR Leaderboard convention. Whisper's figure is AMI-IHM. **TED-LIUM** excludes the corpus's `ignore_time_segment_in_scoring` regions, as its scoring protocol specifies.
|
| 51 |
- **FLEURS and MLS** are averaged over Whistle's seven languages (MLS over six, no English), the same set for every model.
|
| 52 |
- **No test audio appears in Whistle's training or validation data**, verified by comparing audio checksums and speaker IDs across every reported test set.
|
| 53 |
|