Show that WER matches NeMo, with the multilingual results
Browse files
README.md
CHANGED
|
@@ -43,30 +43,39 @@ tags:
|
|
| 43 |
|
| 44 |
# scriptrs models
|
| 45 |
|
| 46 |
-
Model files for [scriptrs](https://github.com/avencera/scriptrs), a Rust library for fast NVIDIA Parakeet TDT v2 and v3 speech-to-text on Core ML (macOS) and native CUDA (Linux). With the default `online` feature, scriptrs downloads these files on first use from a pinned revision of this repository.
|
| 47 |
|
| 48 |
These are format conversions of NVIDIA's models, not new or retrained models. The native CUDA weights are NVIDIA's FP32 weights with their original names; scriptrs converts selected layers to FP16 on the GPU at load time.
|
| 49 |
|
| 50 |
-
##
|
| 51 |
|
| 52 |
-
|
|
|
|
|
|
|
| 53 |
|
| 54 |
| | Speed (x real time) | WER |
|
| 55 |
|---|---|---|
|
| 56 |
-
| scriptrs CUDA
|
| 57 |
| NVIDIA NeMo (PyTorch) | 477x | 1.93 % |
|
| 58 |
| parakeet-rs 0.4.3 (ONNX Runtime) | 253x | 1.93 % |
|
| 59 |
|
| 60 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
|
| 62 |
| Workload | Speed (x real time) | WER |
|
| 63 |
|---|---|---|
|
| 64 |
-
| LibriSpeech test-clean, batches of 16
|
| 65 |
| Earnings-21, 44 calls (39 h), long-form | 3633x | 8.24 % |
|
| 66 |
-
| FLEURS de, es, fr, it, nl, pt (15.8 h) | 1839x | 4.57 % |
|
| 67 |
-
| MLS es, fr, it, pt (15.8 h) | 1921x | 6.25 % |
|
| 68 |
|
| 69 |
-
More results are in the [scriptrs README](https://github.com/avencera/scriptrs#cuda).
|
| 70 |
|
| 71 |
## Usage
|
| 72 |
|
|
|
|
| 43 |
|
| 44 |
# scriptrs models
|
| 45 |
|
| 46 |
+
Model files for [scriptrs](https://github.com/avencera/scriptrs), a Rust library for fast NVIDIA Parakeet TDT v2 and v3 speech-to-text on Core ML (macOS) and native CUDA (Linux), at the same word error rate as NVIDIA's reference implementation. With the default `online` feature, scriptrs downloads these files on first use from a pinned revision of this repository.
|
| 47 |
|
| 48 |
These are format conversions of NVIDIA's models, not new or retrained models. The native CUDA weights are NVIDIA's FP32 weights with their original names; scriptrs converts selected layers to FP16 on the GPU at load time.
|
| 49 |
|
| 50 |
+
## Same accuracy, faster
|
| 51 |
|
| 52 |
+
scriptrs runs NVIDIA's weights unchanged and gets the same word error rate as NVIDIA's own NeMo implementation, at 3x its speed. All results are Parakeet v3 on an RTX 5090.
|
| 53 |
+
|
| 54 |
+
LibriSpeech test-clean, English (5.4 h), one utterance at a time:
|
| 55 |
|
| 56 |
| | Speed (x real time) | WER |
|
| 57 |
|---|---|---|
|
| 58 |
+
| **scriptrs CUDA** | **1534x** | **1.93 %** |
|
| 59 |
| NVIDIA NeMo (PyTorch) | 477x | 1.93 % |
|
| 60 |
| parakeet-rs 0.4.3 (ONNX Runtime) | 253x | 1.93 % |
|
| 61 |
|
| 62 |
+
### Multilingual
|
| 63 |
+
|
| 64 |
+
The multilingual check is nearly 6 times the size of LibriSpeech: 31.6 h over ten language sets in six languages, one utterance at a time with automatic language selection. In scriptrs's parity tests, its FP32 mode matches NeMo layer by layer and gives NeMo's exact text. The FP16 default scores the same as FP32:
|
| 65 |
+
|
| 66 |
+
| Set | Speed, FP16 default | WER, FP16 default | WER, FP32 |
|
| 67 |
+
|---|---|---|---|
|
| 68 |
+
| FLEURS de, es, fr, it, nl, pt (15.8 h) | 1839x | 4.57 % | 4.57 % |
|
| 69 |
+
| MLS es, fr, it, pt (15.8 h) | 1921x | 6.25 % | 6.26 % |
|
| 70 |
+
|
| 71 |
+
### Batches and long recordings
|
| 72 |
|
| 73 |
| Workload | Speed (x real time) | WER |
|
| 74 |
|---|---|---|
|
| 75 |
+
| LibriSpeech test-clean, batches of 16 | 4767x | 1.93 %, unchanged |
|
| 76 |
| Earnings-21, 44 calls (39 h), long-form | 3633x | 8.24 % |
|
|
|
|
|
|
|
| 77 |
|
| 78 |
+
On Earnings-21, NeMo transcribing each whole call with local attention scores 9.58 %. More results, including harder English sets against Whisper large-v3, are in the [scriptrs README](https://github.com/avencera/scriptrs#cuda).
|
| 79 |
|
| 80 |
## Usage
|
| 81 |
|