praveenperera commited on
Commit
5e92d53
·
verified ·
1 Parent(s): 56a556b

Show that WER matches NeMo, with the multilingual results

Browse files
Files changed (1) hide show
  1. README.md +18 -9
README.md CHANGED
@@ -43,30 +43,39 @@ tags:
43
 
44
  # scriptrs models
45
 
46
- Model files for [scriptrs](https://github.com/avencera/scriptrs), a Rust library for fast NVIDIA Parakeet TDT v2 and v3 speech-to-text on Core ML (macOS) and native CUDA (Linux). With the default `online` feature, scriptrs downloads these files on first use from a pinned revision of this repository.
47
 
48
  These are format conversions of NVIDIA's models, not new or retrained models. The native CUDA weights are NVIDIA's FP32 weights with their original names; scriptrs converts selected layers to FP16 on the GPU at load time.
49
 
50
- ## Speed
51
 
52
- RTX 5090, Parakeet v3, LibriSpeech test-clean (5.4 h), one utterance at a time:
 
 
53
 
54
  | | Speed (x real time) | WER |
55
  |---|---|---|
56
- | scriptrs CUDA (FP16 default) | 1534x | 1.93 % |
57
  | NVIDIA NeMo (PyTorch) | 477x | 1.93 % |
58
  | parakeet-rs 0.4.3 (ONNX Runtime) | 253x | 1.93 % |
59
 
60
- More scriptrs CUDA results on the same GPU and model:
 
 
 
 
 
 
 
 
 
61
 
62
  | Workload | Speed (x real time) | WER |
63
  |---|---|---|
64
- | LibriSpeech test-clean, batches of 16 through the queue | 4767x | 1.93 % |
65
  | Earnings-21, 44 calls (39 h), long-form | 3633x | 8.24 % |
66
- | FLEURS de, es, fr, it, nl, pt (15.8 h) | 1839x | 4.57 % |
67
- | MLS es, fr, it, pt (15.8 h) | 1921x | 6.25 % |
68
 
69
- More results are in the [scriptrs README](https://github.com/avencera/scriptrs#cuda).
70
 
71
  ## Usage
72
 
 
43
 
44
  # scriptrs models
45
 
46
+ Model files for [scriptrs](https://github.com/avencera/scriptrs), a Rust library for fast NVIDIA Parakeet TDT v2 and v3 speech-to-text on Core ML (macOS) and native CUDA (Linux), at the same word error rate as NVIDIA's reference implementation. With the default `online` feature, scriptrs downloads these files on first use from a pinned revision of this repository.
47
 
48
  These are format conversions of NVIDIA's models, not new or retrained models. The native CUDA weights are NVIDIA's FP32 weights with their original names; scriptrs converts selected layers to FP16 on the GPU at load time.
49
 
50
+ ## Same accuracy, faster
51
 
52
+ scriptrs runs NVIDIA's weights unchanged and gets the same word error rate as NVIDIA's own NeMo implementation, at 3x its speed. All results are Parakeet v3 on an RTX 5090.
53
+
54
+ LibriSpeech test-clean, English (5.4 h), one utterance at a time:
55
 
56
  | | Speed (x real time) | WER |
57
  |---|---|---|
58
+ | **scriptrs CUDA** | **1534x** | **1.93 %** |
59
  | NVIDIA NeMo (PyTorch) | 477x | 1.93 % |
60
  | parakeet-rs 0.4.3 (ONNX Runtime) | 253x | 1.93 % |
61
 
62
+ ### Multilingual
63
+
64
+ The multilingual check is nearly 6 times the size of LibriSpeech: 31.6 h over ten language sets in six languages, one utterance at a time with automatic language selection. In scriptrs's parity tests, its FP32 mode matches NeMo layer by layer and gives NeMo's exact text. The FP16 default scores the same as FP32:
65
+
66
+ | Set | Speed, FP16 default | WER, FP16 default | WER, FP32 |
67
+ |---|---|---|---|
68
+ | FLEURS de, es, fr, it, nl, pt (15.8 h) | 1839x | 4.57 % | 4.57 % |
69
+ | MLS es, fr, it, pt (15.8 h) | 1921x | 6.25 % | 6.26 % |
70
+
71
+ ### Batches and long recordings
72
 
73
  | Workload | Speed (x real time) | WER |
74
  |---|---|---|
75
+ | LibriSpeech test-clean, batches of 16 | 4767x | 1.93 %, unchanged |
76
  | Earnings-21, 44 calls (39 h), long-form | 3633x | 8.24 % |
 
 
77
 
78
+ On Earnings-21, NeMo transcribing each whole call with local attention scores 9.58 %. More results, including harder English sets against Whisper large-v3, are in the [scriptrs README](https://github.com/avencera/scriptrs#cuda).
79
 
80
  ## Usage
81