Add Open ASR Leaderboard results (English and multilingual)

#3
by bezzam HF Staff - opened
Files changed (1) hide show
  1. README.md +35 -0
README.md CHANGED
@@ -130,3 +130,38 @@ Pass these options through vLLM's `chat_template_kwargs`, or directly to
130
  ## License
131
 
132
  Licensed under the [Apache License 2.0](LICENSE).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
  ## License
131
 
132
  Licensed under the [Apache License 2.0](LICENSE).
133
+
134
+ ## Open ASR Leaderboard results
135
+
136
+ Evaluated on the [Open ASR Leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard?version=1.16.1) (release 18-11-2026, v1.16.1), run on an H200 with vLLM. Compare against other open models on the multilingual tracks with [this view](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard?languages=Reson8%2FRumble-2-Flash&proprietary=false) (API models hidden).
137
+
138
+ ### English track
139
+
140
+ Average WER on the leaderboard: **6.60** (RTFx 6338), over the 7 public splits plus the aggregated Private (scripted) and Private (conversational) scores.
141
+
142
+ | Split | WER (%) | RTFx |
143
+ |---|---|---|
144
+ | AMI | 9.24 | 4245 |
145
+ | Earnings22 | 6.45 | 4811 |
146
+ | GigaSpeech | 7.56 | 5978 |
147
+ | SPGISpeech | 3.08 | 7308 |
148
+ | URGENT24 | 10.25 | 3314 |
149
+ | Voice Arena Monsoon | 3.56 | 3760 |
150
+ | VoxPopuli | 2.43 | 4222 |
151
+ | Private (scripted) | 3.71 | |
152
+ | Private (conversational) | 13.14 | |
153
+
154
+ ### Multilingual track
155
+
156
+ WER (%) per language and dataset; Chinese is CER. The last column is the leaderboard's average for the language.
157
+
158
+ | Language | FLEURS | Common Voice | MLS | Voice Arena Monsoon | AISHELL-1 | Average |
159
+ |---|---|---|---|---|---|---|
160
+ | German | 4.77 | 6.28 | | | | **5.52** |
161
+ | Spanish | 2.82 | 4.33 | 4.04 | | | **3.73** |
162
+ | French | 5.95 | 10.01 | 5.66 | | | **7.21** |
163
+ | Italian | 2.53 | 4.84 | 7.90 | | | **5.09** |
164
+ | Dutch | 6.05 | 4.92 | 9.38 | | | **6.78** |
165
+ | Portuguese | 5.31 | | 7.61 | | | **6.46** |
166
+ | Hindi | | | | 14.11 | | **14.11** |
167
+ | Chinese (CER) | 5.82 | | | | 7.31 | **6.56** |