Spaces:

---

tvosch

/

Dutch-ASR-Leaderboard

Like 11

Running

Results for two murmel-flash models, run with the new murmel-flash backend (https://github.com/tvosch/Dutch-ASR-leaderboard/pull/3), normalizer v2, package defaults (beam 4), on an A100.

• murmel-flash-m-v5: 128M parameters, open weights on Hugging Face (gated, automatic approval), licence murmel-flash-licence-1.0.
• murmel-flash-l-v5: 397M parameters, proprietary. The weights are not public.

All 4,576 samples were transcribed, none failed. RTF was measured on an A100, not on the H100 used for the other local models.

Files:

• results/theaifactorybv__murmel-flash-m-v5.json
• results/theaifactorybv__murmel-flash-l-v5.json
• app.py: murmel-flash-l-v5 links to murmel.nl and counts as proprietary (its weights are private); runtime label for the murmel-flash backend.

Thanks for the PR and for adding the murmel-flash backend!

For the leaderboard we only list results we've run ourselves, so before merging we ran both models on our side and I've added a commit that replaces your result files with those runs:

Model How we ran it FLEURS VoxPopuli MLS Average
murmel-flash-m-v5 open weights, murmel-flash 5.0.0, beam 4, H100 8.63 10.36 8.64 9.21
murmel-flash-l-v5 Murmel API (api.murmel.eu) 6.04 9.17 7.60 7.60
m-v5 matches your run: 97.5% identical transcripts, within 0.02 WER per dataset, same average. Running it on our H100 also keeps its speed comparable with the other local models. It comes in at #8 overall and #3 among open models.

l-v5: the weights aren't public, so we list it from our own API run, as we do for other closed models. It comes in at #3 overall. Does the API serve the same checkpoint and decoding (beam 4) as the package, or a different build? Happy to re-run if that changes.

I've kept your app.py changes and added a changelog entry. I'll merge this now, thanks again and nice work!