| # Evaluation — CAFE-25lang |
|
|
| Scripts to reproduce the Open ASR Leaderboard and Multilingual LibriSpeech WER in the model card. Decoding and scoring are the |
| Open ASR Leaderboard's own: greedy batched decoding with the CUDA-graph decoder disabled, the |
| official English normalizer for English sets and `ml_normalizer` plus compound-word pairing for the |
| multilingual ones. No training code or data is included. |
|
|
| ## The one thing to get right |
|
|
| This model routes by language. **English uses the bare trunk; every other language needs |
| `adapters.safetensors` attached.** Evaluating a non-English set without the adapters silently |
| measures the English path instead. `adapter_wrapper.py` does the attaching; `run_leaderboard.py` |
| wires it into the official runners. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| | --- | --- | |
| | `eval_mls8.py` | Multilingual LibriSpeech, one language per run (the card's 8-language table) | |
| | `run_leaderboard.py` | runs an official Open ASR Leaderboard runner with the adapters attached | |
| | `adapter_wrapper.py` | attaches the adapters to a restored model via forward hooks | |
|
|
| ## Setup |
|
|
| ```bash |
| pip install "nemo_toolkit[asr]" datasets evaluate jiwer soundfile safetensors |
| git clone https://github.com/huggingface/open_asr_leaderboard |
| export PYTHONPATH=$PWD/open_asr_leaderboard:$PYTHONPATH # provides the `normalizer` package |
| ``` |
|
|
| ## Multilingual LibriSpeech (8 languages) |
|
|
| English on the trunk, the other seven with adapters: |
|
|
| ```bash |
| python eval_mls8.py --model_id ../CAFE-25lang.nemo --lang en \ |
| --result_json results/mls_en.json |
| |
| for L in de nl pl es fr it pt; do |
| python eval_mls8.py --model_id ../CAFE-25lang.nemo --lang $L \ |
| --adapters ../adapters.safetensors --result_json results/mls_$L.json |
| done |
| ``` |
|
|
| The card reports the unweighted mean of the eight per-language WERs. |
|
|
| ## Open ASR Leaderboard sets |
|
|
| ```bash |
| cd open_asr_leaderboard/nemo_asr |
| EVAL=/path/to/CAFE_ASR/eval |
| MODEL=/path/to/CAFE_ASR/CAFE-25lang.nemo |
| ADAPTERS=/path/to/CAFE_ASR/adapters.safetensors |
| |
| # English short-form, e.g. LibriSpeech test-clean — bare trunk |
| python $EVAL/run_leaderboard.py run_eval.py NONE -- \ |
| --model_id $MODEL --model_name CAFE-25lang \ |
| --dataset_path hf-audio/esb-datasets-test-only-sorted \ |
| --dataset librispeech --split test.clean \ |
| --batch_size 128 --device 0 --max_eval_samples -1 |
| |
| # a multilingual set, e.g. FLEURS German — adapters attached |
| python $EVAL/run_leaderboard.py run_eval_ml.py $ADAPTERS -- \ |
| --model_id $MODEL --model_name CAFE-25lang \ |
| --dataset nithinraok/asr-leaderboard-datasets --config_name fleurs_de --language de \ |
| --split test --batch_size 64 --device 0 --max_eval_samples -1 |
| ``` |
|
|
| Each run prints `WER: <x> %` and writes a per-utterance manifest under `results/` in the runner's |
| working directory, which the leaderboard's `normalizer/eval_utils.score_results` can re-score. |
|
|
| ## Notes |
|
|
| - A GPU with ~8 GB is enough at these batch sizes; drop `--batch_size` if memory is tight. |
| - Audio is cached as wav under `audio_cache/` on first use, so re-runs are much faster. |
| - The numbers above assume the language of each set is known. For routing with the language gate |
| instead (`inference_gated.py` in the repo root), see the model card and the |
| language-gate appendix of the paper. |
|
|
