File size: 3,254 Bytes
f82a6c3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | # Evaluation — CAFE-25lang
Scripts to reproduce the Open ASR Leaderboard and Multilingual LibriSpeech WER in the model card. Decoding and scoring are the
Open ASR Leaderboard's own: greedy batched decoding with the CUDA-graph decoder disabled, the
official English normalizer for English sets and `ml_normalizer` plus compound-word pairing for the
multilingual ones. No training code or data is included.
## The one thing to get right
This model routes by language. **English uses the bare trunk; every other language needs
`adapters.safetensors` attached.** Evaluating a non-English set without the adapters silently
measures the English path instead. `adapter_wrapper.py` does the attaching; `run_leaderboard.py`
wires it into the official runners.
## Files
| File | Purpose |
| --- | --- |
| `eval_mls8.py` | Multilingual LibriSpeech, one language per run (the card's 8-language table) |
| `run_leaderboard.py` | runs an official Open ASR Leaderboard runner with the adapters attached |
| `adapter_wrapper.py` | attaches the adapters to a restored model via forward hooks |
## Setup
```bash
pip install "nemo_toolkit[asr]" datasets evaluate jiwer soundfile safetensors
git clone https://github.com/huggingface/open_asr_leaderboard
export PYTHONPATH=$PWD/open_asr_leaderboard:$PYTHONPATH # provides the `normalizer` package
```
## Multilingual LibriSpeech (8 languages)
English on the trunk, the other seven with adapters:
```bash
python eval_mls8.py --model_id ../CAFE-25lang.nemo --lang en \
--result_json results/mls_en.json
for L in de nl pl es fr it pt; do
python eval_mls8.py --model_id ../CAFE-25lang.nemo --lang $L \
--adapters ../adapters.safetensors --result_json results/mls_$L.json
done
```
The card reports the unweighted mean of the eight per-language WERs.
## Open ASR Leaderboard sets
```bash
cd open_asr_leaderboard/nemo_asr
EVAL=/path/to/CAFE_ASR/eval
MODEL=/path/to/CAFE_ASR/CAFE-25lang.nemo
ADAPTERS=/path/to/CAFE_ASR/adapters.safetensors
# English short-form, e.g. LibriSpeech test-clean — bare trunk
python $EVAL/run_leaderboard.py run_eval.py NONE -- \
--model_id $MODEL --model_name CAFE-25lang \
--dataset_path hf-audio/esb-datasets-test-only-sorted \
--dataset librispeech --split test.clean \
--batch_size 128 --device 0 --max_eval_samples -1
# a multilingual set, e.g. FLEURS German — adapters attached
python $EVAL/run_leaderboard.py run_eval_ml.py $ADAPTERS -- \
--model_id $MODEL --model_name CAFE-25lang \
--dataset nithinraok/asr-leaderboard-datasets --config_name fleurs_de --language de \
--split test --batch_size 64 --device 0 --max_eval_samples -1
```
Each run prints `WER: <x> %` and writes a per-utterance manifest under `results/` in the runner's
working directory, which the leaderboard's `normalizer/eval_utils.score_results` can re-score.
## Notes
- A GPU with ~8 GB is enough at these batch sizes; drop `--batch_size` if memory is tight.
- Audio is cached as wav under `audio_cache/` on first use, so re-runs are much faster.
- The numbers above assume the language of each set is known. For routing with the language gate
instead (`inference_gated.py` in the repo root), see the model card and the
language-gate appendix of the paper.
|