| # Full VoxLingua107 dev evaluation after model integration |
|
|
| `espnet/geolid_combined_shared_trainable` classified **1501 / 1,609 utterances |
| correctly (93.29%)** across 33 languages. |
| This is a complete new inference run using `ESPnetLIDModel` at |
| `301372e4da311d2a3d9574a5a43b3037ae6e67c1`, completed at `2026-10-23T00:55:52.840208+00:00`. |
|
|
| All 1,609 predicted labels match the earlier full-dev evaluation at `ca777d59eb49519e3827e40cc34997f2123d6682`. |
|
|
| The measured checkpoint SHA-256 is `8a8f9dd767c141dcae0cb34a7780d7f593bc67f560267cd4d2d2fadd54f6bd16`. |
| The model inputs came from this immutable |
| [HF revision](https://huggingface.co/espnet/geolid_combined_shared_trainable/tree/613d18645d1be3bd585a2cbae0485350a2a4c027). |
| The checkpoint and published model configuration have not been modified by |
| this evaluation. Only the local path to `languages.txt` was resolved for the |
| run; its published `./languages.txt` value works from the model directory. |
|
|
| ## Reproduce inference |
|
|
| Install the ESPnet revision documented in the model card, and download this |
| model's `config.yaml`, `model.pth` and `languages.txt`. Run the following command |
| from that model directory. Prepare `data/voxlingua107-dev/wav.scp` as sorted |
| `<utterance-id> <absolute-waveform-path>` lines for all 1,609 dev utterances; |
| use the original full-length, mono, 16 kHz waveforms. Choose a fresh output |
| directory because the CLI can skip inference when a completion marker exists. |
|
|
| ```bash |
| OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 python -m espnet2.bin.lid_inference \ |
| --output_dir evaluation_output \ |
| --ngpu 1 --dtype float32 \ |
| --data_path_and_name_and_type data/voxlingua107-dev/wav.scp,speech,sound \ |
| --valid_batch_size 1 --num_workers 8 \ |
| --lid_train_config config.yaml --lid_model_file model.pth \ |
| --use_preprocessor false --fix_duration false \ |
| --extract_embd false --save_embd_per_utt false \ |
| --save_embd_avg_lang false --save_tsne_plot false \ |
| --checkpoint_interval 200 |
| ``` |
|
|
| The output `evaluation_output/voxlingua107-dev_lids` contains the predicted |
| labels. Compare each utterance ID with the reference language in |
| [evaluation_predictions.jsonl](evaluation_predictions.jsonl). The reported |
| accuracy is `100 * correct / 1609`; no duration weighting or macro averaging |
| is used. The JSONL also records the predictions from this run and a correctness |
| flag for every utterance, so the score can be independently recomputed. |
|
|
| ## Evidence and scope |
|
|
| - [Evaluation metadata](evaluation.json): counts, settings, per-language counts, |
| environment, source revision and hashes. |
| - [All predictions and reference labels](evaluation_predictions.jsonl). |
| - [Complete captured inference log](evaluation.log): paths refer to the actual |
| evaluation workspace; its output is retained unchanged. |
| - [Prediction differences from the previous run](evaluation_prediction_differences.json). |
| - [Previous evaluation metadata](evaluation_previous.json), preserved unchanged. |
| - [Validation history](validation.json): earlier strict-loading checks plus a |
| separate record of this full evaluation through the unmodified CLI. |
|
|
| This run evaluates VoxLingua107 dev only. Historical paper results for other |
| test sets have not been remeasured here. The new full run uses the standard |
| loader without forcing strict loading; the earlier explicit strict-loading |
| check is a separate historical validation. |
|
|
