qingzhengwang commited on
Commit
dda9e40
·
verified ·
1 Parent(s): 613d186

Report complete VoxLingua107 dev rerun with unified ESPnetLIDModel

Browse files
README.md CHANGED
@@ -189,25 +189,39 @@ converted checkpoint is reported separately from historical paper results below.
189
  | --- | ---: | ---: | ---: |
190
  | VoxLingua107 dev | 94.4 | **93.29** | 1501 / 1609 |
191
 
192
- The converted checkpoint was evaluated on all 1,609 VoxLingua107 dev utterances
193
- using the standard `espnet2.bin.lid_inference` CLI at ESPnet revision
194
- `ca777d59eb49519e3827e40cc34997f2123d6682`: GPU float32, batch size 1,
195
- 8 data-loading workers, full-length waveforms, `--use_preprocessor false` and
196
- `--fix_duration false`. The input `wav.scp` was sorted by utterance ID.
197
- The environment used PyTorch 2.11.0+cu128, Transformers 5.14.1 and
198
- espnet-s3prl 2026.5.14. Accuracy is the fraction of correctly classified utterances.
 
 
199
 
200
- [Evaluation metadata](evaluation.json) records the counts, runtime settings,
201
- implementation revision and checkpoint SHA-256. These are batch-size-1 results.
202
- `utterance_mvn` and final `chn_attn_stat` pooling do mask padding. The remaining
203
- batch dependence comes earlier: S3PRL aligns features to `ceil(samples / 320)`,
204
- so the extra frames of a shorter utterance can differ from its single-utterance
205
- run, and ECAPA's temporal convolutions and unmasked SE global means depend on
206
- padded sequence length. Final masked pooling cannot undo those feature changes.
207
- Historical paper scores for other datasets are not new measurements of this
208
- converted checkpoint.
209
 
210
- The configuration now selects the standard `ESPnetLIDModel`; lang2vec prediction and conditioning are optional features of that model. `local/convert_checkpoint.py` updates the earlier `model: lang2vec` selector without changing the converted parameter names or weight file. At implementation revision `ca2530ed59e3a9c916ad3928032df5777555275d`, all 1,131 checkpoint entries loaded strictly and standard-CLI inference on six complete real VoxLingua107 dev utterances reproduced their earlier predictions. This is a compatibility check, not a new full-dev evaluation; the full evaluation above retains its original implementation revision.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
211
 
212
  ## Installation
213
 
@@ -257,9 +271,10 @@ from the original model card and the original sorted recipe label construction;
257
  it is not a copy of the missing original training utterance manifest.
258
 
259
  `conversion.json` records the source checkpoint revision and label provenance.
260
- `validation.json` records strict loading, unchanged tensor values during layout
261
- migration, and successful full-dev CLI inference. `evaluation.json` records the
262
- measured accuracy. The metadata targets are used only for training; inference
 
263
  uses speech and predicts its own conditioning vectors.
264
  The old training logs and figures retained under `exp_*` describe the original
265
  training run. Earlier checkpoint layouts remain available in repository history.
 
189
  | --- | ---: | ---: | ---: |
190
  | VoxLingua107 dev | 94.4 | **93.29** | 1501 / 1609 |
191
 
192
+ The converted checkpoint was evaluated again on all **1,609 VoxLingua107 dev
193
+ utterances** after integrating lang2vec into the standard `ESPnetLIDModel`.
194
+ This full evaluation used the unmodified `espnet2.bin.lid_inference` CLI at
195
+ ESPnet revision `301372e4da311d2a3d9574a5a43b3037ae6e67c1`: GPU float32,
196
+ batch size 1, 8 data-loading workers, full-length waveforms,
197
+ `--use_preprocessor false` and `--fix_duration false`. The input `wav.scp`
198
+ was sorted by utterance ID. The environment used PyTorch 2.11.0+cu128,
199
+ Transformers 5.14.1 and espnet-s3prl 2026.5.14.
200
+ Accuracy is the fraction of correctly classified utterances.
201
 
202
+ [Evaluation report and reproduction command](evaluation_report.md),
203
+ [all 1,609 predictions with reference labels](evaluation_predictions.jsonl),
204
+ [inference log](evaluation.log), and [evaluation metadata](evaluation.json)
205
+ provide the results, runtime settings, input revision and checkpoint SHA-256.
206
+ All 1,609 predicted labels match the earlier full-dev evaluation at `ca777d59eb49519e3827e40cc34997f2123d6682`.
207
+ The previous evaluation metadata is preserved in
208
+ [evaluation_previous.json](evaluation_previous.json).
 
 
209
 
210
+ These are batch-size-1 results. `utterance_mvn` and final `chn_attn_stat`
211
+ pooling do mask padding. The remaining batch dependence comes earlier:
212
+ S3PRL aligns features to `ceil(samples / 320)`, so the extra frames of a
213
+ shorter utterance can differ from its single-utterance run, and ECAPA's
214
+ temporal convolutions and unmasked SE global means depend on padded sequence
215
+ length. Final masked pooling cannot undo those feature changes. Historical
216
+ paper scores for other datasets are not new measurements of this checkpoint.
217
+
218
+ The configuration selects the standard `ESPnetLIDModel`; lang2vec prediction
219
+ and conditioning are optional features of that model. The refactor preserved
220
+ the converted parameter names and weight file. The earlier strict-loading
221
+ and six-utterance checks remain recorded at their original revisions in
222
+ [validation.json](validation.json). Its separate `full_dev_model_integration`
223
+ record describes this new complete evaluation with the stock CLI loader;
224
+ the full evaluation did not instrument that loader to force strict loading.
225
 
226
  ## Installation
227
 
 
271
  it is not a copy of the missing original training utterance manifest.
272
 
273
  `conversion.json` records the source checkpoint revision and label provenance.
274
+ `validation.json` preserves the historical strict-loading and layout-migration
275
+ checks, and separately records the complete evaluation after model integration.
276
+ `evaluation.json` records the latest measured accuracy, with per-utterance
277
+ results in `evaluation_predictions.jsonl`. The metadata targets are used only for training; inference
278
  uses speech and predicts its own conditioning vectors.
279
  The old training logs and figures retained under `exp_*` describe the original
280
  training run. Earlier checkpoint layouts remain available in repository history.
evaluation.json CHANGED
@@ -1,11 +1,20 @@
1
  {
2
  "dataset": "VoxLingua107",
3
  "split": "dev",
 
 
 
 
 
 
 
4
  "utterances": 1609,
5
  "languages": 33,
6
  "metric": "utterance-level accuracy (%)",
 
 
 
7
  "entrypoint": "espnet2.bin.lid_inference",
8
- "implementation_revision": "ca777d59eb49519e3827e40cc34997f2123d6682",
9
  "device": "cuda",
10
  "dtype": "float32",
11
  "batch_size": 1,
@@ -13,26 +22,171 @@
13
  "use_preprocessor": false,
14
  "fix_duration": false,
15
  "input_order": "wav.scp sorted by utterance ID; standard ESPnet streaming iterator",
16
- "manifest_sha256": "6ad3c8fc6c74fab2720fe186525fc3bd20c4ab7b1be35d227c5c7f5456ba6c54",
17
  "environment": {
18
- "python": "3.12.3 (main, Jun 19 2026, 12:46:00) [GCC 13.3.0]",
19
  "torch": "2.11.0+cu128",
20
- "torch_cuda_build": "12.8",
21
  "torchaudio": "2.11.0+cu128",
22
  "transformers": "5.14.1",
23
- "espnet_s3prl": "2026.5.14",
24
  "soundfile": "0.14.0"
25
  },
26
- "repo_id": "espnet/geolid_combined_shared_trainable",
27
- "model_sha256": "8a8f9dd767c141dcae0cb34a7780d7f593bc67f560267cd4d2d2fadd54f6bd16",
28
- "correct": 1501,
29
- "accuracy_percent": 93.2877563704164,
30
- "paper_accuracy_percent": 94.4,
31
- "source_hf_revision_before_layout_migration": "0c7ee08dc9ab7d9ac651c7858fd6ce40849475b6",
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
  "batch_dependence": {
33
  "s3prl_frame_alignment": "S3PRL reports ceil(samples/320) frames and repeats the final frame only at the batch boundary. Shorter items can instead expose padded-query frames from Wav2Vec2.",
34
  "ecapa_tdnn": "Temporal convolutions and unmasked SE global means depend on the padded sequence length before the masked final pooling.",
35
  "mvn_and_final_pooling_mask_padding": true,
36
  "reported_measurement_batch_size": 1
37
- }
 
 
 
 
 
 
 
 
38
  }
 
1
  {
2
  "dataset": "VoxLingua107",
3
  "split": "dev",
4
+ "repo_id": "espnet/geolid_combined_shared_trainable",
5
+ "source_hf_revision": "613d18645d1be3bd585a2cbae0485350a2a4c027",
6
+ "implementation_revision": "301372e4da311d2a3d9574a5a43b3037ae6e67c1",
7
+ "model": "ESPnetLIDModel",
8
+ "model_sha256": "8a8f9dd767c141dcae0cb34a7780d7f593bc67f560267cd4d2d2fadd54f6bd16",
9
+ "config_sha256": "b06ed46b21a44a48ee0860407d1fa772344fedae302380f74ae2b02df7757abb",
10
+ "manifest_sha256": "6ad3c8fc6c74fab2720fe186525fc3bd20c4ab7b1be35d227c5c7f5456ba6c54",
11
  "utterances": 1609,
12
  "languages": 33,
13
  "metric": "utterance-level accuracy (%)",
14
+ "correct": 1501,
15
+ "accuracy_percent": 93.2877563704164,
16
+ "paper_accuracy_percent": 94.4,
17
  "entrypoint": "espnet2.bin.lid_inference",
 
18
  "device": "cuda",
19
  "dtype": "float32",
20
  "batch_size": 1,
 
22
  "use_preprocessor": false,
23
  "fix_duration": false,
24
  "input_order": "wav.scp sorted by utterance ID; standard ESPnet streaming iterator",
 
25
  "environment": {
26
+ "python": "3.12.3",
27
  "torch": "2.11.0+cu128",
 
28
  "torchaudio": "2.11.0+cu128",
29
  "transformers": "5.14.1",
30
+ "espnet-s3prl": "2026.5.14",
31
  "soundfile": "0.14.0"
32
  },
33
+ "finished_at": "2026-10-23T00:55:52.840208+00:00",
34
+ "seconds": 161.17425368726254,
35
+ "predictions_file": "evaluation_predictions.jsonl",
36
+ "predictions_sha256": "40264d2766f60dfce717ebc638ef76bc85380709693ca3e32178829e5df5453e",
37
+ "comparison": {
38
+ "baseline_implementation_revision": "ca777d59eb49519e3827e40cc34997f2123d6682",
39
+ "baseline_correct": 1501,
40
+ "matched_predictions": 1609,
41
+ "changed_predictions": 0,
42
+ "baseline_predictions_sha256": "40264d2766f60dfce717ebc638ef76bc85380709693ca3e32178829e5df5453e"
43
+ },
44
+ "per_language": {
45
+ "ara": {
46
+ "total": 100,
47
+ "correct": 95
48
+ },
49
+ "aze": {
50
+ "total": 34,
51
+ "correct": 31
52
+ },
53
+ "cmn": {
54
+ "total": 23,
55
+ "correct": 22
56
+ },
57
+ "dan": {
58
+ "total": 99,
59
+ "correct": 93
60
+ },
61
+ "deu": {
62
+ "total": 82,
63
+ "correct": 77
64
+ },
65
+ "ell": {
66
+ "total": 5,
67
+ "correct": 4
68
+ },
69
+ "eng": {
70
+ "total": 80,
71
+ "correct": 73
72
+ },
73
+ "est": {
74
+ "total": 85,
75
+ "correct": 80
76
+ },
77
+ "fas": {
78
+ "total": 100,
79
+ "correct": 89
80
+ },
81
+ "fin": {
82
+ "total": 93,
83
+ "correct": 91
84
+ },
85
+ "fra": {
86
+ "total": 100,
87
+ "correct": 96
88
+ },
89
+ "hrv": {
90
+ "total": 4,
91
+ "correct": 3
92
+ },
93
+ "hun": {
94
+ "total": 4,
95
+ "correct": 4
96
+ },
97
+ "hye": {
98
+ "total": 100,
99
+ "correct": 96
100
+ },
101
+ "isl": {
102
+ "total": 26,
103
+ "correct": 26
104
+ },
105
+ "ita": {
106
+ "total": 49,
107
+ "correct": 49
108
+ },
109
+ "jpn": {
110
+ "total": 42,
111
+ "correct": 41
112
+ },
113
+ "lav": {
114
+ "total": 95,
115
+ "correct": 94
116
+ },
117
+ "lit": {
118
+ "total": 13,
119
+ "correct": 12
120
+ },
121
+ "mkd": {
122
+ "total": 4,
123
+ "correct": 4
124
+ },
125
+ "nld": {
126
+ "total": 100,
127
+ "correct": 95
128
+ },
129
+ "nno": {
130
+ "total": 3,
131
+ "correct": 3
132
+ },
133
+ "nor": {
134
+ "total": 24,
135
+ "correct": 13
136
+ },
137
+ "pol": {
138
+ "total": 2,
139
+ "correct": 2
140
+ },
141
+ "por": {
142
+ "total": 1,
143
+ "correct": 1
144
+ },
145
+ "rus": {
146
+ "total": 29,
147
+ "correct": 29
148
+ },
149
+ "slv": {
150
+ "total": 9,
151
+ "correct": 9
152
+ },
153
+ "spa": {
154
+ "total": 55,
155
+ "correct": 47
156
+ },
157
+ "srp": {
158
+ "total": 7,
159
+ "correct": 5
160
+ },
161
+ "swe": {
162
+ "total": 100,
163
+ "correct": 94
164
+ },
165
+ "tur": {
166
+ "total": 72,
167
+ "correct": 71
168
+ },
169
+ "ukr": {
170
+ "total": 17,
171
+ "correct": 17
172
+ },
173
+ "urd": {
174
+ "total": 52,
175
+ "correct": 35
176
+ }
177
+ },
178
  "batch_dependence": {
179
  "s3prl_frame_alignment": "S3PRL reports ceil(samples/320) frames and repeats the final frame only at the batch boundary. Shorter items can instead expose padded-query frames from Wav2Vec2.",
180
  "ecapa_tdnn": "Temporal convolutions and unmasked SE global means depend on the padded sequence length before the masked final pooling.",
181
  "mvn_and_final_pooling_mask_padding": true,
182
  "reported_measurement_batch_size": 1
183
+ },
184
+ "resolved_config_sha256": "fe2e8acd705871a4dea823f80c7afabfcb89b2cb40cf51161178b6f3209b72f5",
185
+ "languages_sha256": "285a350acd7dc5b109e950e54560b8bab70d282bae12b25328327d28a2b5226d",
186
+ "log_file": "evaluation.log",
187
+ "log_sha256": "d6831bc2d1e154c487052b31b10adc9072db4c8d045e4a1c6897125aa7a4c6c1",
188
+ "prediction_differences_file": "evaluation_prediction_differences.json",
189
+ "previous_evaluation_file": "evaluation_previous.json",
190
+ "source_config_url": "https://huggingface.co/espnet/geolid_combined_shared_trainable/blob/613d18645d1be3bd585a2cbae0485350a2a4c027/config.yaml",
191
+ "loader": "Unmodified standard ESPnet CLI loader; strict loading was not forced"
192
  }
evaluation.log ADDED
@@ -0,0 +1,425 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Failed to import Flash Attention, using ESPnet default: No module named 'flash_attn'
2
+ /home/qingzhengw/espnet/.geolid-reproduction/venv/bin/python /home/qingzhengw/espnet/.geolid-reproduction/review-code/espnet2/bin/lid_inference.py --output_dir /home/qingzhengw/espnet/.geolid-reproduction/full-dev-unified-301372e4/models/combined_shared_trainable --ngpu 1 --dtype float32 --data_path_and_name_and_type /home/qingzhengw/espnet/.geolid-reproduction/full-dev-unified-301372e4/data/voxlingua107-dev/wav.scp,speech,sound --valid_batch_size 1 --lid_train_config /home/qingzhengw/espnet/.geolid-reproduction/full-dev-unified-301372e4/models/combined_shared_trainable/config.resolved.yaml --lid_model_file /home/qingzhengw/espnet/.geolid-reproduction/round2-native/combined_shared_trainable/model.pth --use_preprocessor false --fix_duration false --num_workers 8 --extract_embd false --save_embd_per_utt false --save_embd_avg_lang false --save_tsne_plot false --checkpoint_interval 200
3
+ [host-10-0-93-146] 2026-10-23 00:53:32,012 (abs_task:2514) INFO: config file: /home/qingzhengw/espnet/.geolid-reproduction/full-dev-unified-301372e4/models/combined_shared_trainable/config.resolved.yaml
4
+
5
+ [transformers] Wav2Vec2Model LOAD REPORT from: facebook/mms-1b
6
+ Key | Status | |
7
+ -----------------------------+------------+--+-
8
+ project_q.weight | UNEXPECTED | |
9
+ quantizer.codevectors | UNEXPECTED | |
10
+ project_hid.weight | UNEXPECTED | |
11
+ quantizer.weight_proj.weight | UNEXPECTED | |
12
+ project_q.bias | UNEXPECTED | |
13
+ project_hid.bias | UNEXPECTED | |
14
+ quantizer.weight_proj.bias | UNEXPECTED | |
15
+
16
+ Notes:
17
+ - UNEXPECTED: can be ignored when loading from different task/architecture; not ok if you expect identical arch.
18
+ [host-10-0-93-146] 2026-10-23 00:53:43,296 (lid_inference:111) INFO: Model structure:
19
+ ESPnetLIDModel(
20
+ (frontend): S3prlFrontend(
21
+ (upstream): S3PRLUpstream(
22
+ (upstream): UpstreamExpert(
23
+ (model): Wav2Vec2Model(
24
+ (feature_extractor): Wav2Vec2FeatureEncoder(
25
+ (conv_layers): ModuleList(
26
+ (0): Wav2Vec2LayerNormConvLayer(
27
+ (conv): Conv1d(1, 512, kernel_size=(10,), stride=(5,))
28
+ (layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
29
+ (activation): GELUActivation()
30
+ )
31
+ (1-4): 4 x Wav2Vec2LayerNormConvLayer(
32
+ (conv): Conv1d(512, 512, kernel_size=(3,), stride=(2,))
33
+ (layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
34
+ (activation): GELUActivation()
35
+ )
36
+ (5-6): 2 x Wav2Vec2LayerNormConvLayer(
37
+ (conv): Conv1d(512, 512, kernel_size=(2,), stride=(2,))
38
+ (layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
39
+ (activation): GELUActivation()
40
+ )
41
+ )
42
+ )
43
+ (feature_projection): Wav2Vec2FeatureProjection(
44
+ (layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
45
+ (projection): Linear(in_features=512, out_features=1280, bias=True)
46
+ (dropout): Dropout(p=0.1, inplace=False)
47
+ )
48
+ (encoder): Wav2Vec2EncoderStableLayerNorm(
49
+ (pos_conv_embed): Wav2Vec2PositionalConvEmbedding(
50
+ (conv): ParametrizedConv1d(
51
+ 1280, 1280, kernel_size=(128,), stride=(1,), padding=(64,), groups=16
52
+ (parametrizations): ModuleDict(
53
+ (weight): ParametrizationList(
54
+ (0): _WeightNorm()
55
+ )
56
+ )
57
+ )
58
+ (padding): Wav2Vec2SamePadLayer()
59
+ (activation): GELUActivation()
60
+ )
61
+ (layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
62
+ (dropout): Dropout(p=0.1, inplace=False)
63
+ (layers): ModuleList(
64
+ (0-31): 32 x Wav2Vec2EncoderLayerStableLayerNorm(
65
+ (attention): Wav2Vec2Attention(
66
+ (k_proj): Linear(in_features=1280, out_features=1280, bias=True)
67
+ (v_proj): Linear(in_features=1280, out_features=1280, bias=True)
68
+ (q_proj): Linear(in_features=1280, out_features=1280, bias=True)
69
+ (out_proj): Linear(in_features=1280, out_features=1280, bias=True)
70
+ )
71
+ (dropout): Dropout(p=0.1, inplace=False)
72
+ (layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
73
+ (feed_forward): Wav2Vec2FeedForward(
74
+ (intermediate_dropout): Dropout(p=0.0, inplace=False)
75
+ (intermediate_dense): Linear(in_features=1280, out_features=5120, bias=True)
76
+ (intermediate_act_fn): GELUActivation()
77
+ (output_dense): Linear(in_features=5120, out_features=1280, bias=True)
78
+ (output_dropout): Dropout(p=0.1, inplace=False)
79
+ )
80
+ (final_layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
81
+ )
82
+ (32): Lang2VecConditionedLayer(
83
+ (layer): Wav2Vec2EncoderLayerStableLayerNorm(
84
+ (attention): Wav2Vec2Attention(
85
+ (k_proj): Linear(in_features=1280, out_features=1280, bias=True)
86
+ (v_proj): Linear(in_features=1280, out_features=1280, bias=True)
87
+ (q_proj): Linear(in_features=1280, out_features=1280, bias=True)
88
+ (out_proj): Linear(in_features=1280, out_features=1280, bias=True)
89
+ )
90
+ (dropout): Dropout(p=0.1, inplace=False)
91
+ (layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
92
+ (feed_forward): Wav2Vec2FeedForward(
93
+ (intermediate_dropout): Dropout(p=0.0, inplace=False)
94
+ (intermediate_dense): Linear(in_features=1280, out_features=5120, bias=True)
95
+ (intermediate_act_fn): GELUActivation()
96
+ (output_dense): Linear(in_features=5120, out_features=1280, bias=True)
97
+ (output_dropout): Dropout(p=0.1, inplace=False)
98
+ )
99
+ (final_layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
100
+ )
101
+ (encoder): IdentityEncoder()
102
+ (pooling): ChnAttnStatPooling(
103
+ (attention): Sequential(
104
+ (0): Conv1d(3840, 128, kernel_size=(1,), stride=(1,))
105
+ (1): ReLU()
106
+ (2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
107
+ (3): Conv1d(128, 1280, kernel_size=(1,), stride=(1,))
108
+ )
109
+ (softmax): Softmax(dim=2)
110
+ )
111
+ (projector): RawNet3Projector(
112
+ (bn): BatchNorm1d(2560, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
113
+ (fc): Linear(in_features=2560, out_features=192, bias=True)
114
+ )
115
+ (prediction): Linear(in_features=192, out_features=299, bias=True)
116
+ (conditioning_layer): Linear(in_features=299, out_features=1280, bias=True)
117
+ )
118
+ (33-35): 3 x Wav2Vec2EncoderLayerStableLayerNorm(
119
+ (attention): Wav2Vec2Attention(
120
+ (k_proj): Linear(in_features=1280, out_features=1280, bias=True)
121
+ (v_proj): Linear(in_features=1280, out_features=1280, bias=True)
122
+ (q_proj): Linear(in_features=1280, out_features=1280, bias=True)
123
+ (out_proj): Linear(in_features=1280, out_features=1280, bias=True)
124
+ )
125
+ (dropout): Dropout(p=0.1, inplace=False)
126
+ (layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
127
+ (feed_forward): Wav2Vec2FeedForward(
128
+ (intermediate_dropout): Dropout(p=0.0, inplace=False)
129
+ (intermediate_dense): Linear(in_features=1280, out_features=5120, bias=True)
130
+ (intermediate_act_fn): GELUActivation()
131
+ (output_dense): Linear(in_features=5120, out_features=1280, bias=True)
132
+ (output_dropout): Dropout(p=0.1, inplace=False)
133
+ )
134
+ (final_layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
135
+ )
136
+ (36): Lang2VecConditionedLayer(
137
+ (layer): Wav2Vec2EncoderLayerStableLayerNorm(
138
+ (attention): Wav2Vec2Attention(
139
+ (k_proj): Linear(in_features=1280, out_features=1280, bias=True)
140
+ (v_proj): Linear(in_features=1280, out_features=1280, bias=True)
141
+ (q_proj): Linear(in_features=1280, out_features=1280, bias=True)
142
+ (out_proj): Linear(in_features=1280, out_features=1280, bias=True)
143
+ )
144
+ (dropout): Dropout(p=0.1, inplace=False)
145
+ (layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
146
+ (feed_forward): Wav2Vec2FeedForward(
147
+ (intermediate_dropout): Dropout(p=0.0, inplace=False)
148
+ (intermediate_dense): Linear(in_features=1280, out_features=5120, bias=True)
149
+ (intermediate_act_fn): GELUActivation()
150
+ (output_dense): Linear(in_features=5120, out_features=1280, bias=True)
151
+ (output_dropout): Dropout(p=0.1, inplace=False)
152
+ )
153
+ (final_layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
154
+ )
155
+ (encoder): IdentityEncoder()
156
+ (pooling): ChnAttnStatPooling(
157
+ (attention): Sequential(
158
+ (0): Conv1d(3840, 128, kernel_size=(1,), stride=(1,))
159
+ (1): ReLU()
160
+ (2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
161
+ (3): Conv1d(128, 1280, kernel_size=(1,), stride=(1,))
162
+ )
163
+ (softmax): Softmax(dim=2)
164
+ )
165
+ (projector): RawNet3Projector(
166
+ (bn): BatchNorm1d(2560, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
167
+ (fc): Linear(in_features=2560, out_features=192, bias=True)
168
+ )
169
+ (prediction): Linear(in_features=192, out_features=299, bias=True)
170
+ (conditioning_layer): Linear(in_features=299, out_features=1280, bias=True)
171
+ )
172
+ (37-39): 3 x Wav2Vec2EncoderLayerStableLayerNorm(
173
+ (attention): Wav2Vec2Attention(
174
+ (k_proj): Linear(in_features=1280, out_features=1280, bias=True)
175
+ (v_proj): Linear(in_features=1280, out_features=1280, bias=True)
176
+ (q_proj): Linear(in_features=1280, out_features=1280, bias=True)
177
+ (out_proj): Linear(in_features=1280, out_features=1280, bias=True)
178
+ )
179
+ (dropout): Dropout(p=0.1, inplace=False)
180
+ (layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
181
+ (feed_forward): Wav2Vec2FeedForward(
182
+ (intermediate_dropout): Dropout(p=0.0, inplace=False)
183
+ (intermediate_dense): Linear(in_features=1280, out_features=5120, bias=True)
184
+ (intermediate_act_fn): GELUActivation()
185
+ (output_dense): Linear(in_features=5120, out_features=1280, bias=True)
186
+ (output_dropout): Dropout(p=0.1, inplace=False)
187
+ )
188
+ (final_layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
189
+ )
190
+ (40): Lang2VecConditionedLayer(
191
+ (layer): Wav2Vec2EncoderLayerStableLayerNorm(
192
+ (attention): Wav2Vec2Attention(
193
+ (k_proj): Linear(in_features=1280, out_features=1280, bias=True)
194
+ (v_proj): Linear(in_features=1280, out_features=1280, bias=True)
195
+ (q_proj): Linear(in_features=1280, out_features=1280, bias=True)
196
+ (out_proj): Linear(in_features=1280, out_features=1280, bias=True)
197
+ )
198
+ (dropout): Dropout(p=0.1, inplace=False)
199
+ (layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
200
+ (feed_forward): Wav2Vec2FeedForward(
201
+ (intermediate_dropout): Dropout(p=0.0, inplace=False)
202
+ (intermediate_dense): Linear(in_features=1280, out_features=5120, bias=True)
203
+ (intermediate_act_fn): GELUActivation()
204
+ (output_dense): Linear(in_features=5120, out_features=1280, bias=True)
205
+ (output_dropout): Dropout(p=0.1, inplace=False)
206
+ )
207
+ (final_layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
208
+ )
209
+ (encoder): IdentityEncoder()
210
+ (pooling): ChnAttnStatPooling(
211
+ (attention): Sequential(
212
+ (0): Conv1d(3840, 128, kernel_size=(1,), stride=(1,))
213
+ (1): ReLU()
214
+ (2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
215
+ (3): Conv1d(128, 1280, kernel_size=(1,), stride=(1,))
216
+ )
217
+ (softmax): Softmax(dim=2)
218
+ )
219
+ (projector): RawNet3Projector(
220
+ (bn): BatchNorm1d(2560, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
221
+ (fc): Linear(in_features=2560, out_features=192, bias=True)
222
+ )
223
+ (prediction): Linear(in_features=192, out_features=299, bias=True)
224
+ (conditioning_layer): Linear(in_features=299, out_features=1280, bias=True)
225
+ )
226
+ (41-43): 3 x Wav2Vec2EncoderLayerStableLayerNorm(
227
+ (attention): Wav2Vec2Attention(
228
+ (k_proj): Linear(in_features=1280, out_features=1280, bias=True)
229
+ (v_proj): Linear(in_features=1280, out_features=1280, bias=True)
230
+ (q_proj): Linear(in_features=1280, out_features=1280, bias=True)
231
+ (out_proj): Linear(in_features=1280, out_features=1280, bias=True)
232
+ )
233
+ (dropout): Dropout(p=0.1, inplace=False)
234
+ (layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
235
+ (feed_forward): Wav2Vec2FeedForward(
236
+ (intermediate_dropout): Dropout(p=0.0, inplace=False)
237
+ (intermediate_dense): Linear(in_features=1280, out_features=5120, bias=True)
238
+ (intermediate_act_fn): GELUActivation()
239
+ (output_dense): Linear(in_features=5120, out_features=1280, bias=True)
240
+ (output_dropout): Dropout(p=0.1, inplace=False)
241
+ )
242
+ (final_layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
243
+ )
244
+ (44): Lang2VecConditionedLayer(
245
+ (layer): Wav2Vec2EncoderLayerStableLayerNorm(
246
+ (attention): Wav2Vec2Attention(
247
+ (k_proj): Linear(in_features=1280, out_features=1280, bias=True)
248
+ (v_proj): Linear(in_features=1280, out_features=1280, bias=True)
249
+ (q_proj): Linear(in_features=1280, out_features=1280, bias=True)
250
+ (out_proj): Linear(in_features=1280, out_features=1280, bias=True)
251
+ )
252
+ (dropout): Dropout(p=0.1, inplace=False)
253
+ (layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
254
+ (feed_forward): Wav2Vec2FeedForward(
255
+ (intermediate_dropout): Dropout(p=0.0, inplace=False)
256
+ (intermediate_dense): Linear(in_features=1280, out_features=5120, bias=True)
257
+ (intermediate_act_fn): GELUActivation()
258
+ (output_dense): Linear(in_features=5120, out_features=1280, bias=True)
259
+ (output_dropout): Dropout(p=0.1, inplace=False)
260
+ )
261
+ (final_layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
262
+ )
263
+ (encoder): IdentityEncoder()
264
+ (pooling): ChnAttnStatPooling(
265
+ (attention): Sequential(
266
+ (0): Conv1d(3840, 128, kernel_size=(1,), stride=(1,))
267
+ (1): ReLU()
268
+ (2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
269
+ (3): Conv1d(128, 1280, kernel_size=(1,), stride=(1,))
270
+ )
271
+ (softmax): Softmax(dim=2)
272
+ )
273
+ (projector): RawNet3Projector(
274
+ (bn): BatchNorm1d(2560, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
275
+ (fc): Linear(in_features=2560, out_features=192, bias=True)
276
+ )
277
+ (prediction): Linear(in_features=192, out_features=299, bias=True)
278
+ (conditioning_layer): Linear(in_features=299, out_features=1280, bias=True)
279
+ )
280
+ (45-47): 3 x Wav2Vec2EncoderLayerStableLayerNorm(
281
+ (attention): Wav2Vec2Attention(
282
+ (k_proj): Linear(in_features=1280, out_features=1280, bias=True)
283
+ (v_proj): Linear(in_features=1280, out_features=1280, bias=True)
284
+ (q_proj): Linear(in_features=1280, out_features=1280, bias=True)
285
+ (out_proj): Linear(in_features=1280, out_features=1280, bias=True)
286
+ )
287
+ (dropout): Dropout(p=0.1, inplace=False)
288
+ (layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
289
+ (feed_forward): Wav2Vec2FeedForward(
290
+ (intermediate_dropout): Dropout(p=0.0, inplace=False)
291
+ (intermediate_dense): Linear(in_features=1280, out_features=5120, bias=True)
292
+ (intermediate_act_fn): GELUActivation()
293
+ (output_dense): Linear(in_features=5120, out_features=1280, bias=True)
294
+ (output_dropout): Dropout(p=0.1, inplace=False)
295
+ )
296
+ (final_layer_norm): LayerNorm((1280,), eps=1e-05, elementwise_affine=True)
297
+ )
298
+ )
299
+ )
300
+ )
301
+ )
302
+ )
303
+ (featurizer): Featurizer()
304
+ )
305
+ (normalize): UtteranceMVN(norm_means=True, norm_vars=False)
306
+ (encoder): EcapaTdnnEncoder(
307
+ (conv): Conv1d(1280, 512, kernel_size=(5,), stride=(1,), padding=(2,))
308
+ (relu): ReLU()
309
+ (bn): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
310
+ (layer1): EcapaBlock(
311
+ (conv1): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
312
+ (bn1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
313
+ (convs): ModuleList(
314
+ (0-6): 7 x Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(2,), dilation=(2,))
315
+ )
316
+ (bns): ModuleList(
317
+ (0-6): 7 x BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
318
+ )
319
+ (conv3): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
320
+ (bn3): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
321
+ (relu): ReLU()
322
+ (se): SEModule(
323
+ (se): Sequential(
324
+ (0): AdaptiveAvgPool1d(output_size=1)
325
+ (1): Conv1d(512, 128, kernel_size=(1,), stride=(1,))
326
+ (2): ReLU()
327
+ (3): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
328
+ (4): Conv1d(128, 512, kernel_size=(1,), stride=(1,))
329
+ (5): Sigmoid()
330
+ )
331
+ )
332
+ )
333
+ (layer2): EcapaBlock(
334
+ (conv1): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
335
+ (bn1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
336
+ (convs): ModuleList(
337
+ (0-6): 7 x Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(3,), dilation=(3,))
338
+ )
339
+ (bns): ModuleList(
340
+ (0-6): 7 x BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
341
+ )
342
+ (conv3): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
343
+ (bn3): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
344
+ (relu): ReLU()
345
+ (se): SEModule(
346
+ (se): Sequential(
347
+ (0): AdaptiveAvgPool1d(output_size=1)
348
+ (1): Conv1d(512, 128, kernel_size=(1,), stride=(1,))
349
+ (2): ReLU()
350
+ (3): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
351
+ (4): Conv1d(128, 512, kernel_size=(1,), stride=(1,))
352
+ (5): Sigmoid()
353
+ )
354
+ )
355
+ )
356
+ (layer3): EcapaBlock(
357
+ (conv1): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
358
+ (bn1): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
359
+ (convs): ModuleList(
360
+ (0-6): 7 x Conv1d(64, 64, kernel_size=(3,), stride=(1,), padding=(4,), dilation=(4,))
361
+ )
362
+ (bns): ModuleList(
363
+ (0-6): 7 x BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
364
+ )
365
+ (conv3): Conv1d(512, 512, kernel_size=(1,), stride=(1,))
366
+ (bn3): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
367
+ (relu): ReLU()
368
+ (se): SEModule(
369
+ (se): Sequential(
370
+ (0): AdaptiveAvgPool1d(output_size=1)
371
+ (1): Conv1d(512, 128, kernel_size=(1,), stride=(1,))
372
+ (2): ReLU()
373
+ (3): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
374
+ (4): Conv1d(128, 512, kernel_size=(1,), stride=(1,))
375
+ (5): Sigmoid()
376
+ )
377
+ )
378
+ )
379
+ (extra_layers): ModuleList()
380
+ (layer4): Conv1d(1536, 1536, kernel_size=(1,), stride=(1,))
381
+ (mp3): MaxPool1d(kernel_size=3, stride=3, padding=0, dilation=1, ceil_mode=False)
382
+ )
383
+ (pooling): ChnAttnStatPooling(
384
+ (attention): Sequential(
385
+ (0): Conv1d(4608, 128, kernel_size=(1,), stride=(1,))
386
+ (1): ReLU()
387
+ (2): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
388
+ (3): Conv1d(128, 1536, kernel_size=(1,), stride=(1,))
389
+ )
390
+ (softmax): Softmax(dim=2)
391
+ )
392
+ (projector): RawNet3Projector(
393
+ (bn): BatchNorm1d(3072, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
394
+ (fc): Linear(in_features=3072, out_features=192, bias=True)
395
+ )
396
+ (loss): ArcMarginProduct_intertopk_subcenter(
397
+ (ce): CrossEntropyLoss()
398
+ )
399
+ (lang2vec_head): Linear(in_features=192, out_features=299, bias=True)
400
+ (lang2vec_loss): MSELoss()
401
+ )
402
+
403
+ Model summary:
404
+ Class Name: ESPnetLIDModel
405
+ Total Number of model parameters: 977.14 M
406
+ Number of trainable parameters: 977.14 M (100.0%)
407
+ Size: 3.91 GB
408
+ Type: torch.float32
409
+ /home/qingzhengw/espnet/.geolid-reproduction/venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:424: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
410
+ self.check_worker_number_rationality()
411
+ /home/qingzhengw/espnet/.geolid-reproduction/review-code/espnet2/train/reporter.py:320: UserWarning: The stats of the previous epoch=-1doesn't exist.
412
+ warnings.warn(
413
+ [host-10-0-93-146] 2026-10-23 00:53:43,304 (lid_trainer:121) INFO: [Rank 0] Resume: 0 utterances found in /home/qingzhengw/espnet/.geolid-reproduction/full-dev-unified-301372e4/models/combined_shared_trainable/lids0
414
+ /home/qingzhengw/espnet/.geolid-reproduction/venv/lib/python3.12/site-packages/torch/utils/data/dataloader.py:432: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
415
+ self.check_worker_number_rationality()
416
+ [host-10-0-93-146] 2026-10-23 00:54:01,839 (lid_trainer:250) INFO: [Rank 0] Saved 200 utts at step 0
417
+ [host-10-0-93-146] 2026-10-23 00:54:18,771 (lid_trainer:250) INFO: [Rank 0] Saved 200 utts at step 1
418
+ [host-10-0-93-146] 2026-10-23 00:54:34,599 (lid_trainer:250) INFO: [Rank 0] Saved 200 utts at step 2
419
+ [host-10-0-93-146] 2026-10-23 00:54:50,150 (lid_trainer:250) INFO: [Rank 0] Saved 200 utts at step 3
420
+ [host-10-0-93-146] 2026-10-23 00:55:04,729 (lid_trainer:250) INFO: [Rank 0] Saved 200 utts at step 4
421
+ [host-10-0-93-146] 2026-10-23 00:55:19,633 (lid_trainer:250) INFO: [Rank 0] Saved 200 utts at step 5
422
+ [host-10-0-93-146] 2026-10-23 00:55:34,813 (lid_trainer:250) INFO: [Rank 0] Saved 200 utts at step 6
423
+ [host-10-0-93-146] 2026-10-23 00:55:49,924 (lid_trainer:250) INFO: [Rank 0] Saved 200 utts at step 7
424
+ [host-10-0-93-146] 2026-10-23 00:55:50,841 (lid_inference:274) INFO: args.save_embd_per_utt: False
425
+ [host-10-0-93-146] 2026-10-23 00:55:50,842 (lid_inference:305) INFO: args.save_tsne_plot: False
evaluation_prediction_differences.json ADDED
@@ -0,0 +1 @@
 
 
1
+ []
evaluation_predictions.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
evaluation_previous.json ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "dataset": "VoxLingua107",
3
+ "split": "dev",
4
+ "utterances": 1609,
5
+ "languages": 33,
6
+ "metric": "utterance-level accuracy (%)",
7
+ "entrypoint": "espnet2.bin.lid_inference",
8
+ "implementation_revision": "ca777d59eb49519e3827e40cc34997f2123d6682",
9
+ "device": "cuda",
10
+ "dtype": "float32",
11
+ "batch_size": 1,
12
+ "num_workers": 8,
13
+ "use_preprocessor": false,
14
+ "fix_duration": false,
15
+ "input_order": "wav.scp sorted by utterance ID; standard ESPnet streaming iterator",
16
+ "manifest_sha256": "6ad3c8fc6c74fab2720fe186525fc3bd20c4ab7b1be35d227c5c7f5456ba6c54",
17
+ "environment": {
18
+ "python": "3.12.3 (main, Jun 19 2026, 12:46:00) [GCC 13.3.0]",
19
+ "torch": "2.11.0+cu128",
20
+ "torch_cuda_build": "12.8",
21
+ "torchaudio": "2.11.0+cu128",
22
+ "transformers": "5.14.1",
23
+ "espnet_s3prl": "2026.5.14",
24
+ "soundfile": "0.14.0"
25
+ },
26
+ "repo_id": "espnet/geolid_combined_shared_trainable",
27
+ "model_sha256": "8a8f9dd767c141dcae0cb34a7780d7f593bc67f560267cd4d2d2fadd54f6bd16",
28
+ "correct": 1501,
29
+ "accuracy_percent": 93.2877563704164,
30
+ "paper_accuracy_percent": 94.4,
31
+ "source_hf_revision_before_layout_migration": "0c7ee08dc9ab7d9ac651c7858fd6ce40849475b6",
32
+ "batch_dependence": {
33
+ "s3prl_frame_alignment": "S3PRL reports ceil(samples/320) frames and repeats the final frame only at the batch boundary. Shorter items can instead expose padded-query frames from Wav2Vec2.",
34
+ "ecapa_tdnn": "Temporal convolutions and unmasked SE global means depend on the padded sequence length before the masked final pooling.",
35
+ "mvn_and_final_pooling_mask_padding": true,
36
+ "reported_measurement_batch_size": 1
37
+ }
38
+ }
evaluation_report.md ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Full VoxLingua107 dev evaluation after model integration
2
+
3
+ `espnet/geolid_combined_shared_trainable` classified **1501 / 1,609 utterances
4
+ correctly (93.29%)** across 33 languages.
5
+ This is a complete new inference run using `ESPnetLIDModel` at
6
+ `301372e4da311d2a3d9574a5a43b3037ae6e67c1`, completed at `2026-10-23T00:55:52.840208+00:00`.
7
+
8
+ All 1,609 predicted labels match the earlier full-dev evaluation at `ca777d59eb49519e3827e40cc34997f2123d6682`.
9
+
10
+ The measured checkpoint SHA-256 is `8a8f9dd767c141dcae0cb34a7780d7f593bc67f560267cd4d2d2fadd54f6bd16`.
11
+ The model inputs came from this immutable
12
+ [HF revision](https://huggingface.co/espnet/geolid_combined_shared_trainable/tree/613d18645d1be3bd585a2cbae0485350a2a4c027).
13
+ The checkpoint and published model configuration have not been modified by
14
+ this evaluation. Only the local path to `languages.txt` was resolved for the
15
+ run; its published `./languages.txt` value works from the model directory.
16
+
17
+ ## Reproduce inference
18
+
19
+ Install the ESPnet revision documented in the model card, and download this
20
+ model's `config.yaml`, `model.pth` and `languages.txt`. Run the following command
21
+ from that model directory. Prepare `data/voxlingua107-dev/wav.scp` as sorted
22
+ `<utterance-id> <absolute-waveform-path>` lines for all 1,609 dev utterances;
23
+ use the original full-length, mono, 16 kHz waveforms. Choose a fresh output
24
+ directory because the CLI can skip inference when a completion marker exists.
25
+
26
+ ```bash
27
+ OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 python -m espnet2.bin.lid_inference \
28
+ --output_dir evaluation_output \
29
+ --ngpu 1 --dtype float32 \
30
+ --data_path_and_name_and_type data/voxlingua107-dev/wav.scp,speech,sound \
31
+ --valid_batch_size 1 --num_workers 8 \
32
+ --lid_train_config config.yaml --lid_model_file model.pth \
33
+ --use_preprocessor false --fix_duration false \
34
+ --extract_embd false --save_embd_per_utt false \
35
+ --save_embd_avg_lang false --save_tsne_plot false \
36
+ --checkpoint_interval 200
37
+ ```
38
+
39
+ The output `evaluation_output/voxlingua107-dev_lids` contains the predicted
40
+ labels. Compare each utterance ID with the reference language in
41
+ [evaluation_predictions.jsonl](evaluation_predictions.jsonl). The reported
42
+ accuracy is `100 * correct / 1609`; no duration weighting or macro averaging
43
+ is used. The JSONL also records the predictions from this run and a correctness
44
+ flag for every utterance, so the score can be independently recomputed.
45
+
46
+ ## Evidence and scope
47
+
48
+ - [Evaluation metadata](evaluation.json): counts, settings, per-language counts,
49
+ environment, source revision and hashes.
50
+ - [All predictions and reference labels](evaluation_predictions.jsonl).
51
+ - [Complete captured inference log](evaluation.log): paths refer to the actual
52
+ evaluation workspace; its output is retained unchanged.
53
+ - [Prediction differences from the previous run](evaluation_prediction_differences.json).
54
+ - [Previous evaluation metadata](evaluation_previous.json), preserved unchanged.
55
+ - [Validation history](validation.json): earlier strict-loading checks plus a
56
+ separate record of this full evaluation through the unmodified CLI.
57
+
58
+ This run evaluates VoxLingua107 dev only. Historical paper results for other
59
+ test sets have not been remeasured here. The new full run uses the standard
60
+ loader without forcing strict loading; the earlier explicit strict-loading
61
+ check is a separate historical validation.
validation.json CHANGED
@@ -35,5 +35,32 @@
35
  "ell_2KtniiXOx90__U__S216---2054.480-2059.730.wav": "ell"
36
  },
37
  "loading_check": "Strict loading required in the CLI checkpoint loader; inference computation unchanged"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
38
  }
39
  }
 
35
  "ell_2KtniiXOx90__U__S216---2054.480-2059.730.wav": "ell"
36
  },
37
  "loading_check": "Strict loading required in the CLI checkpoint loader; inference computation unchanged"
38
+ },
39
+ "full_dev_model_integration": {
40
+ "implementation_revision": "301372e4da311d2a3d9574a5a43b3037ae6e67c1",
41
+ "model": "ESPnetLIDModel",
42
+ "model_sha256": "8a8f9dd767c141dcae0cb34a7780d7f593bc67f560267cd4d2d2fadd54f6bd16",
43
+ "source_hf_revision": "613d18645d1be3bd585a2cbae0485350a2a4c027",
44
+ "entrypoint": "espnet2.bin.lid_inference",
45
+ "loader": "Unmodified standard ESPnet CLI loader",
46
+ "strict_loading_forced": false,
47
+ "weight_file_changed": false,
48
+ "full_dev_evaluation": true,
49
+ "utterances": 1609,
50
+ "correct": 1501,
51
+ "accuracy_percent": 93.2877563704164,
52
+ "batch_size": 1,
53
+ "num_workers": 8,
54
+ "dtype": "float32",
55
+ "finished_at": "2026-10-23T00:55:52.840208+00:00",
56
+ "predictions_equal_previous": true,
57
+ "matched_predictions": 1609,
58
+ "changed_predictions": 0,
59
+ "baseline_implementation_revision": "ca777d59eb49519e3827e40cc34997f2123d6682",
60
+ "evaluation_file": "evaluation.json",
61
+ "predictions_file": "evaluation_predictions.jsonl",
62
+ "predictions_sha256": "40264d2766f60dfce717ebc638ef76bc85380709693ca3e32178829e5df5453e",
63
+ "log_file": "evaluation.log",
64
+ "log_sha256": "d6831bc2d1e154c487052b31b10adc9072db4c8d045e4a1c6897125aa7a4c6c1"
65
  }
66
  }