GRF-ASR-project/wsp-lgv3-average-merging-final

Model Card for Model ID

This model performs asynchronous automatic speech recognition in English, especially for disfluent, specialized, and accented speeches. It was fine-tuned on Whisper large-v3 using a 15-hour United Nations Simultaneous Interpreting corpus and evaluated with two external corpora: the European Parliament Interpreting Corpus (EPIC) v2 (Lobascio et al. 2024) and the Polish Interpreting Corpus (PINC; Chmiel et al. 2024). The model achieved below-10% word error rates (WERs) on these 37 hours of data. We conducted statistical and qualitative analyses showing that our fine-tuning significantly reduced WERs on disfluent speeches and modestly on specialized vocabulary, but it appeared ineffective for accented speeches.

Model Details

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: Nannan Liu, Minhua Liu, Claudio Fantinuoli, and Li Chen
• Funded by: Hong Kong Research Grants Council under the General Research Fund scheme (No. 12612823)
• Model type: automatic speech recognition
• Language(s) (NLP): en
• License: private, research-only, non-derivative use
• Finetuned from model: Whisper large-v3

Model Sources [optional]

• Repository: [More Information Soon]
• Paper [optional]: [More Information Soon]
• Demo [optional]: [More Information Soon]

Uses

The model is intended to perform asynchronous automatic speech recognition on disfluent, specialized, and accented English speeches, which are commonly found in settings involving simultaneous interpreting.

Direct Use

The model can be used to generate transcripts for disfluent, specialized, and accented speeches, which should be particularly accurate when the speeches are noise-free, fast, long, fluent, and delivered by a native English speaker of any sex on common specialized domains, e.g., politics, society, environment, and economics.

Downstream Use

The model can be further fine-tuned with disfluent, specialized, and accented speeches. It has been plugged into an asynchronous transcription app where users can provide specialized vocabulary to the app, with the caveat that this could worsen WERs.

Out-of-Scope Use

The model will not work well for noisy speeches and those by individuals with a stutter as it may hallucinate. We noted a 1% hallucination rate in the EPIC v2 dataset.

Bias, Risks, and Limitations

Based on our statistical analysis, this model underperformed on very disfluent speeches (WERs > 10% for speeches containing more than one false start, corrected mispronouciation, and filled pause, e.g., 'uh', per 7.28 seconds). It was significantly less accurate on non-native English, slow (syllables per second < 3.5), short (audio length < 1 min), and disfluent speeches relative to native English, fluent, fast, and long speeches.

Based on our qualitative analysis, the model tended to use American written English spelling and may generate words that are phonologically similar but inaccurate at the end of 30-second segments. The latter may be due to the memory constraints of transformers.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

How to Get Started with the Model

The model is only available for personal, research-only, and non-derivative use due to the terms of use of UN Audiovisual Library (https://www.un.org/en/about-us/terms-of-use).

Training Details

Training Data

The training data were a 15-hour United Nations Simultaneous Interpreting (UNSI) corpus; see Table 1 for an overview and https://osf.io/32xp4/files/osfstorage/6a87646fe22bcc70bf6c594c for the list of speeches.

Table 1. UNSI corpus

Domain	n of recordings	Total length	n of tokens	Audio length in seconds
				mean (SD)
politics	44	5:08:25	39,511	420.58 (286.64)
society	44	3:52:41	33,600	317.28 (169.64)
environment	50	4:34:18	38,528	329.30 (177.52)
economics	17	1:24:31	12,309	298.32 (121.66)
Total	155	14:59:56	123,948	348.40 (211.43)

The audio recordings were first transcribed by professional, native English-speaking transcribers and then independently corrected and formatted according to a customized transcription convention (see https://osf.io/32xp4/files/2t8px) based on the UN Editorial Manual. Disfluencies, e.g., filled pauses and false starts, were faithfully transcribed. We required the spelling of specialized vocabulary to follow official conventions such as the UN Term Portal (https://unterm.un.org/unterm2/en/), and that of common words to match the speaker’s accent. For instance, British/Irish English spelling was used for UK and EU speakers, the latter being a requirement by the European Commission (2025).

Training Procedure

This model was fine-tuned based on the OpenAI Whisper pre-trained model Whisper-large-v3 using an end-to-end Automatic Speech Recognition (ASR) approach based on the Transformer architecture. The final model was the ensemble of models fine-tuned on all the 5 folds.

Preprocessing

We first resampled the 155 recordings to 16,000 Hz and then split them into five folds, preserving the proportion of each domain. We used 80% of the data in each fold for training and 20% for testing. We applied SpecAugment (Park et al. 2019) to create different versions of the training data by blanking out, speeding up, or slowing down parts of the audio to enable the model to be exposed to more diverse data than we had.

We pre-processed the training data in three steps. First, we time-aligned our transcripts with the recordings by sentence using ‘stabilizing timestamps for Whisper’ (https://github.com/jianfch/stable-ts) to generate accurate timestamps. Second, because the maximum accepted input duration of Whisper is 30 seconds, we split the training recordings into segments below this threshold, ensuring that each segment contained as many sentences as possible and ended at punctuation marks. Third, we removed silent segments with no corresponding transcription.

Training Hyperparameters

• Training regime
• Optimizer: AdamW
• Learning Rate: Each fold had a different learning rate, ranging from 1.5e-5 to 2e-5.
• Scheduler: cosine schedule with warmups
• Model weights initialization: inherited from Whisper pre-training weights

Evaluation

Testing Data, Factors & Metrics

The internal testing data were 20% of the UNSI corpus in each fold. We used StratifiedGroupKFold as the cross-validation method, with 'Domain' as the group.

The external, generalization evaluation data were the source English speeches in EPIC v2 and PINC; see Table 2 for an overview.

Table 2. External evaluation dataset

Corpus	n of audio files	Total length	Audio length in seconds
			mean (SD)
EPIC v2	302	15:18:25	182.47 (191.21)
PINC	227	6:56:17	110.03 (32.61)

We removed files in the external evaluation dataset containing code-switching for more than two segments. This was because we set the transcription language of Whisper to English, and it tended to translate non-English speech segments into English. We also removed corpus-specific transcription symbols and converted numbers to our format. We disregarded punctuation in EPIC v2 because it allows only ‘//’ to mark sentence-like segments.

Performance was evaluated using WER obtained with ‘jiwer’ based on raw transcripts including punctuation, capitalization, and disfluencies.

Evaluation Results

Table 3 shows the base model and our fine-tuned model's median WERs and IQRs.

Table 3. WERs of the base and fine-tuned model on three datasets

Dataset	n	Base median (IQR)	Fine-tuned median (IQR)
UNSI	155	8.30 (6.50)	6.30 (4.05)
EPIC v2	303	8.99 (7.13)	7.63 (4.82)
PINC	227	10.70 (5.92)	9.71 (5.18)

Model Card Authors

Nannan Liu, Minhua Liu, Claudio Fantinuoli, and Li Chen

Model Card Contact

nliu9@binghamton.edu, minhualiu@hkbu.edu.hk, fantinuoli@uni-mainz.de, lichen@comp.hkbu.edu.hk