Whisper small for Arabic (ScriptVox)

This model is Whisper small, fine-tuned for Arabic speech, with a focus on Gulf and Emirati Arabic. It keeps the English quality of the base model. The files are in the CTranslate2 format (float16), for faster-whisper.

ScriptVox uses this model for local speech to text when the language is Arabic (--lang ar).

Item Value
Base model openai/whisper-small
Version r12-a70
Format CTranslate2, float16, 484 MB
Languages Arabic (Gulf, Emirati, other dialects, Modern Standard Arabic) and English
Licence Apache-2.0

Use

With ScriptVox:

scriptvox setup speech-local --arabic
scriptvox analyze clip.mp4 --transcript --lang ar

With faster-whisper:

from faster_whisper import WhisperModel

model = WhisperModel("scriptvox/whisper-small-ar", compute_type="int8")
segments, info = model.transcribe("clip.wav", language="ar", word_timestamps=True)
for s in segments:
    print(s.start, s.end, s.text)

Results

Word error rate (WER). Lower is better. The base is openai/whisper-small with the same decoder and the same text normalisation. The interval is a 95 percent paired bootstrap over clips.

Test set Clips Base This model Change
Gulf Arabic (speakers and prompts not in training) 402 41.32 % 27.82 % -32.7 % (-35.3 to -30.2)
Arabic, FLEURS test 428 26.36 % 17.69 % -32.9 % (-36.5 to -29.3)
Emirati Arabic (a corpus not used in training) 1,048 26.52 % 20.63 % -22.2 % (-24.7 to -19.5)
English, FLEURS test 647 6.52 % 6.44 % -1.2 % (-6.1 to +4.2)

Long recordings, about 8.5 minutes each, decoded in 30 second windows:

Recording WER Gaps longer than 5 s with no words
Emirati, 533 s 21.4 % 0
Gulf, 513 s 27.9 % 0

Word times are usable on all clips of the four test sets. The model writes punctuation on 94 to 100 percent of the clips.

Training

  1. Fine-tune openai/whisper-small for 14,000 steps at a learning rate of 1e-5.
  2. Half of the training items are 30 second windows made from several clips, each with its own start and end times. This teaches the model the form of a long recording.
  3. Mix the weights: 0.70 fine-tuned and 0.30 base. The mix keeps the English quality. The mix ratio was chosen on the dev sets. The test sets were read one time, after that choice.

Training data:

Data Licence
Meta Omnilingual ASR Corpus: Gulf Arabic and 13 other Arabic dialects CC BY 4.0
Mozilla Common Voice 18, Arabic CC0
Google FLEURS, Arabic and English CC BY 4.0
LibriSpeech, English, with text from Whisper large-v3 CC BY 4.0

No audio from YouTube or from other web video is in the training data.

Limits

  • The error rate on dialect speech is still high. Check the text before you publish it.
  • Some of the gain is in spelling of one letter (for example ه and ة). This can be a spelling change and not a hearing change.
  • The model was not tested on noisy recordings, music under speech, or speech with more than one person at a time.
  • Do not use this model for decisions about a person, such as legal, medical or hiring decisions.

Credits

  • OpenAI Whisper: the base model (Apache-2.0 model card, MIT code).
  • Meta: Omnilingual ASR Corpus (CC BY 4.0).
  • Google: FLEURS (CC BY 4.0).
  • Mozilla: Common Voice (CC0).
  • LibriSpeech: Panayotov, Chen, Povey and Khudanpur (CC BY 4.0).
  • CTranslate2 and faster-whisper: the format and the decoder.

Files

File sha256
model.bin f09579e04bcbca6f42ab69ca2eba36ecb1a0e093a9eac2c93fc3eac93a829f8c
config.json b55496ac7940a7ae47d2c01eab40edfd8701feec1229d9cce3b40014383fb828
preprocessor_config.json 994838f1fa6462c8b9b3c90edada831f11f3dd8b4664634e18f4694d005c9dbf
tokenizer.json 27fc476bfe7f17299480be2273fc0608e4d5a99aba2ab5dec5374b4482d1a566
vocabulary.json 5ad28279db5e546349708b9a74736e2c018737ac5a600f1048f1f26c99b85b47
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for scriptvox/whisper-small-ar

Finetuned
(3,850)
this model

Datasets used to train scriptvox/whisper-small-ar