File size: 4,384 Bytes
77fd1d9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 | ---
license: apache-2.0
language:
- ar
- en
base_model: openai/whisper-small
pipeline_tag: automatic-speech-recognition
library_name: ctranslate2
tags:
- whisper
- arabic
- gulf-arabic
- emirati
- ctranslate2
- faster-whisper
- scriptvox
datasets:
- facebook/omnilingual-asr-corpus
- google/fleurs
- mozilla-foundation/common_voice_18_0
- openslr/librispeech_asr
---
# Whisper small for Arabic (ScriptVox)
This model is Whisper small, fine-tuned for Arabic speech, with a focus on Gulf and Emirati Arabic. It keeps the English quality of the base model. The files are in the CTranslate2 format (float16), for faster-whisper.
ScriptVox uses this model for local speech to text when the language is Arabic (`--lang ar`).
| Item | Value |
|---|---|
| Base model | `openai/whisper-small` |
| Version | r12-a70 |
| Format | CTranslate2, float16, 484 MB |
| Languages | Arabic (Gulf, Emirati, other dialects, Modern Standard Arabic) and English |
| Licence | Apache-2.0 |
## Use
With ScriptVox:
```sh
scriptvox setup speech-local --arabic
scriptvox analyze clip.mp4 --transcript --lang ar
```
With faster-whisper:
```python
from faster_whisper import WhisperModel
model = WhisperModel("scriptvox/whisper-small-ar", compute_type="int8")
segments, info = model.transcribe("clip.wav", language="ar", word_timestamps=True)
for s in segments:
print(s.start, s.end, s.text)
```
## Results
Word error rate (WER). Lower is better. The base is `openai/whisper-small` with the same decoder and the same text normalisation. The interval is a 95 percent paired bootstrap over clips.
| Test set | Clips | Base | This model | Change |
|---|---|---|---|---|
| Gulf Arabic (speakers and prompts not in training) | 402 | 41.32 % | 27.82 % | -32.7 % (-35.3 to -30.2) |
| Arabic, FLEURS test | 428 | 26.36 % | 17.69 % | -32.9 % (-36.5 to -29.3) |
| Emirati Arabic (a corpus not used in training) | 1,048 | 26.52 % | 20.63 % | -22.2 % (-24.7 to -19.5) |
| English, FLEURS test | 647 | 6.52 % | 6.44 % | -1.2 % (-6.1 to +4.2) |
Long recordings, about 8.5 minutes each, decoded in 30 second windows:
| Recording | WER | Gaps longer than 5 s with no words |
|---|---|---|
| Emirati, 533 s | 21.4 % | 0 |
| Gulf, 513 s | 27.9 % | 0 |
Word times are usable on all clips of the four test sets. The model writes punctuation on 94 to 100 percent of the clips.
## Training
1. Fine-tune `openai/whisper-small` for 14,000 steps at a learning rate of 1e-5.
2. Half of the training items are 30 second windows made from several clips, each with its own start and end times. This teaches the model the form of a long recording.
3. Mix the weights: 0.70 fine-tuned and 0.30 base. The mix keeps the English quality. The mix ratio was chosen on the dev sets. The test sets were read one time, after that choice.
Training data:
| Data | Licence |
|---|---|
| Meta Omnilingual ASR Corpus: Gulf Arabic and 13 other Arabic dialects | CC BY 4.0 |
| Mozilla Common Voice 18, Arabic | CC0 |
| Google FLEURS, Arabic and English | CC BY 4.0 |
| LibriSpeech, English, with text from Whisper large-v3 | CC BY 4.0 |
No audio from YouTube or from other web video is in the training data.
## Limits
- The error rate on dialect speech is still high. Check the text before you publish it.
- Some of the gain is in spelling of one letter (for example ه and ة). This can be a spelling change and not a hearing change.
- The model was not tested on noisy recordings, music under speech, or speech with more than one person at a time.
- Do not use this model for decisions about a person, such as legal, medical or hiring decisions.
## Credits
- OpenAI Whisper: the base model (Apache-2.0 model card, MIT code).
- Meta: Omnilingual ASR Corpus (CC BY 4.0).
- Google: FLEURS (CC BY 4.0).
- Mozilla: Common Voice (CC0).
- LibriSpeech: Panayotov, Chen, Povey and Khudanpur (CC BY 4.0).
- CTranslate2 and faster-whisper: the format and the decoder.
## Files
| File | sha256 |
|---|---|
| model.bin | f09579e04bcbca6f42ab69ca2eba36ecb1a0e093a9eac2c93fc3eac93a829f8c |
| config.json | b55496ac7940a7ae47d2c01eab40edfd8701feec1229d9cce3b40014383fb828 |
| preprocessor_config.json | 994838f1fa6462c8b9b3c90edada831f11f3dd8b4664634e18f4694d005c9dbf |
| tokenizer.json | 27fc476bfe7f17299480be2273fc0608e4d5a99aba2ab5dec5374b4482d1a566 |
| vocabulary.json | 5ad28279db5e546349708b9a74736e2c018737ac5a600f1048f1f26c99b85b47 |
|