Nava / README.md
xmanii's picture
Rename to Nava; update card and charts
77bb7ce verified
|
Raw History Blame Contribute Delete
13.8 kB
---
language:
- fa
license: mit
library_name: transformers
pipeline_tag: automatic-speech-recognition
base_model: openai/whisper-large-v3-turbo
tags:
- whisper
- persian
- farsi
- speech-recognition
- asr
- faster-whisper
- whisper.cpp
datasets:
- PerSets/youtube-persian-asr
- PerSets/filimo-persian-asr
- Reza2kn/Common-Voice-25-Persian-Raw
- MahtaFetrat/Mana-TTS
metrics:
- wer
- cer
---
# Nava
**Persian speech-to-text.** Nava is `openai/whisper-large-v3-turbo` (809M parameters) fine-tuned on 579 hours of filtered Persian speech.
- **Strongest on conversational Persian** among the four open models tested here.
- **Writes standard Persian spelling**, with half-spaces where they belong: «می‌شود», «سازمان‌ها».
- **Runs wherever Whisper runs**: transformers, faster-whisper, whisper.cpp.
This is a preview from a single training run. Read [Limitations](#limitations) before relying on it.
| Repo | Use it with |
|---|---|
| [Nimruz/Nava](https://huggingface.co/Nimruz/Nava) (this one) | transformers |
| [Nimruz/Nava-ct2](https://huggingface.co/Nimruz/Nava-ct2) | faster-whisper / CTranslate2 |
| [Nimruz/Nava-ggml](https://huggingface.co/Nimruz/Nava-ggml) | whisper.cpp |
## Results
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/Nimruz/Nava/resolve/main/assets/benchmark-dark.png">
<img src="https://huggingface.co/Nimruz/Nava/resolve/main/assets/benchmark-light.png" alt="Bar charts of word error rate for four Persian speech models on four test sets. Nava is lowest on C1Tech (11.7%) and PSRB (17.2%), second on FLEURS (8.7%) and Common Voice (13.2%)." width="800">
</picture>
Word error rate (%), with character error rate ignoring spaces in brackets. Lower is better.
| Model | Params | C1Tech (conversational) | PSRB (mixed, noisy) | FLEURS fa (read) | Common Voice 22 fa (read) |
|---|---|---|---|---|---|
| **Nava** | 809M | **11.7 (3.8)** | **17.2 (5.8)** | 8.7 (2.3) | 13.2 (3.3) |
| Reza2kn/Shenava-Koochik-v1.5 | 114M | 13.4 (5.2) | 22.5 (9.0) | **8.1** (3.0) | **12.5** (3.2) |
| mehdi-hf/nemotron-asr-streaming-farsi | 600M | 12.4 (4.5) | 23.1 (9.3) | 9.4 (3.2) | 18.4 (5.4) |
| givia/whisper-large-v3-turbo-persian | 809M | 13.4 (5.1) | 22.8 (9.9) | 10.1 (5.2) | 18.3 (6.4) |
| openai/whisper-large-v3-turbo (base) | 809M | not run | not run | 19.7 (6.2) | 47.2 (26.8) |
Test sets:
- **C1Tech/Persian-ASR-Benchmark**, all 2,101 clips.
- **PartAI/PSRB**, all 344 clips.
- **FLEURS** `fa_ir` test, all 871 clips.
- **Common Voice 22** Persian test, a fixed random sample of 1,500 of the 10,676 clips.
Every model was run here, with the same scorer, on the same clips. See [How the scores were computed](#how-the-scores-were-computed) for the method and for the same table under plain scoring.
## Quick start
Input is 16 kHz mono audio. Always pass the language; Nava was trained with the Persian prompt only.
### transformers
```python
import soundfile as sf
import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor
model_id = "Nimruz/Nava"
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id).to("cuda").half()
wav, sr = sf.read("audio.wav", dtype="float32") # 16 kHz mono, up to 30 seconds
features = processor(wav, sampling_rate=16000, return_tensors="pt").input_features
ids = model.generate(features.to("cuda").half(), language="fa", task="transcribe", max_new_tokens=440)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])
```
On a machine without a GPU, drop `.to("cuda")` and `.half()`.
### Any length, any audio format
The script in this repo decodes the file, cuts long audio at quiet points, and joins the text:
```bash
git clone https://huggingface.co/Nimruz/Nava && cd Nava
pip install torch transformers
python transcribe.py interview.mp3
python transcribe.py interview.mp3 --digits fa # numbers, times and dates in digits
```
It uses ffmpeg to read audio if ffmpeg is installed, and librosa otherwise.
### faster-whisper
```python
from faster_whisper import WhisperModel
model = WhisperModel("Nimruz/Nava-ct2", device="cuda", compute_type="float16")
segments, _ = model.transcribe(
"audio.wav", language="fa", beam_size=1,
without_timestamps=True, condition_on_previous_text=False,
)
print(" ".join(s.text.strip() for s in segments))
```
On CPU use `device="cpu", compute_type="float32"`, or `"int8"` for less memory.
### whisper.cpp
```bash
huggingface-cli download Nimruz/Nava-ggml ggml-nava-q5_0.bin --local-dir .
whisper-cli -m ggml-nava-q5_0.bin -l fa -nt -f audio.wav
```
| File | Size | Notes |
|---|---|---|
| `ggml-nava.bin` | 1.51 GB | 16-bit float |
| `ggml-nava-q8_0.bin` | 0.81 GB | 8-bit quantized |
| `ggml-nava-q5_0.bin` | 0.53 GB | 5-bit quantized |
whisper.cpp expects 16 kHz 16-bit WAV: `ffmpeg -i in.mp3 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav`.
### Which one to pick
| You have | Use |
|---|---|
| An NVIDIA GPU and Python | transformers or faster-whisper |
| A CPU server | faster-whisper |
| A laptop or phone, no Python | whisper.cpp with the `q5_0` or `q8_0` file |
Each export was checked against the original model on 10 FLEURS test clips: the transformers weights gave identical text on all 10, and the CTranslate2 and three whisper.cpp files on 9 of 10. The whisper.cpp files were built and tested with whisper.cpp v1.7.4.
## Things to know when running it
### Audio longer than 30 seconds
Whisper's built-in long-form mode relies on timestamp tokens, and Nava was fine-tuned without them. Do not use that mode: on the 13 FLEURS clips over 30 seconds it scored 31.9% word error that way.
Cut long audio into pieces of at most 30 seconds at quiet points and transcribe each piece. `transcribe.py` does this, and with it the same 13 clips score 11.1%. A voice-activity splitter such as `vad_filter=True` in faster-whisper should serve the same purpose, but that path has not been tested with Nava.
### Numbers, dates and times
Nava writes numbers the way they are spoken: «هزار و چهارصد و دو». The formatter in this repo turns them into digits after transcription:
```python
from nimruz.itn import itn
itn("سال هزار و چهارصد و دو بیست و پنج درصد رشد داشتیم") # سال ۱۴۰۲ ۲۵ درصد رشد داشتیم
itn("ساعت ده و نیم دوازده مرداد میام") # ساعت ۱۰:۳۰ ۱۲ مرداد میام
itn("قیمتش دو میلیون و پونصد هزار تومنه", digits="ascii") # قیمتش 2500000 تومنه
```
It is rule-based and conservative: lone small words such as «یک» and «نه» stay as words, and ordinals such as «بیست و پنجم» are left alone.
### Beam search
Greedy decoding is the default everywhere above. Five beams lower word error by 0.1 to 0.3 points (11.6 / 16.9 / 8.6 / 13.1 on the four test sets) at about half the speed.
## How the scores were computed
Persian sources disagree on word spacing: «می‌کنم», «می کنم» and «میکنم» are the same word. Scoring those as errors measures the spelling convention, not the recognition. So before comparing, reference and hypothesis both go through the same steps:
1. Rewrite spacing variants to the standard half-space forms.
2. Fold Arabic and Persian letter variants, and remove diacritics and punctuation.
3. Spell out digits as words.
4. Treat the half-space as a space.
The same table with step 1 switched off (word error rate, %):
| Model | C1Tech | PSRB | FLEURS fa | Common Voice 22 fa |
|---|---|---|---|---|
| **Nava** | 17.0 | **17.2** | 9.7 | 20.6 |
| Reza2kn/Shenava-Koochik-v1.5 | 18.0 | 22.7 | **8.9** | **19.8** |
| mehdi-hf/nemotron-asr-streaming-farsi | 17.3 | 23.3 | 10.3 | 25.7 |
| givia/whisper-large-v3-turbo-persian | **16.4** | 26.1 | 10.3 | 23.4 |
Notes on the comparison:
- Numbers for the other models were measured here and can differ from their own cards, which use other normalizers and clip selections.
- Decoding was greedy for every Whisper model, with long clips cut at quiet points for Nava and Whisper's long-form mode for the others. Shenava used its CTC head; Nemotron used its 1.12 s look-ahead setting.
- Shenava was used to filter Nava's training data (see below), so the two are not fully independent. It was not used on any test set.
To score your own outputs the same way, write one JSON object per line with `ref` and `hyp` fields to `results/<model-name>/<test-set>.jsonl` and run:
```bash
pip install jiwer
python score.py
```
## Training data
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/Nimruz/Nava/resolve/main/assets/data-dark.png">
<img src="https://huggingface.co/Nimruz/Nava/resolve/main/assets/data-light.png" alt="Hours used out of hours available per training source: YouTube subtitles 294 of 385, Filimo subtitles 176 of 244, Mana-TTS 60 of 103, Common Voice 49 of 372." width="800">
</picture>
| Source | Kind | Raw hours | Used hours |
|---|---|---|---|
| PerSets/youtube-persian-asr | YouTube subtitles, conversational | 385 | 294 (824 videos) |
| PerSets/filimo-persian-asr | Film and series subtitles | 244 | 176 (466 titles) |
| MahtaFetrat/Mana-TTS, parts 1 to 69 | Magazine readings, one speaker | 103 | 60 |
| Common Voice 25 fa, validated | Read sentences | 372 | 49 (1,120 speakers) |
| **Total** | | **1,104** | **579** |
How clips were chosen:
- **Agreement check.** Every clip was transcribed by an independent model (Shenava Koochik v1.5). Clips whose label differed from that transcript by more than 30% of characters were dropped. This removed about 120 hours, mostly subtitles that do not line up with the audio.
- **Text checks.** Clips with digits, Latin letters, URLs, stage directions, speaker labels or runs of repeated characters were dropped.
- **Speech-rate and audio checks.** Clips under 0.8 seconds, implausibly slow or fast for their text, or near silent were dropped.
- **Diversity caps.** At most 40 minutes per Common Voice speaker and 3 recordings of the same sentence; Mana-TTS capped at 60 hours so one voice does not dominate.
- **Test protection.** Common Voice test speakers and test sentences were excluded. Mana-TTS parts 70 to 77 (a public leaderboard's test set) were not used. 2% of YouTube and Filimo recordings were held out for development.
Training text was normalized to Persian letter forms, no diacritics, no punctuation, and one spacing convention (the standard half-space forms). The rules and word list for that are in `nimruz/ortho.py` and `assets/lexicon.json`.
## Training
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/Nimruz/Nava/resolve/main/assets/training-dark.png">
<img src="https://huggingface.co/Nimruz/Nava/resolve/main/assets/training-light.png" alt="Line chart of development word error rate by training step. Conversational clips fall from 45.5% to 15.1%; FLEURS dev falls from 18.3% to 8.3%." width="800">
</picture>
| | |
|---|---|
| Base model | openai/whisper-large-v3-turbo |
| Trainable | Decoder and encoder layers 9 to 32 (conv stem and first 8 encoder layers frozen) |
| Steps | 10,386 optimizer steps, effective batch 32 (8 × 4 accumulation) |
| Audio seen | About 1,590 hours, roughly 2.75 passes over the data |
| Learning rate | 1e-5 peak, 400 warmup steps, cosine decay to 1e-6 |
| Optimizer | AdamW, betas (0.9, 0.98), weight decay 0.01, gradient clipping 1.0 |
| Precision | bfloat16 autocast |
| Augmentation | SpecAugment (time and feature masking, 5% each) |
| Sample packing | Neighbouring clips concatenated up to 30 s; 15% of samples left as single clips |
| Compute | 6.5 GPU-hours on one GPU (A100 for the first 500 steps, L40S after) |
Development word error went from 18.3% to 8.3% on FLEURS dev and from 45.5% to 15.1% on held-out conversational clips (250 clips each). The conversational score was 14.1% at step 9,000 and 15.1% at the end; the released weights are the final step, which scored better on the FLEURS test set.
## Limitations
- **No punctuation.** Output is a plain stream of words.
- **Persian only.** English words are written in Persian script or mis-transcribed.
- **No timestamps**, and no built-in long-form decoding (see above).
- **Numbers are spelled out** unless the formatter is applied.
- **Spacing is not fully uniform.** Some forms are still written joined, such as «شدهاند» for «شده‌اند».
- **Behaviour on silence, music and non-speech audio has not been tested.** Whisper models can produce text where there is no speech.
- **One training run.** No second seed and no hyperparameter search.
- **Evaluation is on four public sets only**, and the Common Voice figure is from a 1,500-clip sample.
## What's in this repo
| Path | What it is |
|---|---|
| `model.safetensors`, `config.json`, tokenizer files | The model, half precision |
| `transcribe.py` | Transcribe audio of any length and format |
| `nimruz/itn.py` | Spoken numbers, dates and times to digits |
| `nimruz/textnorm.py`, `nimruz/ortho.py`, `assets/lexicon.json` | Text normalization and spacing rules used for training text and scoring |
| `nimruz/scoring.py`, `score.py` | The scorer behind the tables above |
| `assets/*.png` | The charts on this page |
## Licence and data
Nava is released under the MIT licence, the same as its base model. The four training datasets are published under CC0. The YouTube and Filimo sets consist of audio from online video with its subtitles; check that this suits your use before deploying commercially.
## Acknowledgements
Thanks to the authors of the training datasets, to the Shenava project for the model used in filtering, and to the authors of the FLEURS, Common Voice, C1Tech and PSRB test sets.