Automatic Speech Recognition
Transformers
Safetensors
Persian
whisper
persian
farsi
speech-recognition
asr
faster-whisper
whisper.cpp
Instructions to use Nimruz/Nava with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Nimruz/Nava with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("automatic-speech-recognition", model="Nimruz/Nava")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq processor = AutoProcessor.from_pretrained("Nimruz/Nava") model = AutoModelForSpeechSeq2Seq.from_pretrained("Nimruz/Nava", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Add model card, charts and helper code
Browse files- .gitattributes +2 -0
- README.md +276 -0
- assets/benchmark-dark.png +3 -0
- assets/benchmark-light.png +3 -0
- assets/data-dark.png +0 -0
- assets/data-light.png +0 -0
- assets/lexicon.json +0 -0
- assets/training-dark.png +0 -0
- assets/training-light.png +0 -0
- nimruz/__init__.py +0 -0
- nimruz/itn.py +513 -0
- nimruz/ortho.py +461 -0
- nimruz/scoring.py +51 -0
- nimruz/textnorm.py +465 -0
- score.py +46 -0
- transcribe.py +96 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
assets/benchmark-dark.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
assets/benchmark-light.png filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,276 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- fa
|
| 4 |
+
license: mit
|
| 5 |
+
library_name: transformers
|
| 6 |
+
pipeline_tag: automatic-speech-recognition
|
| 7 |
+
base_model: openai/whisper-large-v3-turbo
|
| 8 |
+
tags:
|
| 9 |
+
- whisper
|
| 10 |
+
- persian
|
| 11 |
+
- farsi
|
| 12 |
+
- speech-recognition
|
| 13 |
+
- asr
|
| 14 |
+
- faster-whisper
|
| 15 |
+
- whisper.cpp
|
| 16 |
+
datasets:
|
| 17 |
+
- PerSets/youtube-persian-asr
|
| 18 |
+
- PerSets/filimo-persian-asr
|
| 19 |
+
- Reza2kn/Common-Voice-25-Persian-Raw
|
| 20 |
+
- MahtaFetrat/Mana-TTS
|
| 21 |
+
metrics:
|
| 22 |
+
- wer
|
| 23 |
+
- cer
|
| 24 |
+
---
|
| 25 |
+
|
| 26 |
+
# NAVA
|
| 27 |
+
|
| 28 |
+
**Persian speech-to-text.** NAVA is `openai/whisper-large-v3-turbo` (809M parameters) fine-tuned on 579 hours of filtered Persian speech.
|
| 29 |
+
|
| 30 |
+
- **Strongest on conversational Persian** among the four open models tested here.
|
| 31 |
+
- **Writes standard Persian spelling**, with half-spaces where they belong: «میشود», «سازمانها».
|
| 32 |
+
- **Runs wherever Whisper runs**: transformers, faster-whisper, whisper.cpp.
|
| 33 |
+
|
| 34 |
+
This is a preview from a single training run. Read [Limitations](#limitations) before relying on it.
|
| 35 |
+
|
| 36 |
+
| Repo | Use it with |
|
| 37 |
+
|---|---|
|
| 38 |
+
| [Nimruz/NAVA](https://huggingface.co/Nimruz/NAVA) (this one) | transformers |
|
| 39 |
+
| [Nimruz/NAVA-ct2](https://huggingface.co/Nimruz/NAVA-ct2) | faster-whisper / CTranslate2 |
|
| 40 |
+
| [Nimruz/NAVA-ggml](https://huggingface.co/Nimruz/NAVA-ggml) | whisper.cpp |
|
| 41 |
+
|
| 42 |
+
## Results
|
| 43 |
+
|
| 44 |
+
<picture>
|
| 45 |
+
<source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/benchmark-dark.png">
|
| 46 |
+
<img src="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/benchmark-light.png" alt="Bar charts of word error rate for four Persian speech models on four test sets. NAVA is lowest on C1Tech (11.7%) and PSRB (17.2%), second on FLEURS (8.7%) and Common Voice (13.2%)." width="800">
|
| 47 |
+
</picture>
|
| 48 |
+
|
| 49 |
+
Word error rate (%), with character error rate ignoring spaces in brackets. Lower is better.
|
| 50 |
+
|
| 51 |
+
| Model | Params | C1Tech (conversational) | PSRB (mixed, noisy) | FLEURS fa (read) | Common Voice 22 fa (read) |
|
| 52 |
+
|---|---|---|---|---|---|
|
| 53 |
+
| **NAVA** | 809M | **11.7 (3.8)** | **17.2 (5.8)** | 8.7 (2.3) | 13.2 (3.3) |
|
| 54 |
+
| Reza2kn/Shenava-Koochik-v1.5 | 114M | 13.4 (5.2) | 22.5 (9.0) | **8.1** (3.0) | **12.5** (3.2) |
|
| 55 |
+
| mehdi-hf/nemotron-asr-streaming-farsi | 600M | 12.4 (4.5) | 23.1 (9.3) | 9.4 (3.2) | 18.4 (5.4) |
|
| 56 |
+
| givia/whisper-large-v3-turbo-persian | 809M | 13.4 (5.1) | 22.8 (9.9) | 10.1 (5.2) | 18.3 (6.4) |
|
| 57 |
+
| openai/whisper-large-v3-turbo (base) | 809M | not run | not run | 19.7 (6.2) | 47.2 (26.8) |
|
| 58 |
+
|
| 59 |
+
Test sets:
|
| 60 |
+
|
| 61 |
+
- **C1Tech/Persian-ASR-Benchmark**, all 2,101 clips.
|
| 62 |
+
- **PartAI/PSRB**, all 344 clips.
|
| 63 |
+
- **FLEURS** `fa_ir` test, all 871 clips.
|
| 64 |
+
- **Common Voice 22** Persian test, a fixed random sample of 1,500 of the 10,676 clips.
|
| 65 |
+
|
| 66 |
+
Every model was run here, with the same scorer, on the same clips. See [How the scores were computed](#how-the-scores-were-computed) for the method and for the same table under plain scoring.
|
| 67 |
+
|
| 68 |
+
## Quick start
|
| 69 |
+
|
| 70 |
+
Input is 16 kHz mono audio. Always pass the language; NAVA was trained with the Persian prompt only.
|
| 71 |
+
|
| 72 |
+
### transformers
|
| 73 |
+
|
| 74 |
+
```python
|
| 75 |
+
import soundfile as sf
|
| 76 |
+
import torch
|
| 77 |
+
from transformers import WhisperForConditionalGeneration, WhisperProcessor
|
| 78 |
+
|
| 79 |
+
model_id = "Nimruz/NAVA"
|
| 80 |
+
processor = WhisperProcessor.from_pretrained(model_id)
|
| 81 |
+
model = WhisperForConditionalGeneration.from_pretrained(model_id).to("cuda").half()
|
| 82 |
+
|
| 83 |
+
wav, sr = sf.read("audio.wav", dtype="float32") # 16 kHz mono, up to 30 seconds
|
| 84 |
+
features = processor(wav, sampling_rate=16000, return_tensors="pt").input_features
|
| 85 |
+
ids = model.generate(features.to("cuda").half(), language="fa", task="transcribe", max_new_tokens=440)
|
| 86 |
+
print(processor.batch_decode(ids, skip_special_tokens=True)[0])
|
| 87 |
+
```
|
| 88 |
+
|
| 89 |
+
On a machine without a GPU, drop `.to("cuda")` and `.half()`.
|
| 90 |
+
|
| 91 |
+
### Any length, any audio format
|
| 92 |
+
|
| 93 |
+
The script in this repo decodes the file, cuts long audio at quiet points, and joins the text:
|
| 94 |
+
|
| 95 |
+
```bash
|
| 96 |
+
git clone https://huggingface.co/Nimruz/NAVA && cd NAVA
|
| 97 |
+
pip install torch transformers
|
| 98 |
+
python transcribe.py interview.mp3
|
| 99 |
+
python transcribe.py interview.mp3 --digits fa # numbers, times and dates in digits
|
| 100 |
+
```
|
| 101 |
+
|
| 102 |
+
It uses ffmpeg to read audio if ffmpeg is installed, and librosa otherwise.
|
| 103 |
+
|
| 104 |
+
### faster-whisper
|
| 105 |
+
|
| 106 |
+
```python
|
| 107 |
+
from faster_whisper import WhisperModel
|
| 108 |
+
|
| 109 |
+
model = WhisperModel("Nimruz/NAVA-ct2", device="cuda", compute_type="float16")
|
| 110 |
+
segments, _ = model.transcribe(
|
| 111 |
+
"audio.wav", language="fa", beam_size=1,
|
| 112 |
+
without_timestamps=True, condition_on_previous_text=False,
|
| 113 |
+
)
|
| 114 |
+
print(" ".join(s.text.strip() for s in segments))
|
| 115 |
+
```
|
| 116 |
+
|
| 117 |
+
On CPU use `device="cpu", compute_type="float32"`, or `"int8"` for less memory.
|
| 118 |
+
|
| 119 |
+
### whisper.cpp
|
| 120 |
+
|
| 121 |
+
```bash
|
| 122 |
+
huggingface-cli download Nimruz/NAVA-ggml ggml-nava-q5_0.bin --local-dir .
|
| 123 |
+
whisper-cli -m ggml-nava-q5_0.bin -l fa -nt -f audio.wav
|
| 124 |
+
```
|
| 125 |
+
|
| 126 |
+
| File | Size | Notes |
|
| 127 |
+
|---|---|---|
|
| 128 |
+
| `ggml-nava.bin` | 1.51 GB | 16-bit float |
|
| 129 |
+
| `ggml-nava-q8_0.bin` | 0.81 GB | 8-bit quantized |
|
| 130 |
+
| `ggml-nava-q5_0.bin` | 0.53 GB | 5-bit quantized |
|
| 131 |
+
|
| 132 |
+
whisper.cpp expects 16 kHz 16-bit WAV: `ffmpeg -i in.mp3 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav`.
|
| 133 |
+
|
| 134 |
+
### Which one to pick
|
| 135 |
+
|
| 136 |
+
| You have | Use |
|
| 137 |
+
|---|---|
|
| 138 |
+
| An NVIDIA GPU and Python | transformers or faster-whisper |
|
| 139 |
+
| A CPU server | faster-whisper |
|
| 140 |
+
| A laptop or phone, no Python | whisper.cpp with the `q5_0` or `q8_0` file |
|
| 141 |
+
|
| 142 |
+
Each export was checked against the original model on 10 FLEURS test clips: the transformers weights gave identical text on all 10, and the CTranslate2 and three whisper.cpp files on 9 of 10. The whisper.cpp files were built and tested with whisper.cpp v1.7.4.
|
| 143 |
+
|
| 144 |
+
## Things to know when running it
|
| 145 |
+
|
| 146 |
+
### Audio longer than 30 seconds
|
| 147 |
+
|
| 148 |
+
Whisper's built-in long-form mode relies on timestamp tokens, and NAVA was fine-tuned without them. Do not use that mode: on the 13 FLEURS clips over 30 seconds it scored 31.9% word error that way.
|
| 149 |
+
|
| 150 |
+
Cut long audio into pieces of at most 30 seconds at quiet points and transcribe each piece. `transcribe.py` does this, and with it the same 13 clips score 11.1%. A voice-activity splitter such as `vad_filter=True` in faster-whisper should serve the same purpose, but that path has not been tested with NAVA.
|
| 151 |
+
|
| 152 |
+
### Numbers, dates and times
|
| 153 |
+
|
| 154 |
+
NAVA writes numbers the way they are spoken: «هزار و چهارصد و دو». The formatter in this repo turns them into digits after transcription:
|
| 155 |
+
|
| 156 |
+
```python
|
| 157 |
+
from nimruz.itn import itn
|
| 158 |
+
|
| 159 |
+
itn("سال هزار و چهارصد و دو بیست و پنج درصد رشد داشتیم") # سال ۱۴۰۲ ۲۵ درصد رشد داشتیم
|
| 160 |
+
itn("ساعت ده و نیم دوازده مرداد میام") # ساعت ۱۰:۳۰ ۱۲ مرداد میام
|
| 161 |
+
itn("قیمتش دو میلیون و پونصد هزار تومنه", digits="ascii") # قیمتش 2500000 تومنه
|
| 162 |
+
```
|
| 163 |
+
|
| 164 |
+
It is rule-based and conservative: lone small words such as «یک» and «نه» stay as words, and ordinals such as «بیست و پنجم» are left alone.
|
| 165 |
+
|
| 166 |
+
### Beam search
|
| 167 |
+
|
| 168 |
+
Greedy decoding is the default everywhere above. Five beams lower word error by 0.1 to 0.3 points (11.6 / 16.9 / 8.6 / 13.1 on the four test sets) at about half the speed.
|
| 169 |
+
|
| 170 |
+
## How the scores were computed
|
| 171 |
+
|
| 172 |
+
Persian sources disagree on word spacing: «میکنم», «می کنم» and «میکنم» are the same word. Scoring those as errors measures the spelling convention, not the recognition. So before comparing, reference and hypothesis both go through the same steps:
|
| 173 |
+
|
| 174 |
+
1. Rewrite spacing variants to the standard half-space forms.
|
| 175 |
+
2. Fold Arabic and Persian letter variants, and remove diacritics and punctuation.
|
| 176 |
+
3. Spell out digits as words.
|
| 177 |
+
4. Treat the half-space as a space.
|
| 178 |
+
|
| 179 |
+
The same table with step 1 switched off (word error rate, %):
|
| 180 |
+
|
| 181 |
+
| Model | C1Tech | PSRB | FLEURS fa | Common Voice 22 fa |
|
| 182 |
+
|---|---|---|---|---|
|
| 183 |
+
| **NAVA** | 17.0 | **17.2** | 9.7 | 20.6 |
|
| 184 |
+
| Reza2kn/Shenava-Koochik-v1.5 | 18.0 | 22.7 | **8.9** | **19.8** |
|
| 185 |
+
| mehdi-hf/nemotron-asr-streaming-farsi | 17.3 | 23.3 | 10.3 | 25.7 |
|
| 186 |
+
| givia/whisper-large-v3-turbo-persian | **16.4** | 26.1 | 10.3 | 23.4 |
|
| 187 |
+
|
| 188 |
+
Notes on the comparison:
|
| 189 |
+
|
| 190 |
+
- Numbers for the other models were measured here and can differ from their own cards, which use other normalizers and clip selections.
|
| 191 |
+
- Decoding was greedy for every Whisper model, with long clips cut at quiet points for NAVA and Whisper's long-form mode for the others. Shenava used its CTC head; Nemotron used its 1.12 s look-ahead setting.
|
| 192 |
+
- Shenava was used to filter NAVA's training data (see below), so the two are not fully independent. It was not used on any test set.
|
| 193 |
+
|
| 194 |
+
To score your own outputs the same way, write one JSON object per line with `ref` and `hyp` fields to `results/<model-name>/<test-set>.jsonl` and run:
|
| 195 |
+
|
| 196 |
+
```bash
|
| 197 |
+
pip install jiwer
|
| 198 |
+
python score.py
|
| 199 |
+
```
|
| 200 |
+
|
| 201 |
+
## Training data
|
| 202 |
+
|
| 203 |
+
<picture>
|
| 204 |
+
<source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/data-dark.png">
|
| 205 |
+
<img src="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/data-light.png" alt="Hours used out of hours available per training source: YouTube subtitles 294 of 385, Filimo subtitles 176 of 244, Mana-TTS 60 of 103, Common Voice 49 of 372." width="800">
|
| 206 |
+
</picture>
|
| 207 |
+
|
| 208 |
+
| Source | Kind | Raw hours | Used hours |
|
| 209 |
+
|---|---|---|---|
|
| 210 |
+
| PerSets/youtube-persian-asr | YouTube subtitles, conversational | 385 | 294 (824 videos) |
|
| 211 |
+
| PerSets/filimo-persian-asr | Film and series subtitles | 244 | 176 (466 titles) |
|
| 212 |
+
| MahtaFetrat/Mana-TTS, parts 1 to 69 | Magazine readings, one speaker | 103 | 60 |
|
| 213 |
+
| Common Voice 25 fa, validated | Read sentences | 372 | 49 (1,120 speakers) |
|
| 214 |
+
| **Total** | | **1,104** | **579** |
|
| 215 |
+
|
| 216 |
+
How clips were chosen:
|
| 217 |
+
|
| 218 |
+
- **Agreement check.** Every clip was transcribed by an independent model (Shenava Koochik v1.5). Clips whose label differed from that transcript by more than 30% of characters were dropped. This removed about 120 hours, mostly subtitles that do not line up with the audio.
|
| 219 |
+
- **Text checks.** Clips with digits, Latin letters, URLs, stage directions, speaker labels or runs of repeated characters were dropped.
|
| 220 |
+
- **Speech-rate and audio checks.** Clips under 0.8 seconds, implausibly slow or fast for their text, or near silent were dropped.
|
| 221 |
+
- **Diversity caps.** At most 40 minutes per Common Voice speaker and 3 recordings of the same sentence; Mana-TTS capped at 60 hours so one voice does not dominate.
|
| 222 |
+
- **Test protection.** Common Voice test speakers and test sentences were excluded. Mana-TTS parts 70 to 77 (a public leaderboard's test set) were not used. 2% of YouTube and Filimo recordings were held out for development.
|
| 223 |
+
|
| 224 |
+
Training text was normalized to Persian letter forms, no diacritics, no punctuation, and one spacing convention (the standard half-space forms). The rules and word list for that are in `nimruz/ortho.py` and `assets/lexicon.json`.
|
| 225 |
+
|
| 226 |
+
## Training
|
| 227 |
+
|
| 228 |
+
<picture>
|
| 229 |
+
<source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/training-dark.png">
|
| 230 |
+
<img src="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/training-light.png" alt="Line chart of development word error rate by training step. Conversational clips fall from 45.5% to 15.1%; FLEURS dev falls from 18.3% to 8.3%." width="800">
|
| 231 |
+
</picture>
|
| 232 |
+
|
| 233 |
+
| | |
|
| 234 |
+
|---|---|
|
| 235 |
+
| Base model | openai/whisper-large-v3-turbo |
|
| 236 |
+
| Trainable | Decoder and encoder layers 9 to 32 (conv stem and first 8 encoder layers frozen) |
|
| 237 |
+
| Steps | 10,386 optimizer steps, effective batch 32 (8 × 4 accumulation) |
|
| 238 |
+
| Audio seen | About 1,590 hours, roughly 2.75 passes over the data |
|
| 239 |
+
| Learning rate | 1e-5 peak, 400 warmup steps, cosine decay to 1e-6 |
|
| 240 |
+
| Optimizer | AdamW, betas (0.9, 0.98), weight decay 0.01, gradient clipping 1.0 |
|
| 241 |
+
| Precision | bfloat16 autocast |
|
| 242 |
+
| Augmentation | SpecAugment (time and feature masking, 5% each) |
|
| 243 |
+
| Sample packing | Neighbouring clips concatenated up to 30 s; 15% of samples left as single clips |
|
| 244 |
+
| Compute | 6.5 GPU-hours on one GPU (A100 for the first 500 steps, L40S after) |
|
| 245 |
+
|
| 246 |
+
Development word error went from 18.3% to 8.3% on FLEURS dev and from 45.5% to 15.1% on held-out conversational clips (250 clips each). The conversational score was 14.1% at step 9,000 and 15.1% at the end; the released weights are the final step, which scored better on the FLEURS test set.
|
| 247 |
+
|
| 248 |
+
## Limitations
|
| 249 |
+
|
| 250 |
+
- **No punctuation.** Output is a plain stream of words.
|
| 251 |
+
- **Persian only.** English words are written in Persian script or mis-transcribed.
|
| 252 |
+
- **No timestamps**, and no built-in long-form decoding (see above).
|
| 253 |
+
- **Numbers are spelled out** unless the formatter is applied.
|
| 254 |
+
- **Spacing is not fully uniform.** Some forms are still written joined, such as «شدهاند» for «شدهاند».
|
| 255 |
+
- **Behaviour on silence, music and non-speech audio has not been tested.** Whisper models can produce text where there is no speech.
|
| 256 |
+
- **One training run.** No second seed and no hyperparameter search.
|
| 257 |
+
- **Evaluation is on four public sets only**, and the Common Voice figure is from a 1,500-clip sample.
|
| 258 |
+
|
| 259 |
+
## What's in this repo
|
| 260 |
+
|
| 261 |
+
| Path | What it is |
|
| 262 |
+
|---|---|
|
| 263 |
+
| `model.safetensors`, `config.json`, tokenizer files | The model, half precision |
|
| 264 |
+
| `transcribe.py` | Transcribe audio of any length and format |
|
| 265 |
+
| `nimruz/itn.py` | Spoken numbers, dates and times to digits |
|
| 266 |
+
| `nimruz/textnorm.py`, `nimruz/ortho.py`, `assets/lexicon.json` | Text normalization and spacing rules used for training text and scoring |
|
| 267 |
+
| `nimruz/scoring.py`, `score.py` | The scorer behind the tables above |
|
| 268 |
+
| `assets/*.png` | The charts on this page |
|
| 269 |
+
|
| 270 |
+
## Licence and data
|
| 271 |
+
|
| 272 |
+
NAVA is released under the MIT licence, the same as its base model. The four training datasets are published under CC0. The YouTube and Filimo sets consist of audio from online video with its subtitles; check that this suits your use before deploying commercially.
|
| 273 |
+
|
| 274 |
+
## Acknowledgements
|
| 275 |
+
|
| 276 |
+
Thanks to the authors of the training datasets, to the Shenava project for the model used in filtering, and to the authors of the FLEURS, Common Voice, C1Tech and PSRB test sets.
|
assets/benchmark-dark.png
ADDED
|
Git LFS Details
|
assets/benchmark-light.png
ADDED
|
Git LFS Details
|
assets/data-dark.png
ADDED
|
assets/data-light.png
ADDED
|
assets/lexicon.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
assets/training-dark.png
ADDED
|
assets/training-light.png
ADDED
|
nimruz/__init__.py
ADDED
|
File without changes
|
nimruz/itn.py
ADDED
|
@@ -0,0 +1,513 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Rule-based Persian (Farsi) inverse text normalisation (ITN).
|
| 2 |
+
|
| 3 |
+
The ASR model writes numbers the way they are spoken ("هزار و چهارصد و دو");
|
| 4 |
+
``itn`` rewrites cardinals, percentages, decimals, clock times, dates and
|
| 5 |
+
spoken digit strings as digits ("۱۴۰۲"). It is deliberately conservative:
|
| 6 |
+
anything that is not a well-formed number expression is left untouched,
|
| 7 |
+
byte for byte.
|
| 8 |
+
|
| 9 |
+
Standard library only; all tables and regexes are built once at import time
|
| 10 |
+
(no I/O). Invisible or easily-confused characters are written as ``\\uXXXX``
|
| 11 |
+
escapes with a comment naming them. Every yeh below is PERSIAN YEH U+06CC and
|
| 12 |
+
every kaf is KEHEH U+06A9 (enforced by a test); input with Arabic yeh/kaf is
|
| 13 |
+
folded before lookup.
|
| 14 |
+
"""
|
| 15 |
+
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
import re
|
| 19 |
+
|
| 20 |
+
__all__ = ["itn", "words_to_int"]
|
| 21 |
+
|
| 22 |
+
ZWNJ = "" # ZERO WIDTH NON-JOINER (nim-fasele)
|
| 23 |
+
|
| 24 |
+
# ---------------------------------------------------------------------------
|
| 25 |
+
# Lexicon
|
| 26 |
+
# ---------------------------------------------------------------------------
|
| 27 |
+
|
| 28 |
+
# Word classes of the number grammar.
|
| 29 |
+
_Z, _O, _T, _D, _H, _S = range(6) # zero, 1-9, 10-19, 20-90, 100-900, scales
|
| 30 |
+
|
| 31 |
+
_AND = "و" # WAW, the conjunction "va"
|
| 32 |
+
|
| 33 |
+
_LEX: dict[str, tuple[int, int]] = {} # normalised word -> (value, class)
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def _add(cls: int, value: int, *spellings: str) -> None:
|
| 37 |
+
for s in spellings:
|
| 38 |
+
_LEX[s] = (value, cls)
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
_add(_Z, 0, "صفر")
|
| 42 |
+
_add(_O, 1, "یک", "یه") # "یه" only ever converts inside a larger number
|
| 43 |
+
_add(_O, 2, "دو")
|
| 44 |
+
_add(_O, 3, "سه")
|
| 45 |
+
_add(_O, 4, "چهار", "چار")
|
| 46 |
+
_add(_O, 5, "پنج")
|
| 47 |
+
_add(_O, 6, "شش", "شیش")
|
| 48 |
+
_add(_O, 7, "هفت")
|
| 49 |
+
_add(_O, 8, "هشت")
|
| 50 |
+
_add(_O, 9, "نه")
|
| 51 |
+
_add(_T, 10, "ده")
|
| 52 |
+
_add(_T, 11, "یازده")
|
| 53 |
+
_add(_T, 12, "دوازده")
|
| 54 |
+
_add(_T, 13, "سیزده")
|
| 55 |
+
_add(_T, 14, "چهارده", "چارده")
|
| 56 |
+
_add(_T, 15, "پانزده", "پونزده")
|
| 57 |
+
_add(_T, 16, "شانزده", "شونزده")
|
| 58 |
+
_add(_T, 17, "هفده", "هیفده", "هیوده")
|
| 59 |
+
_add(_T, 18, "هجده", "هیجده", "هژده", "هیژده")
|
| 60 |
+
_add(_T, 19, "نوزده")
|
| 61 |
+
_add(_D, 20, "بیست", "بیس")
|
| 62 |
+
_add(_D, 30, "سی")
|
| 63 |
+
_add(_D, 40, "چهل")
|
| 64 |
+
_add(_D, 50, "پنجاه")
|
| 65 |
+
_add(_D, 60, "شصت")
|
| 66 |
+
_add(_D, 70, "هفتاد")
|
| 67 |
+
_add(_D, 80, "هشتاد")
|
| 68 |
+
_add(_D, 90, "نود")
|
| 69 |
+
_add(_H, 100, "صد", "یکصد")
|
| 70 |
+
_add(_H, 200, "دویست", "دویس", "دوصد")
|
| 71 |
+
_add(_H, 300, "سیصد", "سهصد")
|
| 72 |
+
_add(_H, 400, "چهارصد", "چارصد")
|
| 73 |
+
_add(_H, 500, "پانصد", "پونصد", "پنجصد")
|
| 74 |
+
_add(_H, 600, "ششصد", "شیشصد")
|
| 75 |
+
_add(_H, 700, "هفتصد")
|
| 76 |
+
_add(_H, 800, "هشتصد")
|
| 77 |
+
_add(_H, 900, "نهصد")
|
| 78 |
+
_add(_S, 10**3, "هزار")
|
| 79 |
+
_add(_S, 10**6, "میلیون", "ملیون")
|
| 80 |
+
_add(_S, 10**9, "میلیارد", "ملیارد")
|
| 81 |
+
_add(_S, 10**12, "تریلیون")
|
| 82 |
+
|
| 83 |
+
_HUNDRED = "صد"
|
| 84 |
+
_YE_COLLOQ = "یه" # colloquial "one"; also the indefinite article
|
| 85 |
+
_POINT = "ممیز"
|
| 86 |
+
_PERCENT = "درصد"
|
| 87 |
+
_SAAT = "ساعت"
|
| 88 |
+
_MINUTE = "دقیقه"
|
| 89 |
+
_HALF = "نیم"
|
| 90 |
+
_QUARTER = "ربع"
|
| 91 |
+
_TO = "به"
|
| 92 |
+
_DURATION_WORDS = frozenset({_QUARTER, _MINUTE, "ثانیه", _SAAT})
|
| 93 |
+
|
| 94 |
+
_MONTHS = frozenset({
|
| 95 |
+
# Solar Hijri
|
| 96 |
+
"فروردین", "اردیبهشت", "خرداد", "تیر", "مرداد", "امرداد", "شهریور",
|
| 97 |
+
"مهر", "آبان", "آذر", "دی", "بهمن", "اسفند",
|
| 98 |
+
# Gregorian, as written in Persian
|
| 99 |
+
"ژانویه", "فوریه", "مارس", "آوریل", "مه", "می", "ژوئن", "ژوئیه", "جولای",
|
| 100 |
+
"اوت", "آگوست", "سپتامبر", "اکتبر", "نوامبر", "دسامبر",
|
| 101 |
+
})
|
| 102 |
+
# Month names that are also common words (arrow, seal/kindness, "dey", fog,
|
| 103 |
+
# the verb prefix "mi"): a lone 1-9 day before them converts only if a year
|
| 104 |
+
# follows.
|
| 105 |
+
_AMBIGUOUS_MONTHS = frozenset({"تیر", "مهر", "دی", "مه", "می"})
|
| 106 |
+
|
| 107 |
+
# ---------------------------------------------------------------------------
|
| 108 |
+
# Character handling
|
| 109 |
+
# ---------------------------------------------------------------------------
|
| 110 |
+
|
| 111 |
+
# Arabic-script letters, tatweel and harakat (no digits, no punctuation).
|
| 112 |
+
_LETTER = (
|
| 113 |
+
"ء-غ" # hamza .. ghain
|
| 114 |
+
"ـ-ٟ" # tatweel, feh .. Arabic yeh, harakat
|
| 115 |
+
"ٰ-ۓ" # superscript alef .. yeh barree (Persian letters)
|
| 116 |
+
"ەۺ-ۼۿ"
|
| 117 |
+
)
|
| 118 |
+
_DIGIT = "0-9٠-٩۰-۹" # ASCII, Arabic-Indic, Persian
|
| 119 |
+
_WORD_PAT = f"[{_LETTER}]+(?:{ZWNJ}+[{_LETTER}]+)*"
|
| 120 |
+
# A digit token keeps its internal separators: 1,234 ۳٫۵ 10:30 1402/05/12
|
| 121 |
+
_DIGITS_PAT = f"[{_DIGIT}]+(?:[.,:/٫٬][{_DIGIT}]+)*" # + ARABIC DECIMAL / THOUSANDS SEPARATOR
|
| 122 |
+
_TOKEN_RE = re.compile(f"(?P<w>{_WORD_PAT})|(?P<d>{_DIGITS_PAT})")
|
| 123 |
+
# Gap allowed inside a number phrase: horizontal whitespace only.
|
| 124 |
+
_SPACE_GAP_RE = re.compile("[ \t ]+") # space, tab, NBSP, NARROW NBSP
|
| 125 |
+
|
| 126 |
+
_NORM_TABLE: dict[int, str | None] = {
|
| 127 |
+
0x064A: "ی", # ARABIC YEH -> PERSIAN YEH
|
| 128 |
+
0x0649: "ی", # ALEF MAKSURA -> PERSIAN YEH
|
| 129 |
+
0x0643: "ک", # ARABIC KAF -> PERSIAN KEHEH
|
| 130 |
+
0x200C: None, # ZWNJ
|
| 131 |
+
0x0640: None, # TATWEEL
|
| 132 |
+
0x0670: None, # SUPERSCRIPT ALEF
|
| 133 |
+
}
|
| 134 |
+
for _cp in range(0x064B, 0x0660): # harakat
|
| 135 |
+
_NORM_TABLE[_cp] = None
|
| 136 |
+
|
| 137 |
+
_TO_FA = {ord(str(i)): chr(0x06F0 + i) for i in range(10)} # -> EXTENDED ARABIC-INDIC
|
| 138 |
+
_TO_FA.update({0x0660 + i: chr(0x06F0 + i) for i in range(10)})
|
| 139 |
+
_TO_ASCII = {0x06F0 + i: str(i) for i in range(10)}
|
| 140 |
+
_TO_ASCII.update({0x0660 + i: str(i) for i in range(10)})
|
| 141 |
+
_DECIMAL_SEP = {"fa": "٫", "ascii": "."} # ARABIC DECIMAL SEPARATOR
|
| 142 |
+
|
| 143 |
+
|
| 144 |
+
def _norm(word: str) -> str:
|
| 145 |
+
return word.translate(_NORM_TABLE)
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
def _lookup(norm: str) -> tuple[int, int, bool] | None:
|
| 149 |
+
"""(value, class, glued_and) for a number word, else None."""
|
| 150 |
+
hit = _LEX.get(norm)
|
| 151 |
+
if hit is not None:
|
| 152 |
+
return hit[0], hit[1], False
|
| 153 |
+
# "هزارو", "صدو": conjunction glued on. Not for 1-9 ("سهو" = mistake).
|
| 154 |
+
if len(norm) > 2 and norm[-1] == _AND:
|
| 155 |
+
hit = _LEX.get(norm[:-1])
|
| 156 |
+
if hit is not None and hit[1] not in (_O, _Z):
|
| 157 |
+
return hit[0], hit[1], True
|
| 158 |
+
return None
|
| 159 |
+
|
| 160 |
+
|
| 161 |
+
def _merge_hundreds(a: str, b: str) -> tuple[int, int, bool] | None:
|
| 162 |
+
""""سی" + "صد" -> 300 when a hundreds word was split by a space."""
|
| 163 |
+
if not b.startswith(_HUNDRED):
|
| 164 |
+
return None
|
| 165 |
+
hit = _lookup(a + b)
|
| 166 |
+
return hit if hit is not None and hit[1] == _H else None
|
| 167 |
+
|
| 168 |
+
|
| 169 |
+
def _is_ordinal(norm: str) -> bool:
|
| 170 |
+
""""پنجم", "پنجمین", "سوم", "سیام", "هزارمین" ..."""
|
| 171 |
+
if norm.endswith("مین"):
|
| 172 |
+
norm = norm[:-2]
|
| 173 |
+
if norm.endswith("ام"):
|
| 174 |
+
base = norm[:-2]
|
| 175 |
+
elif norm.endswith("م"):
|
| 176 |
+
base = norm[:-1]
|
| 177 |
+
else:
|
| 178 |
+
return False
|
| 179 |
+
return base in _LEX or base == "سو"
|
| 180 |
+
|
| 181 |
+
|
| 182 |
+
# ---------------------------------------------------------------------------
|
| 183 |
+
# Grammar
|
| 184 |
+
# ---------------------------------------------------------------------------
|
| 185 |
+
|
| 186 |
+
_AND_ITEM = None # marker for "va" in a flattened item list
|
| 187 |
+
|
| 188 |
+
|
| 189 |
+
def _can_follow(small: list[int], cls: int) -> bool:
|
| 190 |
+
if not small:
|
| 191 |
+
return cls in (_O, _T, _D, _H)
|
| 192 |
+
last = small[-1]
|
| 193 |
+
return (last == _H and cls in (_O, _T, _D)) or (last == _D and cls == _O)
|
| 194 |
+
|
| 195 |
+
|
| 196 |
+
def _parse(items: list) -> int | None:
|
| 197 |
+
"""Value of a flattened item list [(value, class) | None-for-va], or None."""
|
| 198 |
+
if not items or items[0] is _AND_ITEM or items[-1] is _AND_ITEM:
|
| 199 |
+
return None
|
| 200 |
+
if len(items) == 1 and items[0][1] == _Z:
|
| 201 |
+
return 0
|
| 202 |
+
parts: list[list[tuple[int, int]]] = [[]]
|
| 203 |
+
for it in items:
|
| 204 |
+
if it is _AND_ITEM:
|
| 205 |
+
parts.append([])
|
| 206 |
+
else:
|
| 207 |
+
parts[-1].append(it)
|
| 208 |
+
total = 0
|
| 209 |
+
prev_scale: int | None = None
|
| 210 |
+
small: list[int] = []
|
| 211 |
+
small_val = 0
|
| 212 |
+
for part in parts:
|
| 213 |
+
if not part or len(part) > 2 or any(c == _Z for _, c in part):
|
| 214 |
+
return None
|
| 215 |
+
if len(part) == 2: # "<small> <scale>": "پنج هزار", "سیصد میلیون"
|
| 216 |
+
(v1, c1), (v2, c2) = part
|
| 217 |
+
if c2 != _S or not _can_follow(small, c1):
|
| 218 |
+
return None
|
| 219 |
+
small_val += v1
|
| 220 |
+
scale = v2
|
| 221 |
+
else:
|
| 222 |
+
v, c = part[0]
|
| 223 |
+
if c != _S:
|
| 224 |
+
if not _can_follow(small, c):
|
| 225 |
+
return None
|
| 226 |
+
small.append(c)
|
| 227 |
+
small_val += v
|
| 228 |
+
continue
|
| 229 |
+
if small or v != 1000: # only "هزار" stands alone
|
| 230 |
+
return None
|
| 231 |
+
small_val, scale = 1, v
|
| 232 |
+
if prev_scale is not None and scale >= prev_scale:
|
| 233 |
+
return None
|
| 234 |
+
total += small_val * scale
|
| 235 |
+
prev_scale, small, small_val = scale, [], 0
|
| 236 |
+
return total + small_val
|
| 237 |
+
|
| 238 |
+
|
| 239 |
+
def words_to_int(words: list[str]) -> int | None:
|
| 240 |
+
"""Parse a cardinal phrase already split into tokens; None if invalid.
|
| 241 |
+
|
| 242 |
+
``["هزار", "و", "چهارصد", "و", "دو"] -> 1402``. Accepts the colloquial
|
| 243 |
+
spellings, a glued "و" ("هزارو") and a split hundreds word (["سی", "صد"]).
|
| 244 |
+
A lone "یه" is not a number.
|
| 245 |
+
"""
|
| 246 |
+
norms = [_norm(t) for w in words for t in w.split()]
|
| 247 |
+
if not norms:
|
| 248 |
+
return None
|
| 249 |
+
items: list = []
|
| 250 |
+
i = 0
|
| 251 |
+
n_units = 0
|
| 252 |
+
while i < len(norms):
|
| 253 |
+
w = norms[i]
|
| 254 |
+
hit = _merge_hundreds(w, norms[i + 1]) if i + 1 < len(norms) else None
|
| 255 |
+
if hit is not None:
|
| 256 |
+
i += 2
|
| 257 |
+
else:
|
| 258 |
+
i += 1
|
| 259 |
+
if w == _AND:
|
| 260 |
+
items.append(_AND_ITEM)
|
| 261 |
+
continue
|
| 262 |
+
hit = _lookup(w)
|
| 263 |
+
if hit is None:
|
| 264 |
+
return None
|
| 265 |
+
n_units += 1
|
| 266 |
+
items.append((hit[0], hit[1]))
|
| 267 |
+
if hit[2]:
|
| 268 |
+
items.append(_AND_ITEM)
|
| 269 |
+
if n_units == 1 and norms == [_YE_COLLOQ]:
|
| 270 |
+
return None
|
| 271 |
+
return _parse(items)
|
| 272 |
+
|
| 273 |
+
|
| 274 |
+
# ---------------------------------------------------------------------------
|
| 275 |
+
# Scanner
|
| 276 |
+
# ---------------------------------------------------------------------------
|
| 277 |
+
|
| 278 |
+
_NUM, _VA, _WORD, _DIGITS = range(4) # unit kinds
|
| 279 |
+
|
| 280 |
+
|
| 281 |
+
class _Unit:
|
| 282 |
+
__slots__ = ("kind", "start", "end", "norm", "value", "cls", "glued")
|
| 283 |
+
|
| 284 |
+
def __init__(self, kind, start, end, norm="", value=0, cls=-1, glued=False):
|
| 285 |
+
self.kind, self.start, self.end, self.norm = kind, start, end, norm
|
| 286 |
+
self.value, self.cls, self.glued = value, cls, glued
|
| 287 |
+
|
| 288 |
+
|
| 289 |
+
def _tokenize(text: str) -> list[_Unit]:
|
| 290 |
+
raw = [(m.start(), m.end(), m.lastgroup) for m in _TOKEN_RE.finditer(text)]
|
| 291 |
+
units: list[_Unit] = []
|
| 292 |
+
i = 0
|
| 293 |
+
while i < len(raw):
|
| 294 |
+
s, e, g = raw[i]
|
| 295 |
+
if g == "d":
|
| 296 |
+
units.append(_Unit(_DIGITS, s, e))
|
| 297 |
+
i += 1
|
| 298 |
+
continue
|
| 299 |
+
norm = _norm(text[s:e])
|
| 300 |
+
if i + 1 < len(raw) and raw[i + 1][2] == "w" and _SPACE_GAP_RE.fullmatch(text[e:raw[i + 1][0]]):
|
| 301 |
+
s2, e2, _ = raw[i + 1]
|
| 302 |
+
hit = _merge_hundreds(norm, _norm(text[s2:e2]))
|
| 303 |
+
if hit is not None:
|
| 304 |
+
units.append(_Unit(_NUM, s, e2, norm, *hit))
|
| 305 |
+
i += 2
|
| 306 |
+
continue
|
| 307 |
+
if norm == _AND:
|
| 308 |
+
units.append(_Unit(_VA, s, e, norm))
|
| 309 |
+
else:
|
| 310 |
+
hit = _lookup(norm)
|
| 311 |
+
if hit is None:
|
| 312 |
+
units.append(_Unit(_WORD, s, e, norm))
|
| 313 |
+
else:
|
| 314 |
+
units.append(_Unit(_NUM, s, e, norm, *hit))
|
| 315 |
+
i += 1
|
| 316 |
+
return units
|
| 317 |
+
|
| 318 |
+
|
| 319 |
+
class _Scanner:
|
| 320 |
+
def __init__(self, text: str, sep: str):
|
| 321 |
+
self.text = text
|
| 322 |
+
self.sep = sep
|
| 323 |
+
self.u = _tokenize(text)
|
| 324 |
+
self.n = len(self.u)
|
| 325 |
+
# linked[i]: unit i follows unit i-1 with only horizontal whitespace between.
|
| 326 |
+
self.linked = [False] + [
|
| 327 |
+
bool(_SPACE_GAP_RE.fullmatch(text[self.u[k - 1].end:self.u[k].start]))
|
| 328 |
+
for k in range(1, self.n)
|
| 329 |
+
]
|
| 330 |
+
self.reps: list[tuple[int, int, str]] = []
|
| 331 |
+
|
| 332 |
+
# -- helpers -----------------------------------------------------------
|
| 333 |
+
|
| 334 |
+
def _is_word(self, k: int, words) -> bool:
|
| 335 |
+
return k < self.n and self.linked[k] and self.u[k].kind == _WORD and self.u[k].norm in words
|
| 336 |
+
|
| 337 |
+
def _is_num(self, k: int) -> bool:
|
| 338 |
+
return k < self.n and self.linked[k] and self.u[k].kind == _NUM
|
| 339 |
+
|
| 340 |
+
def _run_end(self, i: int) -> int:
|
| 341 |
+
"""End of the maximal run of number words and "va" starting at i."""
|
| 342 |
+
m = i + 1
|
| 343 |
+
while m < self.n and self.linked[m] and self.u[m].kind in (_NUM, _VA):
|
| 344 |
+
m += 1
|
| 345 |
+
return m
|
| 346 |
+
|
| 347 |
+
def _ordinal_end(self, i: int) -> int | None:
|
| 348 |
+
"""If the run at i ends "... و <ordinal>", the index after the ordinal."""
|
| 349 |
+
m = self._run_end(i)
|
| 350 |
+
if (m < self.n and self.linked[m] and self.u[m].kind == _WORD and _is_ordinal(self.u[m].norm)
|
| 351 |
+
and (self.u[m - 1].kind == _VA or self.u[m - 1].glued)):
|
| 352 |
+
return m + 1
|
| 353 |
+
return None
|
| 354 |
+
|
| 355 |
+
def _items(self, i: int, j: int) -> list:
|
| 356 |
+
items: list = []
|
| 357 |
+
for k in range(i, j):
|
| 358 |
+
un = self.u[k]
|
| 359 |
+
if un.kind == _VA:
|
| 360 |
+
items.append(_AND_ITEM)
|
| 361 |
+
else:
|
| 362 |
+
items.append((un.value, un.cls))
|
| 363 |
+
if un.glued:
|
| 364 |
+
items.append(_AND_ITEM)
|
| 365 |
+
return items
|
| 366 |
+
|
| 367 |
+
def _longest(self, i: int) -> tuple[int, int] | None:
|
| 368 |
+
"""(end, value) of the longest grammatical cardinal starting at unit i."""
|
| 369 |
+
if i >= self.n or self.u[i].kind != _NUM:
|
| 370 |
+
return None
|
| 371 |
+
for j in range(self._run_end(i), i, -1):
|
| 372 |
+
if self.u[j - 1].kind == _VA:
|
| 373 |
+
continue
|
| 374 |
+
v = _parse(self._items(i, j))
|
| 375 |
+
if v is not None:
|
| 376 |
+
if j == i + 1 and self.u[i].norm == _YE_COLLOQ:
|
| 377 |
+
return None
|
| 378 |
+
return j, v
|
| 379 |
+
return None
|
| 380 |
+
|
| 381 |
+
def _digit_run(self, i: int) -> int:
|
| 382 |
+
k = i
|
| 383 |
+
while (k < self.n and (k == i or self.linked[k]) and self.u[k].kind == _NUM
|
| 384 |
+
and self.u[k].cls in (_Z, _O) and self.u[k].norm != _YE_COLLOQ):
|
| 385 |
+
k += 1
|
| 386 |
+
return k - i
|
| 387 |
+
|
| 388 |
+
def _rep(self, i: int, j: int, s: str) -> None:
|
| 389 |
+
self.reps.append((self.u[i].start, self.u[j - 1].end, s))
|
| 390 |
+
|
| 391 |
+
# -- patterns ----------------------------------------------------------
|
| 392 |
+
|
| 393 |
+
def _time(self, k: int) -> tuple[int, str | None] | None:
|
| 394 |
+
"""Clock time after "ساعت"; (end, replacement-or-None-to-skip) or None."""
|
| 395 |
+
if not self._is_num(k):
|
| 396 |
+
return None
|
| 397 |
+
end = self._ordinal_end(k)
|
| 398 |
+
if end is not None:
|
| 399 |
+
return end, None
|
| 400 |
+
sp = self._longest(k)
|
| 401 |
+
if sp is None:
|
| 402 |
+
return None
|
| 403 |
+
j, h = sp
|
| 404 |
+
if self._is_word(j, (_QUARTER, _MINUTE)) and self._is_word(j + 1, (_TO,)):
|
| 405 |
+
# "یک ربع به ده": leave the whole expression as words
|
| 406 |
+
end = j + 2
|
| 407 |
+
if self._is_num(end):
|
| 408 |
+
sp2 = self._longest(end)
|
| 409 |
+
if sp2 is not None:
|
| 410 |
+
end = sp2[0]
|
| 411 |
+
return end, None
|
| 412 |
+
if h > 24 or self._is_word(j, _DURATION_WORDS):
|
| 413 |
+
return None
|
| 414 |
+
if j < self.n and self.linked[j] and self.u[j].kind == _VA:
|
| 415 |
+
if self._is_word(j + 1, (_HALF,)):
|
| 416 |
+
return j + 2, f"{h}:30"
|
| 417 |
+
if self._is_word(j + 1, (_QUARTER,)):
|
| 418 |
+
return j + 2, f"{h}:15"
|
| 419 |
+
if self._is_num(j + 1):
|
| 420 |
+
sp2 = self._longest(j + 1)
|
| 421 |
+
if sp2 is not None and self._is_word(sp2[0], (_MINUTE,)):
|
| 422 |
+
if sp2[1] > 59:
|
| 423 |
+
return None
|
| 424 |
+
return sp2[0] + 1, f"{h}:{sp2[1]:02d}"
|
| 425 |
+
return j, str(h)
|
| 426 |
+
|
| 427 |
+
def _frac(self, k: int) -> tuple[int, str] | None:
|
| 428 |
+
"""Fractional part after "ممیز"."""
|
| 429 |
+
if not self._is_num(k):
|
| 430 |
+
return None
|
| 431 |
+
r = self._digit_run(k)
|
| 432 |
+
if r >= 2 or (r == 1 and self.u[k].cls == _Z):
|
| 433 |
+
return k + r, "".join(str(self.u[x].value) for x in range(k, k + r))
|
| 434 |
+
sp = self._longest(k)
|
| 435 |
+
return None if sp is None else (sp[0], str(sp[1]))
|
| 436 |
+
|
| 437 |
+
def _plain_converts(self, i: int, sp: tuple[int, int]) -> bool:
|
| 438 |
+
return not (sp[0] == i + 1 and sp[1] < 10)
|
| 439 |
+
|
| 440 |
+
def _number(self, i: int) -> int:
|
| 441 |
+
end = self._ordinal_end(i)
|
| 442 |
+
if end is not None:
|
| 443 |
+
return end
|
| 444 |
+
r = self._digit_run(i)
|
| 445 |
+
if r >= 3 and not self._is_num(i + r):
|
| 446 |
+
self._rep(i, i + r, "".join(str(self.u[x].value) for x in range(i, i + r)))
|
| 447 |
+
return i + r
|
| 448 |
+
sp = self._longest(i)
|
| 449 |
+
if sp is None:
|
| 450 |
+
return i + 1
|
| 451 |
+
j, v = sp
|
| 452 |
+
if self._is_word(j, (_POINT,)):
|
| 453 |
+
fr = self._frac(j + 1)
|
| 454 |
+
if fr is not None:
|
| 455 |
+
self._rep(i, fr[0], f"{v}{self.sep}{fr[1]}")
|
| 456 |
+
return fr[0]
|
| 457 |
+
if self._is_word(j, (_PERCENT,)):
|
| 458 |
+
self._rep(i, j, str(v))
|
| 459 |
+
return j
|
| 460 |
+
if 1 <= v <= 31 and self._is_word(j, _MONTHS):
|
| 461 |
+
month = self.u[j].norm
|
| 462 |
+
year = self._longest(j + 1) if self._is_num(j + 1) else None
|
| 463 |
+
if month not in _AMBIGUOUS_MONTHS or (year is not None and self._plain_converts(j + 1, year)):
|
| 464 |
+
self._rep(i, j, str(v))
|
| 465 |
+
return j
|
| 466 |
+
if self._plain_converts(i, sp):
|
| 467 |
+
self._rep(i, j, str(v))
|
| 468 |
+
return j
|
| 469 |
+
|
| 470 |
+
def run(self) -> list[tuple[int, int, str]]:
|
| 471 |
+
u, n, i = self.u, self.n, 0
|
| 472 |
+
while i < n:
|
| 473 |
+
un = u[i]
|
| 474 |
+
if un.kind == _NUM:
|
| 475 |
+
i = self._number(i)
|
| 476 |
+
continue
|
| 477 |
+
if un.kind == _WORD and un.norm == _SAAT:
|
| 478 |
+
t = self._time(i + 1)
|
| 479 |
+
if t is not None:
|
| 480 |
+
end, rep = t
|
| 481 |
+
if rep is not None:
|
| 482 |
+
self._rep(i + 1, end, rep)
|
| 483 |
+
i = end
|
| 484 |
+
continue
|
| 485 |
+
elif un.kind == _DIGITS and self._is_num(i + 1) and u[i + 1].cls == _S:
|
| 486 |
+
# "۲۰ هزار": mixed digits + scale words are left alone
|
| 487 |
+
i = self._run_end(i + 1)
|
| 488 |
+
continue
|
| 489 |
+
i += 1
|
| 490 |
+
return self.reps
|
| 491 |
+
|
| 492 |
+
|
| 493 |
+
def itn(text: str, digits: str = "fa") -> str:
|
| 494 |
+
"""Rewrite spoken Persian number expressions in ``text`` as digits.
|
| 495 |
+
|
| 496 |
+
``digits="fa"`` writes Persian digits (U+06F0..U+06F9) and U+066B as the
|
| 497 |
+
decimal separator; ``digits="ascii"`` writes 0-9 and ".". Digits already
|
| 498 |
+
in the text are converted to the requested style; nothing else that is
|
| 499 |
+
not a number expression is changed.
|
| 500 |
+
"""
|
| 501 |
+
if digits not in _DECIMAL_SEP:
|
| 502 |
+
raise ValueError("digits must be 'fa' or 'ascii'")
|
| 503 |
+
if not text:
|
| 504 |
+
return text
|
| 505 |
+
reps = _Scanner(text, _DECIMAL_SEP[digits]).run()
|
| 506 |
+
out = []
|
| 507 |
+
pos = 0
|
| 508 |
+
for s, e, r in reps:
|
| 509 |
+
out.append(text[pos:s])
|
| 510 |
+
out.append(r)
|
| 511 |
+
pos = e
|
| 512 |
+
out.append(text[pos:])
|
| 513 |
+
return "".join(out).translate(_TO_FA if digits == "fa" else _TO_ASCII)
|
nimruz/ortho.py
ADDED
|
@@ -0,0 +1,461 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Persian word-spacing standardiser for speech-recognition training targets.
|
| 2 |
+
|
| 3 |
+
Transcripts from different sources write the same word three ways: with a
|
| 4 |
+
ZWNJ (``\\u200c``, nim-fasele -- the standard), with a space, or joined.
|
| 5 |
+
``SpacingFixer`` rewrites target text to ONE convention, the standard ZWNJ
|
| 6 |
+
forms, for a small set of high-frequency morphemes:
|
| 7 |
+
|
| 8 |
+
* ``verb_prefix_spaced`` ``mi X`` / ``nemi X`` -> ``mi<ZWNJ>X``
|
| 9 |
+
* ``verb_prefix_joined`` ``miX`` / ``nemiX`` -> ``mi<ZWNJ>X``
|
| 10 |
+
* ``plural_spaced`` ``X ha(y..)`` -> ``X<ZWNJ>ha(y..)``
|
| 11 |
+
* ``plural_joined`` ``Xha(y..)`` -> ``X<ZWNJ>ha(y..)`` (lexicon only)
|
| 12 |
+
* ``comparative_spaced`` ``X tar(in)`` -> ``X<ZWNJ>tar(in)`` (lexicon only)
|
| 13 |
+
|
| 14 |
+
Precision matters far more than recall: a wrong rewrite corrupts a training
|
| 15 |
+
label, a missed one leaves a harmless variant. Every rule therefore needs
|
| 16 |
+
positive evidence (a built-in verb grammar, or attestation in a lexicon built
|
| 17 |
+
from the corpus) and is blocked by curated exception lists.
|
| 18 |
+
|
| 19 |
+
ZWNJ after non-joining letters: the only source that uses ZWNJ (YouTube
|
| 20 |
+
subtitles) almost never writes it after a letter that does not join forward
|
| 21 |
+
(alef, dal, zal, reh, zain, jeh, waw ...): e.g. before "ha" 1-2 ZWNJ forms vs
|
| 22 |
+
64-276 plain joins per letter. The two spellings render identically, so the
|
| 23 |
+
convention here is: ZWNJ after joining letters, plain join after non-joining
|
| 24 |
+
letters (``kshvr-ha`` -> joined, ``ketab-ha`` -> ZWNJ). The prefix "mi" ends
|
| 25 |
+
in yeh (joining), so it always takes a ZWNJ.
|
| 26 |
+
|
| 27 |
+
Input is expected to be the output of
|
| 28 |
+
``textnorm.normalize_for_training(text, keep_punct=False)``. Standard
|
| 29 |
+
library only; all tables and regexes are built at import time (no I/O).
|
| 30 |
+
Every yeh is PERSIAN YEH U+06CC and every kaf is KEHEH U+06A9.
|
| 31 |
+
"""
|
| 32 |
+
|
| 33 |
+
from __future__ import annotations
|
| 34 |
+
|
| 35 |
+
import json
|
| 36 |
+
import re
|
| 37 |
+
from collections import Counter
|
| 38 |
+
|
| 39 |
+
__all__ = ["build_lexicon", "save_lexicon", "load_lexicon", "SpacingFixer"]
|
| 40 |
+
|
| 41 |
+
ZWNJ = "" # ZERO WIDTH NON-JOINER (nim-fasele)
|
| 42 |
+
|
| 43 |
+
# Letters that never join to the following letter: a ZWNJ after them is
|
| 44 |
+
# invisible, and the convention here is to write a plain join instead.
|
| 45 |
+
_NON_JOINING = frozenset(
|
| 46 |
+
"ا" # ALEF
|
| 47 |
+
"آ" # ALEF WITH MADDA
|
| 48 |
+
"أ" # ALEF WITH HAMZA ABOVE
|
| 49 |
+
"إ" # ALEF WITH HAMZA BELOW
|
| 50 |
+
"ٱ" # ALEF WASLA
|
| 51 |
+
"د" # DAL
|
| 52 |
+
"ذ" # THAL
|
| 53 |
+
"ر" # REH
|
| 54 |
+
"ز" # ZAIN
|
| 55 |
+
"ژ" # JEH
|
| 56 |
+
"و" # WAW
|
| 57 |
+
"ؤ" # WAW WITH HAMZA
|
| 58 |
+
"ة" # TEH MARBUTA
|
| 59 |
+
"ء" # HAMZA
|
| 60 |
+
)
|
| 61 |
+
|
| 62 |
+
# A token the rules may touch: Arabic-script letters only (no digits, Latin, ZWNJ).
|
| 63 |
+
_FA_WORD_RE = re.compile("[ء-غف-يپچژکگی]+")
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def _glue(left: str, right: str) -> str:
|
| 67 |
+
"""Join two pieces with ZWNJ, or plainly after a non-joining letter."""
|
| 68 |
+
return left + right if left[-1] in _NON_JOINING else left + ZWNJ + right
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def _words(s: str) -> frozenset:
|
| 72 |
+
return frozenset(s.split())
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
# ---------------------------------------------------------------------------
|
| 76 |
+
# Verb grammar (rules 1 and 2)
|
| 77 |
+
# ---------------------------------------------------------------------------
|
| 78 |
+
|
| 79 |
+
_MI = "می"
|
| 80 |
+
_NEMI = "نمی"
|
| 81 |
+
|
| 82 |
+
# Present stems, formal and colloquial (single-letter colloquial stems are
|
| 83 |
+
# handled separately with a restricted set of endings).
|
| 84 |
+
_PRESENT_STEMS = _words("""
|
| 85 |
+
کن شو گو گوی دون دان تون توان خوا خواه دار بین ده گیر زن خور ذار زار
|
| 86 |
+
گذار مون مان رس کش ساز خون خوان نویس فهم شناس بر پرس ترس خند
|
| 87 |
+
افت فت وفت ایست وایس ریز پذیر یاب بخش دوز سوز فروش خر پر چرخ جنگ کوش گرد گذر
|
| 88 |
+
گذر شور نشین شین خواب پوش چسب باش فرست بند بار پیچ لرز شکن شکون کوب رقص دو
|
| 89 |
+
چین شنو کار کاو خیز رو رون رسون رسان گذرون گذران ترسون ترسان چرخون چرخان
|
| 90 |
+
پوشون پوشان خوابون خوابان سوزون سوزان کشون شکاف مال پاش جوش درخش تاب تپ چک
|
| 91 |
+
پز پسند پرست پرداز پوس چپون انداز نداز اندیش آموز آمیز آرا گشا نما نمای نام
|
| 92 |
+
شمار شمر جو جوی خار لیس مک بوس سنج طلب کاه افزا افزای رنج رنجان لغز نگر
|
| 93 |
+
نگار جنب جنبون گز تراش ران پرور فشار فشر ارز میر بال شکف لرزون دزد بخشون
|
| 94 |
+
بر دوون پرون پیوند گوش چرخ خواهی ترکون ترک سپار سپر خرا کوچ شمر گرای
|
| 95 |
+
""")
|
| 96 |
+
# Past stems (also take the empty ending: 3rd person singular).
|
| 97 |
+
_PAST_STEMS = _words("""
|
| 98 |
+
کرد شد رفت گفت دونست دانست تونست توانست خواست داشت دید اومد داد گرفت
|
| 99 |
+
زد خورد ذاشت زاشت گذاشت موند ماند رسید کشید کشت ساخت خوند خواند نوشت فهمید
|
| 100 |
+
شناخت برد پرسید ترسید خندید افتاد فتاد وفتاد ایستاد وایساد ریخت
|
| 101 |
+
پذیرفت یافت بخشید دوخت سوخت فروخت خرید پرید چرخید جنگید کوشید گشت گذشت شست
|
| 102 |
+
نشست خوابید پوشید چسبید فرستاد بست بارید پیچید لرزید شکست کوبید رقصید دوید
|
| 103 |
+
چید شنید کاشت رسوند رساند گذروند گذراند انداخت نداخت اندیشید آموخت پنداشت
|
| 104 |
+
مرد زیست باخت پخت سوزوند سوزاند تابید تپید چکید درخشید جوشید مالید پاشید
|
| 105 |
+
جنبید لغزید نامید شمرد بالید خارید نمود کاست افزود طلبید رنجاند راند پرستید
|
| 106 |
+
پرداخت پسندید نگریست گریست گریخت شکافت کند جست بوسید لیسید مکید پوسید گزید
|
| 107 |
+
چرخوند ترسوند خوابوند بود شکوند پوشوند کشوند دزدید ارزید آمیخت گذروند رسید
|
| 108 |
+
""")
|
| 109 |
+
# Single-letter colloquial stems: ش (shodan) ر (raftan) گ (goftan) د (dadan)
|
| 110 |
+
# and آ/ا (amadan). Restricted endings keep nouns such as میش، میر، مید out.
|
| 111 |
+
_SHORT_STEMS_CLITIC = "گد" # may take an object clitic: میگمش، میدمت
|
| 112 |
+
_SHORT_STEMS_PLAIN = "شر"
|
| 113 |
+
|
| 114 |
+
_END_PRES = "م ی د ه یم ید ند ن ین یی ییم یید یند"
|
| 115 |
+
_END_PAST = "م ی یم ید ند ین ن ه"
|
| 116 |
+
_END_SHORT = "م ی ه یم ید ین ن"
|
| 117 |
+
_END_COME = "م ی د یم ید ین"
|
| 118 |
+
_CLITIC = "م ت ش مون تون شون"
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def _alt(words) -> str:
|
| 122 |
+
return "|".join(sorted((re.escape(w) for w in words), key=len, reverse=True))
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
_VERB_BODY_RE = re.compile(
|
| 126 |
+
"(?:"
|
| 127 |
+
f"(?:{_alt(_PRESENT_STEMS)})(?:(?:{_alt(_END_PRES.split())})(?:{_alt(_CLITIC.split())})?"
|
| 128 |
+
f"|ت(?:{_alt(_CLITIC.split())}))"
|
| 129 |
+
f"|(?:{_alt(_PAST_STEMS)})(?:{_alt(_END_PAST.split())})?(?:{_alt(_CLITIC.split())})?"
|
| 130 |
+
f"|[{_SHORT_STEMS_CLITIC}](?:{_alt(_END_SHORT.split())})(?:{_alt(_CLITIC.split())})?"
|
| 131 |
+
f"|[{_SHORT_STEMS_PLAIN}](?:{_alt(_END_SHORT.split())})"
|
| 132 |
+
# impersonal / modal forms with no personal ending
|
| 133 |
+
"|توان|بایست|بایستی|باید"
|
| 134 |
+
")"
|
| 135 |
+
)
|
| 136 |
+
|
| 137 |
+
# amadan / avardan: always spelled with ALEF WITH MADDA after the prefix
|
| 138 |
+
# (YouTube: "می<ZWNJ>آد" 30 times, "می<ZWNJ>اد" never), so a bare-alef body
|
| 139 |
+
# such as colloquial "میاد" is rewritten to "می<ZWNJ>آد".
|
| 140 |
+
_COME_BODY_RE = re.compile(
|
| 141 |
+
"\u0622(?:" # ALEF WITH MADDA
|
| 142 |
+
f"(?:{_alt(_END_COME.split())})"
|
| 143 |
+
f"|(?:ی|ر|ور)(?:{_alt(_END_PRES.split())})(?:{_alt(_CLITIC.split())})?"
|
| 144 |
+
f"|(?:ورد|مد)(?:{_alt(_END_PAST.split())})?(?:{_alt(_CLITIC.split())})?"
|
| 145 |
+
")"
|
| 146 |
+
)
|
| 147 |
+
|
| 148 |
+
_PAST_TAIL_RE = re.compile(f"(?:{_alt(_PAST_STEMS)})$")
|
| 149 |
+
|
| 150 |
+
# Real words that begin with می / نمی and must never be split.
|
| 151 |
+
_MI_EXCEPTIONS = _words("""
|
| 152 |
+
می نمی میز میزان میزانی میان میانه میانی میانگین میانجی میانمار میدان میدون
|
| 153 |
+
میدانی میهن میهنی میهمان میهمانی میهمانان میوه میل میله میلاد میلادی میلیون
|
| 154 |
+
میلیونی میلیونر میلیونها میلیارد میلیاردی میلیاردها میلیمتر میکروب میکروفون
|
| 155 |
+
میکروفن میکروسکوپ میکرو میمون میمونها میخ میخک میخکوب میراث میرزا میرزای میش
|
| 156 |
+
میگو مینا مینی میت میسر میثاق میکاپ میکس میس میترا میر میرا میزبان میزبانی
|
| 157 |
+
میخانه میکده میگرن میگون میخواره میخوارگی میگسار میهنپرست مینو مینیاتور
|
| 158 |
+
میشل میلان میامی میا مید مین میم میه میسیون میسون میلر میلاس مینسک میرویس
|
| 159 |
+
میلتون میکائیل میکل میکی میگل میناب میبد میاندوآب میانرو میانداری میدانداری
|
| 160 |
+
میزگرد میزها میزی میزش میزو میوهها میوههای میوهای مینیبوس مینیمال میراثش
|
| 161 |
+
میلیاردش میلیاردو میانبر میانسال میانجیگری میقات میعاد میمنت میمیک میکرون
|
| 162 |
+
میلگرد میمیرم نمک نمکی نمایش نمایان نماز نمونه نمره نمد نمود نمودار نمناک
|
| 163 |
+
نمایشگاه نماینده نمایندگی نمین نمیر
|
| 164 |
+
""")
|
| 165 |
+
|
| 166 |
+
# Never join "mi" to these (function words, pronouns, interjections).
|
| 167 |
+
_MI_STOP_NEXT = _words("""
|
| 168 |
+
و که را رو به از در تا با هم یا اگر اگه ولی اما من تو او اون ما شما اونا آنها
|
| 169 |
+
این آن یه یک چی چه کی نه خب هی بله برای واسه هر همه
|
| 170 |
+
""")
|
| 171 |
+
|
| 172 |
+
_NUMBER_WORDS = _words("""
|
| 173 |
+
صفر یک یه دو سه چهار پنج شش هفت هشت نه ده یازده دوازده سیزده چهارده پانزده
|
| 174 |
+
پونزده شانزده شونزده هفده هجده هیجده نوزده بیست سی چهل پنجاه شصت هفتاد هشتاد
|
| 175 |
+
نود صد دویست سیصد چهارصد پانصد پو��صد ششصد هفتصد هشتصد نهصد هزار میلیون میلیارد
|
| 176 |
+
اول دوم سوم
|
| 177 |
+
""")
|
| 178 |
+
_DIGIT_RE = re.compile("[0-9٠-٩۰-۹]")
|
| 179 |
+
|
| 180 |
+
# ---------------------------------------------------------------------------
|
| 181 |
+
# Plural suffix (rules 3 and 4)
|
| 182 |
+
# ---------------------------------------------------------------------------
|
| 183 |
+
|
| 184 |
+
_PLURAL_SUFFIXES = (
|
| 185 |
+
"ها", "های", "هایی", "هایم", "هایت", "هایش", "هایمان", "هایتان", "هایشان",
|
| 186 |
+
"هام", "هات", "هاش", "هامون", "هاتون", "هاشون", "هارو",
|
| 187 |
+
)
|
| 188 |
+
_PLURAL_SET = frozenset(_PLURAL_SUFFIXES)
|
| 189 |
+
# Xهای / Xهام ... can also be Xه + ای / ام ... (indefinite, clitic, perfect).
|
| 190 |
+
_HEH_CLITIC_SUFFIXES = frozenset(
|
| 191 |
+
("های", "هام", "هات", "هاش", "هایم", "هاید", "هامون", "هاتون", "هاشون")
|
| 192 |
+
)
|
| 193 |
+
_AMBIGUOUS_SPACED = frozenset(("ها", "هام", "هات", "هاش"))
|
| 194 |
+
# Longest first so "Xهایشان" is split as X + هایشان.
|
| 195 |
+
_PLURAL_SPLIT_RE = re.compile(f"^(.{{2,}}?)({_alt(_PLURAL_SUFFIXES)})$")
|
| 196 |
+
|
| 197 |
+
# Words that cannot be the stem of a spaced plural ("آره ها" = interjection).
|
| 198 |
+
_PLURAL_STOP_LEFT = _words("""
|
| 199 |
+
آره نه بله خب ها وای آها آهان اوه چی کی که و یا تا با از به در را رو هم اگه اگر
|
| 200 |
+
ولی اما من تو او ما شما اینه اونه بیا برو بگو ببین باشه نباشه هست نیست داره
|
| 201 |
+
دارم داری شه میشه نمیشه بشه کرده شده بود بوده دیگه حالا البته واقعا اصلا
|
| 202 |
+
خوبه بده آخه پس چرا یعنی مگه دیگر چون وقتی ببخشید بفرمایید بالا پایین کن
|
| 203 |
+
بکن نکن بره نره بیا نیا بشه برم بریم واسه برای مثل مثه مثلا اینجا اونجا
|
| 204 |
+
کجا پیش جلو عقب زیر بیرون تو توی
|
| 205 |
+
""")
|
| 206 |
+
|
| 207 |
+
# Real words that merely end in a plural-suffix string (rule 4).
|
| 208 |
+
_PLURAL_EXCEPTIONS = _words("""
|
| 209 |
+
تنها تنهای تنهایی تنهاییم تنهاییت تنهاییش بها بهای بهایی رها رهایی نها نهایی
|
| 210 |
+
نهایت بینهایت نهایتا انتها انتهای ابتدا منها منهای الها الهام اشتها اشتهای
|
| 211 |
+
ابهام اتهام اوهام ایهام سهام مهام استفهام شبهات جهات توجهات امهات دهات کراهات
|
| 212 |
+
مباهات شبهای تشبیهات تنبیهات نزاهت سفاهت کراهت فقاهت وجاهت بداهت سراهام
|
| 213 |
+
گواها آگاها شاها پادشاها الهی الهایم سپاها منتها منتهای منتهایش
|
| 214 |
+
""")
|
| 215 |
+
|
| 216 |
+
# ---------------------------------------------------------------------------
|
| 217 |
+
# Comparatives (rule 5)
|
| 218 |
+
# ---------------------------------------------------------------------------
|
| 219 |
+
|
| 220 |
+
_COMPARATIVE_SUFFIXES = ("تر", "ترین")
|
| 221 |
+
|
| 222 |
+
|
| 223 |
+
# ---------------------------------------------------------------------------
|
| 224 |
+
# Lexicon
|
| 225 |
+
# ---------------------------------------------------------------------------
|
| 226 |
+
|
| 227 |
+
|
| 228 |
+
def build_lexicon(texts) -> dict[str, int]:
|
| 229 |
+
"""Token counts over an iterable of lines (whitespace tokens; ZWNJ stays inside)."""
|
| 230 |
+
c: Counter = Counter()
|
| 231 |
+
for line in texts:
|
| 232 |
+
c.update(line.split())
|
| 233 |
+
return dict(c)
|
| 234 |
+
|
| 235 |
+
|
| 236 |
+
def save_lexicon(lexicon, path) -> None:
|
| 237 |
+
with open(path, "w", encoding="utf-8") as f:
|
| 238 |
+
json.dump(dict(lexicon), f, ensure_ascii=False, sort_keys=True)
|
| 239 |
+
|
| 240 |
+
|
| 241 |
+
def load_lexicon(path) -> dict[str, int]:
|
| 242 |
+
with open(path, encoding="utf-8") as f:
|
| 243 |
+
return {str(k): int(v) for k, v in json.load(f).items()}
|
| 244 |
+
|
| 245 |
+
|
| 246 |
+
# ---------------------------------------------------------------------------
|
| 247 |
+
# Fixer
|
| 248 |
+
# ---------------------------------------------------------------------------
|
| 249 |
+
|
| 250 |
+
_MIN_STEM_COUNT = 3 # rule 4: X must occur standalone at least this often
|
| 251 |
+
_MIN_HEH_COUNT = 2 # rule 4: stem+heh attested this often makes Xهای ambiguous
|
| 252 |
+
_MIN_ZWNJ_EVIDENCE = 2 # rules 1/2: "mi<ZWNJ>X" attested at least this often
|
| 253 |
+
|
| 254 |
+
|
| 255 |
+
class SpacingFixer:
|
| 256 |
+
"""Rewrite Persian text to the standard ZWNJ spacing convention.
|
| 257 |
+
|
| 258 |
+
``lexicon`` (token -> count, see ``build_lexicon``) enables the
|
| 259 |
+
evidence-based rules; with ``None`` only the grammar/list based rules
|
| 260 |
+
(1, 2, 3) apply, rule 3 with stricter guards. Decisions are cached per token / token pair, so the
|
| 261 |
+
fixer is cheap to call on every sample in a data loader.
|
| 262 |
+
"""
|
| 263 |
+
|
| 264 |
+
def __init__(self, lexicon: dict[str, int] | None = None):
|
| 265 |
+
self.lexicon = lexicon
|
| 266 |
+
self._tok_cache: dict[str, tuple[str, str] | None] = {}
|
| 267 |
+
self._pair_cache: dict[tuple[str, str, bool], tuple[str, str] | None] = {}
|
| 268 |
+
|
| 269 |
+
# -- evidence ------------------------------------------------------------
|
| 270 |
+
|
| 271 |
+
def _count(self, w: str) -> int:
|
| 272 |
+
return self.lexicon.get(w, 0) if self.lexicon is not None else 0
|
| 273 |
+
|
| 274 |
+
def _verb_body(self, prefix: str, body: str) -> str | None:
|
| 275 |
+
"""Standard spelling of ``body`` (the part after می/نمی) if it is a
|
| 276 |
+
known verb form, else ``None``."""
|
| 277 |
+
if len(body) < 2 or not _FA_WORD_RE.fullmatch(body):
|
| 278 |
+
return None
|
| 279 |
+
# "ده" is far more often the verb (می ده = gives) than the number
|
| 280 |
+
# after "mi"; the month "May" is caught by the context guard instead.
|
| 281 |
+
if body in _MI_STOP_NEXT or (body in _NUMBER_WORDS and body != "ده"):
|
| 282 |
+
return None
|
| 283 |
+
if prefix + body in _MI_EXCEPTIONS:
|
| 284 |
+
return None
|
| 285 |
+
if body[0] in "اآ": # ALEF, ALEF WITH MADDA
|
| 286 |
+
madda = "آ" + body[1:]
|
| 287 |
+
if _COME_BODY_RE.fullmatch(madda):
|
| 288 |
+
return madda
|
| 289 |
+
if _VERB_BODY_RE.fullmatch(body):
|
| 290 |
+
return body
|
| 291 |
+
if self._count(prefix + ZWNJ + body) >= _MIN_ZWNJ_EVIDENCE:
|
| 292 |
+
return body
|
| 293 |
+
return None
|
| 294 |
+
|
| 295 |
+
def _is_verbish(self, tok: str) -> bool:
|
| 296 |
+
"""A verb with the mi/nemi prefix (joined or ZWNJ)."""
|
| 297 |
+
for p in (_NEMI, _MI):
|
| 298 |
+
if tok.startswith(p):
|
| 299 |
+
body = tok[len(p):]
|
| 300 |
+
if body.startswith(ZWNJ):
|
| 301 |
+
return True
|
| 302 |
+
return self._verb_body(p, body) is not None
|
| 303 |
+
return False
|
| 304 |
+
|
| 305 |
+
def _plural_attested(self, stem: str) -> bool:
|
| 306 |
+
for suf in ("ها", "های", "هایی"):
|
| 307 |
+
if self._count(stem + ZWNJ + suf) or self._count(stem + suf):
|
| 308 |
+
return True
|
| 309 |
+
return False
|
| 310 |
+
|
| 311 |
+
# -- per-token rules -------------------------------------------------------
|
| 312 |
+
|
| 313 |
+
def _fix_token(self, tok: str) -> tuple[str, str] | None:
|
| 314 |
+
try:
|
| 315 |
+
return self._tok_cache[tok]
|
| 316 |
+
except KeyError:
|
| 317 |
+
pass
|
| 318 |
+
res = None
|
| 319 |
+
if _FA_WORD_RE.fullmatch(tok):
|
| 320 |
+
res = self._verb_joined(tok) or self._plural_joined(tok)
|
| 321 |
+
self._tok_cache[tok] = res
|
| 322 |
+
return res
|
| 323 |
+
|
| 324 |
+
def _verb_joined(self, tok: str) -> tuple[str, str] | None:
|
| 325 |
+
for p in (_NEMI, _MI):
|
| 326 |
+
if tok.startswith(p):
|
| 327 |
+
if tok in _MI_EXCEPTIONS:
|
| 328 |
+
return None
|
| 329 |
+
body = self._verb_body(p, tok[len(p):])
|
| 330 |
+
if body is not None:
|
| 331 |
+
return p + ZWNJ + body, "verb_prefix_joined"
|
| 332 |
+
return None
|
| 333 |
+
return None
|
| 334 |
+
|
| 335 |
+
def _plural_joined(self, tok: str) -> tuple[str, str] | None:
|
| 336 |
+
if self.lexicon is None or tok in _PLURAL_EXCEPTIONS:
|
| 337 |
+
return None
|
| 338 |
+
m = _PLURAL_SPLIT_RE.match(tok)
|
| 339 |
+
if not m:
|
| 340 |
+
return None
|
| 341 |
+
stem, suf = m.groups()
|
| 342 |
+
# After a non-joining letter the plain join already is the standard.
|
| 343 |
+
if stem[-1] in _NON_JOINING or stem[-1] == ZWNJ:
|
| 344 |
+
return None
|
| 345 |
+
if stem in _PLURAL_STOP_LEFT or self._is_verbish(stem):
|
| 346 |
+
return None
|
| 347 |
+
if self._count(stem) < _MIN_STEM_COUNT:
|
| 348 |
+
return None
|
| 349 |
+
# "خانهای" is خانه+ای (a house), not خان+های; "خانهام" is خانه+ام.
|
| 350 |
+
# For suffixes that can also be read as heh + clitic, skip whenever
|
| 351 |
+
# stem+heh is an attested word. For the others only a typo such as
|
| 352 |
+
# "بچها" (for بچهها) is possible: skip if stem+heh is the commoner.
|
| 353 |
+
heh = self._count(stem + "\u0647") # HEH
|
| 354 |
+
if heh >= (_MIN_HEH_COUNT if suf in _HEH_CLITIC_SUFFIXES else max(1, self._count(stem))):
|
| 355 |
+
return None
|
| 356 |
+
# The heh+clitic reading is also possible for words the corpus never
|
| 357 |
+
# shows bare ("بیشهای" = بیشهای): demand that the plain plural
|
| 358 |
+
# "Xها" or the ZWNJ spelling of this very token is attested.
|
| 359 |
+
# (Impossible after a final heh or yeh: "Xهه" / "Xیه" are not words.)
|
| 360 |
+
if suf in _HEH_CLITIC_SUFFIXES and stem[-1] not in "\u0647\u06cc" and not (
|
| 361 |
+
self._count(stem + "ها") or self._count(stem + ZWNJ + "ها")
|
| 362 |
+
or self._count(stem + ZWNJ + suf)
|
| 363 |
+
):
|
| 364 |
+
return None
|
| 365 |
+
# "پرداختهام" is a perfect tense (پرداخته+ام), not پرداخت+هام.
|
| 366 |
+
if suf != "ها" and _PAST_TAIL_RE.search(stem):
|
| 367 |
+
return None
|
| 368 |
+
return stem + ZWNJ + suf, "plural_joined"
|
| 369 |
+
|
| 370 |
+
# -- pair rules ------------------------------------------------------------
|
| 371 |
+
|
| 372 |
+
def _merge(self, prev2: str | None, left: str, right: str, last: bool):
|
| 373 |
+
"""Decide whether ``left right`` should become one token."""
|
| 374 |
+
key = (left, right, last if right == "ها" else False)
|
| 375 |
+
if left in (_MI, _NEMI):
|
| 376 |
+
# Context-dependent guards (month "May" after a number) first.
|
| 377 |
+
if prev2 is not None and (prev2 in _NUMBER_WORDS or _DIGIT_RE.search(prev2)):
|
| 378 |
+
return None
|
| 379 |
+
try:
|
| 380 |
+
return self._pair_cache[key]
|
| 381 |
+
except KeyError:
|
| 382 |
+
pass
|
| 383 |
+
res = self._merge_uncached(left, right, key[2])
|
| 384 |
+
self._pair_cache[key] = res
|
| 385 |
+
return res
|
| 386 |
+
|
| 387 |
+
def _merge_uncached(self, left: str, right: str, last: bool):
|
| 388 |
+
if left in (_MI, _NEMI):
|
| 389 |
+
body = self._verb_body(left, right)
|
| 390 |
+
if body is not None:
|
| 391 |
+
return left + ZWNJ + body, "verb_prefix_spaced"
|
| 392 |
+
return None
|
| 393 |
+
if right in _PLURAL_SET:
|
| 394 |
+
return self._plural_spaced(left, right, last)
|
| 395 |
+
if right in _COMPARATIVE_SUFFIXES:
|
| 396 |
+
return self._comparative_spaced(left, right)
|
| 397 |
+
return None
|
| 398 |
+
|
| 399 |
+
def _plural_spaced(self, left: str, suf: str, last: bool):
|
| 400 |
+
if len(left) < 2 or not _FA_WORD_RE.fullmatch(left.replace(ZWNJ, "")):
|
| 401 |
+
return None
|
| 402 |
+
if left[-1] == ZWNJ or left in _PLURAL_STOP_LEFT or self._is_verbish(left):
|
| 403 |
+
return None
|
| 404 |
+
if left.endswith(_PLURAL_SUFFIXES):
|
| 405 |
+
return None # already plural (keeps the rewrite idempotent)
|
| 406 |
+
if suf in _AMBIGUOUS_SPACED:
|
| 407 |
+
# Bare "ha" is also a colloquial interjection ("خوبه ها"), and the
|
| 408 |
+
# short possessives are easily mis-split ("واسه هات"). With a
|
| 409 |
+
# lexicon, demand that some plural of X is attested; without one,
|
| 410 |
+
# never merge a line-final "ha".
|
| 411 |
+
if self.lexicon is not None:
|
| 412 |
+
if not self._plural_attested(left):
|
| 413 |
+
return None
|
| 414 |
+
elif suf == "ها" and last:
|
| 415 |
+
return None
|
| 416 |
+
return _glue(left, suf), "plural_spaced"
|
| 417 |
+
|
| 418 |
+
def _comparative_spaced(self, left: str, suf: str):
|
| 419 |
+
if self.lexicon is None or not _FA_WORD_RE.fullmatch(left):
|
| 420 |
+
return None
|
| 421 |
+
if len(left) < 2 or left.endswith("تر") or self._count(left) < _MIN_STEM_COUNT:
|
| 422 |
+
return None
|
| 423 |
+
zw = self._count(left + ZWNJ + "تر") + self._count(left + ZWNJ + "ترین")
|
| 424 |
+
joined_both = self._count(left + "تر") and self._count(left + "ترین")
|
| 425 |
+
if not (zw or joined_both):
|
| 426 |
+
return None
|
| 427 |
+
return _glue(left, suf), "comparative_spaced"
|
| 428 |
+
|
| 429 |
+
# -- public ----------------------------------------------------------------
|
| 430 |
+
|
| 431 |
+
def _run(self, text: str, log: list | None) -> str:
|
| 432 |
+
toks = text.split()
|
| 433 |
+
if not toks:
|
| 434 |
+
return text
|
| 435 |
+
out: list[str] = []
|
| 436 |
+
for i, tok in enumerate(toks):
|
| 437 |
+
if out:
|
| 438 |
+
prev2 = out[-2] if len(out) >= 2 else None
|
| 439 |
+
res = self._merge(prev2, out[-1], tok, i == len(toks) - 1)
|
| 440 |
+
if res is not None:
|
| 441 |
+
if log is not None:
|
| 442 |
+
log.append((f"{out[-1]} {tok}", res[0], res[1]))
|
| 443 |
+
out[-1] = res[0]
|
| 444 |
+
continue
|
| 445 |
+
out.append(tok)
|
| 446 |
+
for i, tok in enumerate(out):
|
| 447 |
+
res = self._fix_token(tok)
|
| 448 |
+
if res is not None:
|
| 449 |
+
if log is not None:
|
| 450 |
+
log.append((tok, res[0], res[1]))
|
| 451 |
+
out[i] = res[0]
|
| 452 |
+
return " ".join(out)
|
| 453 |
+
|
| 454 |
+
def __call__(self, text: str) -> str:
|
| 455 |
+
return self._run(text, None)
|
| 456 |
+
|
| 457 |
+
def explain(self, text: str) -> list[tuple[str, str, str]]:
|
| 458 |
+
"""``(before, after, rule_name)`` for every rewrite ``__call__`` makes."""
|
| 459 |
+
log: list[tuple[str, str, str]] = []
|
| 460 |
+
self._run(text, log)
|
| 461 |
+
return log
|
nimruz/scoring.py
ADDED
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Scoring text: one canonical spelling for reference and hypothesis.
|
| 2 |
+
|
| 3 |
+
Persian sources disagree on word spacing ("میکنم" / "می کنم" / "میکنم", "آنها" / "آنها").
|
| 4 |
+
Scoring those as errors measures the convention, not the recognition, so both sides are
|
| 5 |
+
first rewritten to the standard forms with the same rules used on the training text,
|
| 6 |
+
then folded by `normalize_for_scoring`.
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
from __future__ import annotations
|
| 10 |
+
|
| 11 |
+
import os
|
| 12 |
+
from functools import lru_cache
|
| 13 |
+
from pathlib import Path
|
| 14 |
+
|
| 15 |
+
from nimruz.ortho import SpacingFixer, load_lexicon
|
| 16 |
+
from nimruz.textnorm import normalize_for_scoring, normalize_for_training
|
| 17 |
+
|
| 18 |
+
LEXICON_PATHS = (
|
| 19 |
+
os.environ.get("NIMRUZ_LEXICON", ""),
|
| 20 |
+
str(Path(__file__).resolve().parent.parent / "assets" / "lexicon.json"),
|
| 21 |
+
"/data/train/lexicon.json",
|
| 22 |
+
)
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
@lru_cache(maxsize=1)
|
| 26 |
+
def fixer() -> SpacingFixer:
|
| 27 |
+
for p in LEXICON_PATHS:
|
| 28 |
+
if p and Path(p).exists():
|
| 29 |
+
return SpacingFixer(load_lexicon(p))
|
| 30 |
+
return SpacingFixer(None)
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def canonical(text: str) -> str:
|
| 34 |
+
"""Canonical scoring form: standard spacing, folded letters, spelled-out numbers, no punctuation."""
|
| 35 |
+
return normalize_for_scoring(fixer()(normalize_for_training(text or "", keep_punct=False)))
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
def error_rates(refs: list[str], hyps: list[str]) -> dict:
|
| 39 |
+
"""WER / CER on canonical text, CER ignoring spaces, and WER without the spacing rules."""
|
| 40 |
+
import jiwer
|
| 41 |
+
|
| 42 |
+
pairs = [(canonical(r), canonical(h), normalize_for_scoring(r), normalize_for_scoring(h)) for r, h in zip(refs, hyps)]
|
| 43 |
+
pairs = [p for p in pairs if p[0]]
|
| 44 |
+
r, h = [p[0] for p in pairs], [p[1] for p in pairs]
|
| 45 |
+
return {
|
| 46 |
+
"n": len(pairs),
|
| 47 |
+
"wer": 100 * jiwer.wer(r, h),
|
| 48 |
+
"cer": 100 * jiwer.cer(r, h),
|
| 49 |
+
"ncer": 100 * jiwer.cer([x.replace(" ", "") for x in r], [x.replace(" ", "") or "-" for x in h]),
|
| 50 |
+
"wer_raw": 100 * jiwer.wer([p[2] for p in pairs], [p[3] for p in pairs]),
|
| 51 |
+
}
|
nimruz/textnorm.py
ADDED
|
@@ -0,0 +1,465 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Persian (Farsi) text normalisation for speech recognition.
|
| 2 |
+
|
| 3 |
+
Two normalisers share a common base cleanup:
|
| 4 |
+
|
| 5 |
+
* ``normalize_for_scoring`` -- aggressive folding applied to both reference and
|
| 6 |
+
hypothesis before WER/CER, so that spellings a Persian reader considers the
|
| 7 |
+
same become identical strings.
|
| 8 |
+
* ``normalize_for_training`` -- conservative cleanup that keeps correct
|
| 9 |
+
orthography (hamza forms, ZWNJ, digits) for training targets.
|
| 10 |
+
|
| 11 |
+
Standard library only; all regexes and translation tables are built once at
|
| 12 |
+
import time (no I/O). Invisible or easily-confused characters are written as
|
| 13 |
+
``\\uXXXX`` escapes with a comment naming them.
|
| 14 |
+
"""
|
| 15 |
+
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
import re
|
| 19 |
+
import unicodedata
|
| 20 |
+
|
| 21 |
+
__all__ = [
|
| 22 |
+
"normalize_for_scoring",
|
| 23 |
+
"normalize_for_training",
|
| 24 |
+
"int_to_fa_words",
|
| 25 |
+
"spell_out_numbers",
|
| 26 |
+
"text_stats",
|
| 27 |
+
]
|
| 28 |
+
|
| 29 |
+
ZWNJ = "\u200c" # ZERO WIDTH NON-JOINER (nim-fasele)
|
| 30 |
+
|
| 31 |
+
# ---------------------------------------------------------------------------
|
| 32 |
+
# Character classes (strings to be placed inside a regex [...] class)
|
| 33 |
+
# ---------------------------------------------------------------------------
|
| 34 |
+
|
| 35 |
+
# Arabic-script letters. U+0640 (tatweel) and U+064B-U+065F (harakat) are
|
| 36 |
+
# deliberately outside these ranges.
|
| 37 |
+
_AR_LETTERS = (
|
| 38 |
+
"\u0620-\u063f" # Arabic letters incl. standalone hamza U+0621
|
| 39 |
+
"\u0641-\u064a" # feh .. Arabic yeh U+064A
|
| 40 |
+
"\u066e\u066f" # dotless beh / qaf
|
| 41 |
+
"\u0671-\u06d3" # alef wasla .. yeh barree (incl. Persian peh/tcheh/jeh/gaf/kaf U+06A9/yeh U+06CC)
|
| 42 |
+
"\u06d5" # Arabic ae
|
| 43 |
+
"\u06ee\u06ef"
|
| 44 |
+
"\u06fa-\u06fc\u06ff"
|
| 45 |
+
"\u0750-\u077f" # Arabic Supplement letters
|
| 46 |
+
)
|
| 47 |
+
# For stats on raw text: also count presentation-form letters.
|
| 48 |
+
_AR_LETTERS_RAW = _AR_LETTERS + "\ufb50-\ufd3d\ufd50-\ufdfb\ufe70-\ufefc"
|
| 49 |
+
_LATIN_LETTERS = "A-Za-z\u00c0-\u00d6\u00d8-\u00f6\u00f8-\u024f" # excludes x U+00D7, / U+00F7
|
| 50 |
+
_ANY_DIGIT = "0-9\u0660-\u0669\u06f0-\u06f9" # ASCII, Arabic-Indic, Extended Arabic-Indic (Persian)
|
| 51 |
+
_MUSIC = "\u2669\u266a\u266b\u266c\U0001f3b5\U0001f3b6\U0001f3bc" # music notes / emoji
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def _char_class(codepoints) -> str:
|
| 55 |
+
"""Compress code points into a regex character-class body."""
|
| 56 |
+
cps = sorted(set(codepoints))
|
| 57 |
+
out = []
|
| 58 |
+
i = 0
|
| 59 |
+
while i < len(cps):
|
| 60 |
+
j = i
|
| 61 |
+
while j + 1 < len(cps) and cps[j + 1] == cps[j] + 1:
|
| 62 |
+
j += 1
|
| 63 |
+
a, b = re.escape(chr(cps[i])), re.escape(chr(cps[j]))
|
| 64 |
+
out.append(a if i == j else f"{a}-{b}")
|
| 65 |
+
i = j + 1
|
| 66 |
+
return "".join(out)
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
def _junk_codepoints():
|
| 70 |
+
"""Punctuation, symbols and control/format chars (not whitespace, not ZWNJ)."""
|
| 71 |
+
scan = (
|
| 72 |
+
range(0x0000, 0x10000),
|
| 73 |
+
range(0x1D100, 0x1D200), # musical symbols
|
| 74 |
+
range(0x1F000, 0x1FB00), # emoji, pictographs, regional indicators
|
| 75 |
+
range(0xE0000, 0xE0080), # tag characters (emoji flags)
|
| 76 |
+
)
|
| 77 |
+
for r in scan:
|
| 78 |
+
for cp in r:
|
| 79 |
+
if 0xD800 <= cp <= 0xDFFF:
|
| 80 |
+
continue
|
| 81 |
+
ch = chr(cp)
|
| 82 |
+
if ch.isspace() or ch == ZWNJ:
|
| 83 |
+
continue
|
| 84 |
+
cat = unicodedata.category(ch)
|
| 85 |
+
if cat[0] in "PS" or cat in ("Cc", "Cf"):
|
| 86 |
+
yield cp
|
| 87 |
+
yield from range(0xFE00, 0xFE10) # variation selectors (emoji presentation)
|
| 88 |
+
yield from range(0x20D0, 0x2100) # combining marks for symbols (keycap U+20E3)
|
| 89 |
+
|
| 90 |
+
|
| 91 |
+
_JUNK = set(_junk_codepoints())
|
| 92 |
+
_KEEP_PUNCT = "\u060c\u061b\u061f.!:" # Arabic comma, Arabic semicolon, Arabic question mark, . ! :
|
| 93 |
+
_JUNK_RE = re.compile(f"[{_char_class(_JUNK)}]+")
|
| 94 |
+
_JUNK_EXCEPT_KEPT_RE = re.compile(
|
| 95 |
+
f"[{_char_class(_JUNK - {ord(c) for c in _KEEP_PUNCT})}]+"
|
| 96 |
+
)
|
| 97 |
+
|
| 98 |
+
# ---------------------------------------------------------------------------
|
| 99 |
+
# Base cleanup
|
| 100 |
+
# ---------------------------------------------------------------------------
|
| 101 |
+
|
| 102 |
+
# Arabic presentation forms A (U+FB50-U+FDFF) and B (U+FE70-U+FEFC): NFKC only these.
|
| 103 |
+
_PRES_RE = re.compile("[\ufb50-\ufdff\ufe70-\ufefc]+")
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
def _nfkc(m: re.Match) -> str:
|
| 107 |
+
return unicodedata.normalize("NFKC", m.group())
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
# Decomposed hamza / madda sequences -> precomposed (or plain heh).
|
| 111 |
+
_COMPOSE_MAP = {
|
| 112 |
+
"\u064a\u0654": "\u0626", # Arabic yeh + HAMZA ABOVE -> yeh with hamza
|
| 113 |
+
"\u06cc\u0654": "\u0626", # Persian yeh + HAMZA ABOVE -> yeh with hamza
|
| 114 |
+
"\u0649\u0654": "\u0626", # alef maksura + HAMZA ABOVE -> yeh with hamza
|
| 115 |
+
"\u0648\u0654": "\u0624", # waw + HAMZA ABOVE -> waw with hamza
|
| 116 |
+
"\u0627\u0653": "\u0622", # alef + MADDA ABOVE -> alef madda
|
| 117 |
+
"\u0627\u0654": "\u0623", # alef + HAMZA ABOVE -> alef hamza above
|
| 118 |
+
"\u0627\u0655": "\u0625", # alef + HAMZA BELOW -> alef hamza below
|
| 119 |
+
"\u0647\u0654": "\u0647", # heh + HAMZA ABOVE (ezafe) -> heh
|
| 120 |
+
"\u06d5\u0654": "\u0647", # ae + HAMZA ABOVE -> heh
|
| 121 |
+
}
|
| 122 |
+
_COMPOSE_RE = re.compile("|".join(_COMPOSE_MAP))
|
| 123 |
+
|
| 124 |
+
_BASE_TABLE: dict[int, str | None] = {
|
| 125 |
+
0x064A: "\u06cc", # ARABIC YEH -> PERSIAN YEH
|
| 126 |
+
0x0649: "\u06cc", # ALEF MAKSURA -> PERSIAN YEH
|
| 127 |
+
0x0643: "\u06a9", # ARABIC KAF -> PERSIAN KEHEH
|
| 128 |
+
0x06D5: "\u0647", # ARABIC AE -> HEH
|
| 129 |
+
0x06C0: "\u0647", # HEH WITH YEH ABOVE (heh + hamza) -> HEH
|
| 130 |
+
0x0640: None, # TATWEEL
|
| 131 |
+
0x0670: None, # SUPERSCRIPT ALEF
|
| 132 |
+
0x00AC: ZWNJ, # NOT SIGN, a common ZWNJ substitute
|
| 133 |
+
0x00AD: None, # SOFT HYPHEN
|
| 134 |
+
0x061C: None, # ARABIC LETTER MARK
|
| 135 |
+
0x200B: None, # ZERO WIDTH SPACE
|
| 136 |
+
0x200D: None, # ZERO WIDTH JOINER
|
| 137 |
+
0x200E: None, # LEFT-TO-RIGHT MARK
|
| 138 |
+
0x200F: None, # RIGHT-TO-LEFT MARK
|
| 139 |
+
0x2060: None, # WORD JOINER
|
| 140 |
+
0xFEFF: None, # BYTE ORDER MARK / ZWNBSP
|
| 141 |
+
}
|
| 142 |
+
for _cp in range(0x064B, 0x0660): # harakat, shadda, sukun, madda, hamza marks ...
|
| 143 |
+
_BASE_TABLE[_cp] = None
|
| 144 |
+
for _cp in (*range(0x0610, 0x061B), *range(0x06D6, 0x06DD), *range(0x06DF, 0x06E9),
|
| 145 |
+
*range(0x06EA, 0x06EE)): # Quranic annotation marks
|
| 146 |
+
_BASE_TABLE[_cp] = None
|
| 147 |
+
for _cp in (*range(0x202A, 0x202F), *range(0x2066, 0x206A)): # bidi embeddings/isolates
|
| 148 |
+
_BASE_TABLE[_cp] = None
|
| 149 |
+
for _cp in range(0xE000, 0xE004): # private-use code points we use as placeholders
|
| 150 |
+
_BASE_TABLE[_cp] = None
|
| 151 |
+
for _i in range(10): # all digit systems -> ASCII (per-mode handling later)
|
| 152 |
+
_BASE_TABLE[0x0660 + _i] = str(_i) # Arabic-Indic
|
| 153 |
+
_BASE_TABLE[0x06F0 + _i] = str(_i) # Extended Arabic-Indic (Persian)
|
| 154 |
+
for _cp in range(0x3001): # NBSP, thin/em/en spaces, etc. -> plain space
|
| 155 |
+
_ch = chr(_cp)
|
| 156 |
+
if _ch.isspace() and _ch not in " \t\n\r\x0b\x0c":
|
| 157 |
+
_BASE_TABLE[_cp] = "\n" if _ch in "\x85\u2028\u2029" else " "
|
| 158 |
+
|
| 159 |
+
|
| 160 |
+
def _base_cleanup(text: str) -> str:
|
| 161 |
+
text = _PRES_RE.sub(_nfkc, text)
|
| 162 |
+
text = _COMPOSE_RE.sub(lambda m: _COMPOSE_MAP[m.group()], text)
|
| 163 |
+
return text.translate(_BASE_TABLE)
|
| 164 |
+
|
| 165 |
+
|
| 166 |
+
# ---------------------------------------------------------------------------
|
| 167 |
+
# Non-speech tags
|
| 168 |
+
# ---------------------------------------------------------------------------
|
| 169 |
+
|
| 170 |
+
_BRACKET_TAG_RE = re.compile(r"\[[^\[\]\n]*\]")
|
| 171 |
+
_PAREN_RE = re.compile(r"\(([^()\n]*)\)")
|
| 172 |
+
_ANY_DIGIT_RE = re.compile(f"[{_ANY_DIGIT}]")
|
| 173 |
+
_MUSIC_SPAN_RE = re.compile(f"[{_MUSIC}][^\\n]*[{_MUSIC}]")
|
| 174 |
+
_MUSIC_CHAR_RE = re.compile(f"[{_MUSIC}]")
|
| 175 |
+
_SPEAKER_DASH_RE = re.compile(r"(?m)^[ \t]*[-\u2013\u2014]+(?=[ \t]|$)") # hyphen, en dash, em dash
|
| 176 |
+
_CHEVRON_RE = re.compile(r">{2,}")
|
| 177 |
+
_PAREN_TAG_MAX_WORDS = 3
|
| 178 |
+
|
| 179 |
+
|
| 180 |
+
def _is_paren_tag(inner: str) -> bool:
|
| 181 |
+
return len(inner.split()) <= _PAREN_TAG_MAX_WORDS and not _ANY_DIGIT_RE.search(inner)
|
| 182 |
+
|
| 183 |
+
|
| 184 |
+
def _paren_sub(m: re.Match) -> str:
|
| 185 |
+
return " " if _is_paren_tag(m.group(1)) else m.group()
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def _remove_tags(text: str, parens: bool = True) -> str:
|
| 189 |
+
text = _BRACKET_TAG_RE.sub(" ", text)
|
| 190 |
+
if parens:
|
| 191 |
+
text = _PAREN_RE.sub(_paren_sub, text)
|
| 192 |
+
text = _MUSIC_SPAN_RE.sub(" ", text)
|
| 193 |
+
text = _SPEAKER_DASH_RE.sub(" ", text)
|
| 194 |
+
return _CHEVRON_RE.sub(" ", text)
|
| 195 |
+
|
| 196 |
+
|
| 197 |
+
# ---------------------------------------------------------------------------
|
| 198 |
+
# Numbers
|
| 199 |
+
# ---------------------------------------------------------------------------
|
| 200 |
+
|
| 201 |
+
# NOTE: every yeh below is PERSIAN YEH U+06CC and every kaf is KEHEH U+06A9
|
| 202 |
+
# (enforced by a test).
|
| 203 |
+
_ONES = ("", "یک", "دو", "سه", "چهار", "پنج", "شش", "هفت", "هشت", "نه")
|
| 204 |
+
_TEENS = ("ده", "یازده", "دوازده", "سیزده", "چهارده", "پانزده", "شانزده", "هفده", "هجده", "نوزده")
|
| 205 |
+
_TENS = ("", "", "بیست", "سی", "چهل", "پنجاه", "شصت", "هفتاد", "هشتاد", "نود")
|
| 206 |
+
_HUNDREDS = ("", "صد", "دویست", "سیصد", "چهارصد", "پانصد", "ششصد", "هفتصد", "هشتصد", "نهصد")
|
| 207 |
+
_SCALES = ("", "هزار", "میلیون", "میلیارد", "تریلیون")
|
| 208 |
+
_ZERO = "صفر"
|
| 209 |
+
_NEG = "منفی"
|
| 210 |
+
_POINT = "ممیز"
|
| 211 |
+
_PERCENT = "درصد"
|
| 212 |
+
_AND = " \u0648 "
|
| 213 |
+
_DIGIT_WORDS = (_ZERO,) + _ONES[1:]
|
| 214 |
+
_MAX_SPELL_DIGITS = 15
|
| 215 |
+
|
| 216 |
+
|
| 217 |
+
def _three(n: int) -> str:
|
| 218 |
+
parts = []
|
| 219 |
+
h, r = divmod(n, 100)
|
| 220 |
+
if h:
|
| 221 |
+
parts.append(_HUNDREDS[h])
|
| 222 |
+
if 10 <= r < 20:
|
| 223 |
+
parts.append(_TEENS[r - 10])
|
| 224 |
+
else:
|
| 225 |
+
t, o = divmod(r, 10)
|
| 226 |
+
if t:
|
| 227 |
+
parts.append(_TENS[t])
|
| 228 |
+
if o:
|
| 229 |
+
parts.append(_ONES[o])
|
| 230 |
+
return _AND.join(parts)
|
| 231 |
+
|
| 232 |
+
|
| 233 |
+
def int_to_fa_words(n: int) -> str:
|
| 234 |
+
"""Persian cardinal words for ``-10**15 < n < 10**15``."""
|
| 235 |
+
if isinstance(n, bool) or not isinstance(n, int):
|
| 236 |
+
raise TypeError(f"expected int, got {type(n).__name__}")
|
| 237 |
+
if n < 0:
|
| 238 |
+
return f"{_NEG} {int_to_fa_words(-n)}"
|
| 239 |
+
if n >= 10**_MAX_SPELL_DIGITS:
|
| 240 |
+
raise ValueError("n must be < 10**15")
|
| 241 |
+
if n == 0:
|
| 242 |
+
return _ZERO
|
| 243 |
+
parts = []
|
| 244 |
+
idx = 0
|
| 245 |
+
while n:
|
| 246 |
+
n, g = divmod(n, 1000)
|
| 247 |
+
if g:
|
| 248 |
+
if idx == 1 and g == 1:
|
| 249 |
+
parts.append(_SCALES[1]) # 1000 -> "hezar", not "yek hezar"
|
| 250 |
+
elif idx:
|
| 251 |
+
parts.append(f"{_three(g)} {_SCALES[idx]}")
|
| 252 |
+
else:
|
| 253 |
+
parts.append(_three(g))
|
| 254 |
+
idx += 1
|
| 255 |
+
return _AND.join(reversed(parts))
|
| 256 |
+
|
| 257 |
+
|
| 258 |
+
_DIGITS_TO_ASCII = {0x0660 + i: str(i) for i in range(10)}
|
| 259 |
+
_DIGITS_TO_ASCII.update({0x06F0 + i: str(i) for i in range(10)})
|
| 260 |
+
_ASCII_TO_FA_DIGITS = {ord(str(i)): chr(0x06F0 + i) for i in range(10)}
|
| 261 |
+
|
| 262 |
+
_THOUSANDS_SEPS = ",\u066c" # comma, ARABIC THOUSANDS SEPARATOR
|
| 263 |
+
_DECIMAL_SEPS = ".\u066b/" # full stop, ARABIC DECIMAL SEPARATOR, slash
|
| 264 |
+
_NUM_CHAIN_RE = re.compile(f"[0-9]+(?:[{_THOUSANDS_SEPS}{_DECIMAL_SEPS}][0-9]+)*")
|
| 265 |
+
_NUM_SPLIT_RE = re.compile(f"([{_THOUSANDS_SEPS}{_DECIMAL_SEPS}])")
|
| 266 |
+
|
| 267 |
+
|
| 268 |
+
def _read_digits(s: str) -> str:
|
| 269 |
+
"""Integer words, or digit-by-digit for leading zeros / very long runs."""
|
| 270 |
+
if len(s) > _MAX_SPELL_DIGITS or (len(s) > 1 and s[0] == "0"):
|
| 271 |
+
return " ".join(_DIGIT_WORDS[ord(c) - 48] for c in s)
|
| 272 |
+
return int_to_fa_words(int(s))
|
| 273 |
+
|
| 274 |
+
|
| 275 |
+
def _spell_chain(chain: str) -> str:
|
| 276 |
+
if chain.isdigit():
|
| 277 |
+
return _read_digits(chain)
|
| 278 |
+
parts = _NUM_SPLIT_RE.split(chain)
|
| 279 |
+
groups, seps = parts[0::2], parts[1::2]
|
| 280 |
+
has_dec = seps[-1] in _DECIMAL_SEPS
|
| 281 |
+
int_seps = seps[:-1] if has_dec else seps
|
| 282 |
+
n_int = len(int_seps) + 1
|
| 283 |
+
if all(s in _THOUSANDS_SEPS for s in int_seps) and (
|
| 284 |
+
not int_seps
|
| 285 |
+
or (len(groups[0]) <= 3 and groups[0][0] != "0"
|
| 286 |
+
and all(len(g) == 3 for g in groups[1:n_int]))
|
| 287 |
+
):
|
| 288 |
+
words = _read_digits("".join(groups[:n_int]))
|
| 289 |
+
if has_dec:
|
| 290 |
+
words = f"{words} {_POINT} {_read_digits(groups[-1])}"
|
| 291 |
+
return words
|
| 292 |
+
# Not a well-formed number (dates, versions, lists): read each group.
|
| 293 |
+
out = [_read_digits(groups[0])]
|
| 294 |
+
for sep, g in zip(seps, groups[1:]):
|
| 295 |
+
out.append(sep)
|
| 296 |
+
out.append(_read_digits(g))
|
| 297 |
+
return " ".join(out)
|
| 298 |
+
|
| 299 |
+
|
| 300 |
+
def _spell_sub(m: re.Match) -> str:
|
| 301 |
+
s = m.string
|
| 302 |
+
words = _spell_chain(m.group())
|
| 303 |
+
if m.start() > 0 and s[m.start() - 1].isalnum():
|
| 304 |
+
words = " " + words
|
| 305 |
+
if m.end() < len(s) and s[m.end()].isalnum():
|
| 306 |
+
words += " "
|
| 307 |
+
return words
|
| 308 |
+
|
| 309 |
+
|
| 310 |
+
def spell_out_numbers(text: str) -> str:
|
| 311 |
+
"""Replace every digit run (any digit system) with Persian words."""
|
| 312 |
+
text = text.translate(_DIGITS_TO_ASCII)
|
| 313 |
+
return _NUM_CHAIN_RE.sub(_spell_sub, text)
|
| 314 |
+
|
| 315 |
+
|
| 316 |
+
# ---------------------------------------------------------------------------
|
| 317 |
+
# Scoring
|
| 318 |
+
# ---------------------------------------------------------------------------
|
| 319 |
+
|
| 320 |
+
_SCORING_TABLE: dict[int, str | None] = {
|
| 321 |
+
0x0623: "\u0627", # ALEF WITH HAMZA ABOVE -> ALEF
|
| 322 |
+
0x0625: "\u0627", # ALEF WITH HAMZA BELOW -> ALEF
|
| 323 |
+
0x0671: "\u0627", # ALEF WASLA -> ALEF
|
| 324 |
+
0x0672: "\u0627", # ALEF WITH WAVY HAMZA ABOVE -> ALEF
|
| 325 |
+
0x0673: "\u0627", # ALEF WITH WAVY HAMZA BELOW -> ALEF
|
| 326 |
+
0x0624: "\u0648", # WAW WITH HAMZA ABOVE -> WAW
|
| 327 |
+
0x0626: "\u06cc", # YEH WITH HAMZA ABOVE -> PERSIAN YEH
|
| 328 |
+
0x0629: "\u0647", # TEH MARBUTA -> HEH
|
| 329 |
+
0x0621: None, # standalone HAMZA
|
| 330 |
+
ord(ZWNJ): " ",
|
| 331 |
+
ord("%"): f" {_PERCENT} ",
|
| 332 |
+
0x066A: f" {_PERCENT} ", # ARABIC PERCENT SIGN
|
| 333 |
+
}
|
| 334 |
+
# Anything that is not a Latin/Arabic-script letter or a space is dropped.
|
| 335 |
+
_SCORING_DROP_RE = re.compile(f"[^{_LATIN_LETTERS}{_AR_LETTERS} ]+")
|
| 336 |
+
|
| 337 |
+
|
| 338 |
+
def normalize_for_scoring(text: str) -> str:
|
| 339 |
+
"""Aggressive normalisation for WER/CER (apply to reference and hypothesis)."""
|
| 340 |
+
if not text:
|
| 341 |
+
return ""
|
| 342 |
+
text = _base_cleanup(text)
|
| 343 |
+
# [..] tags and music spans are never speech; parentheses may be, so in
|
| 344 |
+
# scoring they are treated as plain punctuation (content kept).
|
| 345 |
+
text = _remove_tags(text, parens=False)
|
| 346 |
+
text = text.translate(_SCORING_TABLE)
|
| 347 |
+
text = spell_out_numbers(text)
|
| 348 |
+
text = text.lower()
|
| 349 |
+
text = _SCORING_DROP_RE.sub(" ", text)
|
| 350 |
+
return " ".join(text.split())
|
| 351 |
+
|
| 352 |
+
|
| 353 |
+
# ---------------------------------------------------------------------------
|
| 354 |
+
# Training
|
| 355 |
+
# ---------------------------------------------------------------------------
|
| 356 |
+
|
| 357 |
+
# "1,234,567" -> "1234567" (only well-formed 3-digit groups).
|
| 358 |
+
_THOUSANDS_RE = re.compile(f"(?<=[0-9])[{_THOUSANDS_SEPS}](?=[0-9]{{3}}(?![0-9]))")
|
| 359 |
+
# Number-internal marks are protected from punctuation removal via private-use placeholders.
|
| 360 |
+
_PROTECT_RE = re.compile(r"(?<=[0-9])(?:([.\u066b/:])(?=[0-9])|[ \t]*[%\u066a])")
|
| 361 |
+
_PROTECT_MAP = {".": "\ue000", "\u066b": "\ue000", "/": "\ue001", ":": "\ue002"}
|
| 362 |
+
_RESTORE_TABLE = {
|
| 363 |
+
0xE000: "\u066b", # ARABIC DECIMAL SEPARATOR
|
| 364 |
+
0xE001: "/",
|
| 365 |
+
0xE002: ":",
|
| 366 |
+
0xE003: "\u066a", # ARABIC PERCENT SIGN
|
| 367 |
+
}
|
| 368 |
+
_LATIN_TO_FA_PUNCT = {
|
| 369 |
+
ord(","): "\u060c", # ARABIC COMMA
|
| 370 |
+
ord(";"): "\u061b", # ARABIC SEMICOLON
|
| 371 |
+
ord("?"): "\u061f", # ARABIC QUESTION MARK
|
| 372 |
+
}
|
| 373 |
+
_ZWNJ_RUN_RE = re.compile(f"{ZWNJ}{{2,}}")
|
| 374 |
+
# ZWNJ is meaningful only between two Arabic-script letters.
|
| 375 |
+
_ZWNJ_STRAY_RE = re.compile(f"(?<![{_AR_LETTERS}]){ZWNJ}|{ZWNJ}(?![{_AR_LETTERS}])")
|
| 376 |
+
_PUNCT_RUN_RE = re.compile(f"\\s*([{re.escape(_KEEP_PUNCT)}]+)\\s*")
|
| 377 |
+
_REPEAT_CHAR_RE = re.compile(r"(.)\1+")
|
| 378 |
+
_LEADING_PUNCT_RE = re.compile(f"^[{re.escape(_KEEP_PUNCT)}\\s]+")
|
| 379 |
+
|
| 380 |
+
|
| 381 |
+
def _protect_sub(m: re.Match) -> str:
|
| 382 |
+
sep = m.group(1)
|
| 383 |
+
return _PROTECT_MAP[sep] if sep else "\ue003"
|
| 384 |
+
|
| 385 |
+
|
| 386 |
+
def _punct_run_sub(m: re.Match) -> str:
|
| 387 |
+
return _REPEAT_CHAR_RE.sub(r"\1", m.group(1)) + " "
|
| 388 |
+
|
| 389 |
+
|
| 390 |
+
def normalize_for_training(text: str, keep_punct: bool = False) -> str:
|
| 391 |
+
"""Clean, orthographically consistent Persian target text."""
|
| 392 |
+
if not text:
|
| 393 |
+
return ""
|
| 394 |
+
text = _base_cleanup(text)
|
| 395 |
+
text = _remove_tags(text)
|
| 396 |
+
text = _THOUSANDS_RE.sub("", text)
|
| 397 |
+
text = _PROTECT_RE.sub(_protect_sub, text)
|
| 398 |
+
if keep_punct:
|
| 399 |
+
text = text.translate(_LATIN_TO_FA_PUNCT)
|
| 400 |
+
text = _JUNK_EXCEPT_KEPT_RE.sub(" ", text)
|
| 401 |
+
else:
|
| 402 |
+
text = _JUNK_RE.sub(" ", text)
|
| 403 |
+
text = _ZWNJ_RUN_RE.sub(ZWNJ, text)
|
| 404 |
+
text = _ZWNJ_STRAY_RE.sub("", text)
|
| 405 |
+
text = " ".join(text.split())
|
| 406 |
+
if keep_punct:
|
| 407 |
+
text = _PUNCT_RUN_RE.sub(_punct_run_sub, text).strip()
|
| 408 |
+
text = _LEADING_PUNCT_RE.sub("", text)
|
| 409 |
+
text = text.translate(_RESTORE_TABLE)
|
| 410 |
+
return text.translate(_ASCII_TO_FA_DIGITS)
|
| 411 |
+
|
| 412 |
+
|
| 413 |
+
# ---------------------------------------------------------------------------
|
| 414 |
+
# Stats (raw text, for data filtering)
|
| 415 |
+
# ---------------------------------------------------------------------------
|
| 416 |
+
|
| 417 |
+
_WS_RE = re.compile(r"\s+")
|
| 418 |
+
_AR_LETTER_RAW_RE = re.compile(f"[{_AR_LETTERS_RAW}]+")
|
| 419 |
+
_LATIN_LETTER_RE = re.compile(f"[{_LATIN_LETTERS}]+")
|
| 420 |
+
_URL_HANDLE_RE = re.compile(
|
| 421 |
+
r"(?:https?://|www\.)\S+"
|
| 422 |
+
r"|(?<![\w@])@[A-Za-z0-9_.]{2,}"
|
| 423 |
+
r"|(?<!\w)#[^\s#]+"
|
| 424 |
+
r"|\b[A-Za-z0-9-]+\.(?:com|ir|org|net|io|info|me|co|tv|app|dev|xyz)\b",
|
| 425 |
+
re.IGNORECASE,
|
| 426 |
+
)
|
| 427 |
+
_CHAR_RUN_RE = re.compile(r"(\S)\1*")
|
| 428 |
+
|
| 429 |
+
|
| 430 |
+
def _count(regex: re.Pattern, text: str) -> int:
|
| 431 |
+
return sum(m.end() - m.start() for m in regex.finditer(text))
|
| 432 |
+
|
| 433 |
+
|
| 434 |
+
def text_stats(text: str) -> dict:
|
| 435 |
+
"""Cheap statistics on the raw text for filtering training data."""
|
| 436 |
+
n_nonspace = len(_WS_RE.sub("", text))
|
| 437 |
+
n_ar = _count(_AR_LETTER_RAW_RE, text)
|
| 438 |
+
n_lat = _count(_LATIN_LETTER_RE, text)
|
| 439 |
+
max_run = max((m.end() - m.start() for m in _CHAR_RUN_RE.finditer(text)), default=0)
|
| 440 |
+
|
| 441 |
+
max_rep = 0
|
| 442 |
+
prev, run = None, 0
|
| 443 |
+
for tok in text.split():
|
| 444 |
+
key = _JUNK_RE.sub("", tok).replace(ZWNJ, "").casefold()
|
| 445 |
+
if not key:
|
| 446 |
+
continue
|
| 447 |
+
run = run + 1 if key == prev else 1
|
| 448 |
+
prev = key
|
| 449 |
+
max_rep = max(max_rep, run)
|
| 450 |
+
|
| 451 |
+
has_tag = bool(_BRACKET_TAG_RE.search(text)) or any(
|
| 452 |
+
_is_paren_tag(m.group(1)) for m in _PAREN_RE.finditer(text)
|
| 453 |
+
)
|
| 454 |
+
return {
|
| 455 |
+
"n_chars": len(text),
|
| 456 |
+
"n_words": len(text.split()),
|
| 457 |
+
"persian_letter_ratio": n_ar / n_nonspace if n_nonspace else 0.0,
|
| 458 |
+
"latin_letter_ratio": n_lat / n_nonspace if n_nonspace else 0.0,
|
| 459 |
+
"n_digits": len(_ANY_DIGIT_RE.findall(text)),
|
| 460 |
+
"has_bracket_tag": has_tag,
|
| 461 |
+
"has_music_symbol": bool(_MUSIC_CHAR_RE.search(text)),
|
| 462 |
+
"has_url_or_handle": bool(_URL_HANDLE_RE.search(text)),
|
| 463 |
+
"max_char_run": max_run,
|
| 464 |
+
"max_word_repeat": max_rep,
|
| 465 |
+
}
|
score.py
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Score saved hypotheses: results/<model>/<testset>.jsonl -> error-rate table.
|
| 2 |
+
|
| 3 |
+
uv run --no-project --with jiwer python score.py [results_dir] [--show N]
|
| 4 |
+
|
| 5 |
+
Columns per test set: WER / CER / CER ignoring spaces (all on canonical spelling, see
|
| 6 |
+
nimruz/scoring.py), then in brackets the WER without the word-spacing rules.
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
from __future__ import annotations
|
| 10 |
+
|
| 11 |
+
import json
|
| 12 |
+
import sys
|
| 13 |
+
from pathlib import Path
|
| 14 |
+
|
| 15 |
+
import jiwer
|
| 16 |
+
|
| 17 |
+
from nimruz.scoring import canonical, error_rates
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def main():
|
| 21 |
+
args = [a for a in sys.argv[1:] if not a.startswith("--")]
|
| 22 |
+
show = int(sys.argv[sys.argv.index("--show") + 1]) if "--show" in sys.argv else 0
|
| 23 |
+
root = Path(args[0] if args else "results")
|
| 24 |
+
table: dict[str, dict[str, dict]] = {}
|
| 25 |
+
rows_of = {}
|
| 26 |
+
for f in sorted(root.glob("*/*.jsonl")):
|
| 27 |
+
rows = [json.loads(l) for l in open(f, encoding="utf-8")]
|
| 28 |
+
table.setdefault(f.parent.name, {})[f.stem] = error_rates([r["ref"] for r in rows], [r["hyp"] for r in rows])
|
| 29 |
+
rows_of[(f.parent.name, f.stem)] = rows
|
| 30 |
+
sets = sorted({s for m in table.values() for s in m})
|
| 31 |
+
print("| model | " + " | ".join(sets) + " |")
|
| 32 |
+
print("|---|" + "---|" * len(sets))
|
| 33 |
+
key = lambda kv: sum(v["wer"] for v in kv[1].values()) / len(kv[1])
|
| 34 |
+
for model, res in sorted(table.items(), key=key):
|
| 35 |
+
cells = [f"{res[s]['wer']:.1f} / {res[s]['cer']:.1f} / {res[s]['ncer']:.1f} [{res[s]['wer_raw']:.1f}]" if s in res else "-" for s in sets]
|
| 36 |
+
print(f"| {model.replace('__', '/')} | " + " | ".join(cells) + " |")
|
| 37 |
+
if show:
|
| 38 |
+
for (model, s), rows in rows_of.items():
|
| 39 |
+
print(f"\n## {model} / {s}: worst {show}")
|
| 40 |
+
per = sorted(((jiwer.wer(canonical(r["ref"]) or "-", canonical(r["hyp"])), canonical(r["ref"]), canonical(r["hyp"])) for r in rows), reverse=True)[:show]
|
| 41 |
+
for e, a, b in per:
|
| 42 |
+
print(f"{e:.2f}\n ref: {a}\n hyp: {b}")
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
if __name__ == "__main__":
|
| 46 |
+
main()
|
transcribe.py
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Transcribe Persian audio of any length with NAVA.
|
| 2 |
+
|
| 3 |
+
pip install torch transformers
|
| 4 |
+
python transcribe.py audio.mp3
|
| 5 |
+
python transcribe.py audio.mp3 --digits fa # numbers, times and dates in Persian digits
|
| 6 |
+
python transcribe.py audio.mp3 --digits ascii # ... in 0-9
|
| 7 |
+
python transcribe.py audio.mp3 --beams 5 # slightly more accurate, about half the speed
|
| 8 |
+
|
| 9 |
+
Audio is decoded with ffmpeg if it is installed, otherwise with librosa (`pip install librosa`).
|
| 10 |
+
|
| 11 |
+
NAVA transcribes one window of up to 30 seconds at a time and does not produce timestamps,
|
| 12 |
+
so Whisper's built-in long-form mode must not be used with it. This script cuts longer audio
|
| 13 |
+
at quiet points into pieces of at most 29 seconds and joins the text.
|
| 14 |
+
"""
|
| 15 |
+
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
import argparse
|
| 19 |
+
import shutil
|
| 20 |
+
import subprocess
|
| 21 |
+
|
| 22 |
+
import numpy as np
|
| 23 |
+
import torch
|
| 24 |
+
|
| 25 |
+
SR = 16000
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def load_audio(path: str) -> np.ndarray:
|
| 29 |
+
"""Any audio file -> mono float32 at 16 kHz."""
|
| 30 |
+
if shutil.which("ffmpeg"):
|
| 31 |
+
raw = subprocess.run(
|
| 32 |
+
["ffmpeg", "-v", "error", "-i", path, "-f", "f32le", "-ac", "1", "-ar", str(SR), "pipe:1"],
|
| 33 |
+
capture_output=True, check=True,
|
| 34 |
+
).stdout
|
| 35 |
+
return np.frombuffer(raw, dtype=np.float32).copy()
|
| 36 |
+
import librosa
|
| 37 |
+
|
| 38 |
+
return librosa.load(path, sr=SR, mono=True)[0].astype(np.float32)
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
def split_long(wav: np.ndarray, max_s: float = 29.0, min_s: float = 12.0) -> list[np.ndarray]:
|
| 42 |
+
"""Pieces of at most `max_s` seconds, each cut at the quietest 0.3 s between `min_s` and `max_s`
|
| 43 |
+
after the previous cut."""
|
| 44 |
+
if len(wav) <= int(30.0 * SR):
|
| 45 |
+
return [wav]
|
| 46 |
+
hop, win = int(0.02 * SR), int(0.3 * SR)
|
| 47 |
+
energy = np.convolve(wav.astype(np.float32) ** 2, np.ones(win, dtype=np.float32) / win, mode="same")[::hop]
|
| 48 |
+
pieces, start = [], 0
|
| 49 |
+
while len(wav) - start > int(max_s * SR):
|
| 50 |
+
lo, hi = (start + int(min_s * SR)) // hop, (start + int(max_s * SR)) // hop
|
| 51 |
+
cut = (lo + int(np.argmin(energy[lo:hi]))) * hop
|
| 52 |
+
pieces.append(wav[start:cut])
|
| 53 |
+
start = cut
|
| 54 |
+
pieces.append(wav[start:])
|
| 55 |
+
return pieces
|
| 56 |
+
|
| 57 |
+
|
| 58 |
+
@torch.inference_mode()
|
| 59 |
+
def transcribe(wav: np.ndarray, model, processor, beams: int = 1, batch_size: int = 8) -> str:
|
| 60 |
+
pieces = [p for p in split_long(wav) if len(p) > 0.1 * SR]
|
| 61 |
+
texts = []
|
| 62 |
+
for i in range(0, len(pieces), batch_size):
|
| 63 |
+
feats = processor.feature_extractor(pieces[i : i + batch_size], sampling_rate=SR, return_tensors="pt").input_features
|
| 64 |
+
ids = model.generate(feats.to(model.device, model.dtype), language="fa", task="transcribe",
|
| 65 |
+
num_beams=beams, max_new_tokens=440)
|
| 66 |
+
texts += [t.strip() for t in processor.batch_decode(ids, skip_special_tokens=True)]
|
| 67 |
+
return " ".join(t for t in texts if t)
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def main():
|
| 71 |
+
from transformers import WhisperForConditionalGeneration, WhisperProcessor
|
| 72 |
+
|
| 73 |
+
ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
|
| 74 |
+
ap.add_argument("audio", nargs="+")
|
| 75 |
+
ap.add_argument("--model", default="Nimruz/NAVA")
|
| 76 |
+
ap.add_argument("--digits", choices=["none", "fa", "ascii"], default="none",
|
| 77 |
+
help="rewrite spoken numbers, times and dates in digits")
|
| 78 |
+
ap.add_argument("--beams", type=int, default=1)
|
| 79 |
+
args = ap.parse_args()
|
| 80 |
+
|
| 81 |
+
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
|
| 82 |
+
dtype = torch.float16 if device == "cuda" else torch.float32
|
| 83 |
+
processor = WhisperProcessor.from_pretrained(args.model)
|
| 84 |
+
model = WhisperForConditionalGeneration.from_pretrained(args.model).to(device=device, dtype=dtype).eval()
|
| 85 |
+
|
| 86 |
+
for path in args.audio:
|
| 87 |
+
text = transcribe(load_audio(path), model, processor, beams=args.beams)
|
| 88 |
+
if args.digits != "none":
|
| 89 |
+
from nimruz.itn import itn # the nimruz/ folder next to this script
|
| 90 |
+
|
| 91 |
+
text = itn(text, digits=args.digits)
|
| 92 |
+
print(text)
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
if __name__ == "__main__":
|
| 96 |
+
main()
|