xmanii commited on
Commit
3ecb728
·
verified ·
1 Parent(s): 92163ed

Add model card, charts and helper code

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ assets/benchmark-dark.png filter=lfs diff=lfs merge=lfs -text
37
+ assets/benchmark-light.png filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,276 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - fa
4
+ license: mit
5
+ library_name: transformers
6
+ pipeline_tag: automatic-speech-recognition
7
+ base_model: openai/whisper-large-v3-turbo
8
+ tags:
9
+ - whisper
10
+ - persian
11
+ - farsi
12
+ - speech-recognition
13
+ - asr
14
+ - faster-whisper
15
+ - whisper.cpp
16
+ datasets:
17
+ - PerSets/youtube-persian-asr
18
+ - PerSets/filimo-persian-asr
19
+ - Reza2kn/Common-Voice-25-Persian-Raw
20
+ - MahtaFetrat/Mana-TTS
21
+ metrics:
22
+ - wer
23
+ - cer
24
+ ---
25
+
26
+ # NAVA
27
+
28
+ **Persian speech-to-text.** NAVA is `openai/whisper-large-v3-turbo` (809M parameters) fine-tuned on 579 hours of filtered Persian speech.
29
+
30
+ - **Strongest on conversational Persian** among the four open models tested here.
31
+ - **Writes standard Persian spelling**, with half-spaces where they belong: «می‌شود», «سازمان‌ها».
32
+ - **Runs wherever Whisper runs**: transformers, faster-whisper, whisper.cpp.
33
+
34
+ This is a preview from a single training run. Read [Limitations](#limitations) before relying on it.
35
+
36
+ | Repo | Use it with |
37
+ |---|---|
38
+ | [Nimruz/NAVA](https://huggingface.co/Nimruz/NAVA) (this one) | transformers |
39
+ | [Nimruz/NAVA-ct2](https://huggingface.co/Nimruz/NAVA-ct2) | faster-whisper / CTranslate2 |
40
+ | [Nimruz/NAVA-ggml](https://huggingface.co/Nimruz/NAVA-ggml) | whisper.cpp |
41
+
42
+ ## Results
43
+
44
+ <picture>
45
+ <source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/benchmark-dark.png">
46
+ <img src="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/benchmark-light.png" alt="Bar charts of word error rate for four Persian speech models on four test sets. NAVA is lowest on C1Tech (11.7%) and PSRB (17.2%), second on FLEURS (8.7%) and Common Voice (13.2%)." width="800">
47
+ </picture>
48
+
49
+ Word error rate (%), with character error rate ignoring spaces in brackets. Lower is better.
50
+
51
+ | Model | Params | C1Tech (conversational) | PSRB (mixed, noisy) | FLEURS fa (read) | Common Voice 22 fa (read) |
52
+ |---|---|---|---|---|---|
53
+ | **NAVA** | 809M | **11.7 (3.8)** | **17.2 (5.8)** | 8.7 (2.3) | 13.2 (3.3) |
54
+ | Reza2kn/Shenava-Koochik-v1.5 | 114M | 13.4 (5.2) | 22.5 (9.0) | **8.1** (3.0) | **12.5** (3.2) |
55
+ | mehdi-hf/nemotron-asr-streaming-farsi | 600M | 12.4 (4.5) | 23.1 (9.3) | 9.4 (3.2) | 18.4 (5.4) |
56
+ | givia/whisper-large-v3-turbo-persian | 809M | 13.4 (5.1) | 22.8 (9.9) | 10.1 (5.2) | 18.3 (6.4) |
57
+ | openai/whisper-large-v3-turbo (base) | 809M | not run | not run | 19.7 (6.2) | 47.2 (26.8) |
58
+
59
+ Test sets:
60
+
61
+ - **C1Tech/Persian-ASR-Benchmark**, all 2,101 clips.
62
+ - **PartAI/PSRB**, all 344 clips.
63
+ - **FLEURS** `fa_ir` test, all 871 clips.
64
+ - **Common Voice 22** Persian test, a fixed random sample of 1,500 of the 10,676 clips.
65
+
66
+ Every model was run here, with the same scorer, on the same clips. See [How the scores were computed](#how-the-scores-were-computed) for the method and for the same table under plain scoring.
67
+
68
+ ## Quick start
69
+
70
+ Input is 16 kHz mono audio. Always pass the language; NAVA was trained with the Persian prompt only.
71
+
72
+ ### transformers
73
+
74
+ ```python
75
+ import soundfile as sf
76
+ import torch
77
+ from transformers import WhisperForConditionalGeneration, WhisperProcessor
78
+
79
+ model_id = "Nimruz/NAVA"
80
+ processor = WhisperProcessor.from_pretrained(model_id)
81
+ model = WhisperForConditionalGeneration.from_pretrained(model_id).to("cuda").half()
82
+
83
+ wav, sr = sf.read("audio.wav", dtype="float32") # 16 kHz mono, up to 30 seconds
84
+ features = processor(wav, sampling_rate=16000, return_tensors="pt").input_features
85
+ ids = model.generate(features.to("cuda").half(), language="fa", task="transcribe", max_new_tokens=440)
86
+ print(processor.batch_decode(ids, skip_special_tokens=True)[0])
87
+ ```
88
+
89
+ On a machine without a GPU, drop `.to("cuda")` and `.half()`.
90
+
91
+ ### Any length, any audio format
92
+
93
+ The script in this repo decodes the file, cuts long audio at quiet points, and joins the text:
94
+
95
+ ```bash
96
+ git clone https://huggingface.co/Nimruz/NAVA && cd NAVA
97
+ pip install torch transformers
98
+ python transcribe.py interview.mp3
99
+ python transcribe.py interview.mp3 --digits fa # numbers, times and dates in digits
100
+ ```
101
+
102
+ It uses ffmpeg to read audio if ffmpeg is installed, and librosa otherwise.
103
+
104
+ ### faster-whisper
105
+
106
+ ```python
107
+ from faster_whisper import WhisperModel
108
+
109
+ model = WhisperModel("Nimruz/NAVA-ct2", device="cuda", compute_type="float16")
110
+ segments, _ = model.transcribe(
111
+ "audio.wav", language="fa", beam_size=1,
112
+ without_timestamps=True, condition_on_previous_text=False,
113
+ )
114
+ print(" ".join(s.text.strip() for s in segments))
115
+ ```
116
+
117
+ On CPU use `device="cpu", compute_type="float32"`, or `"int8"` for less memory.
118
+
119
+ ### whisper.cpp
120
+
121
+ ```bash
122
+ huggingface-cli download Nimruz/NAVA-ggml ggml-nava-q5_0.bin --local-dir .
123
+ whisper-cli -m ggml-nava-q5_0.bin -l fa -nt -f audio.wav
124
+ ```
125
+
126
+ | File | Size | Notes |
127
+ |---|---|---|
128
+ | `ggml-nava.bin` | 1.51 GB | 16-bit float |
129
+ | `ggml-nava-q8_0.bin` | 0.81 GB | 8-bit quantized |
130
+ | `ggml-nava-q5_0.bin` | 0.53 GB | 5-bit quantized |
131
+
132
+ whisper.cpp expects 16 kHz 16-bit WAV: `ffmpeg -i in.mp3 -ar 16000 -ac 1 -c:a pcm_s16le audio.wav`.
133
+
134
+ ### Which one to pick
135
+
136
+ | You have | Use |
137
+ |---|---|
138
+ | An NVIDIA GPU and Python | transformers or faster-whisper |
139
+ | A CPU server | faster-whisper |
140
+ | A laptop or phone, no Python | whisper.cpp with the `q5_0` or `q8_0` file |
141
+
142
+ Each export was checked against the original model on 10 FLEURS test clips: the transformers weights gave identical text on all 10, and the CTranslate2 and three whisper.cpp files on 9 of 10. The whisper.cpp files were built and tested with whisper.cpp v1.7.4.
143
+
144
+ ## Things to know when running it
145
+
146
+ ### Audio longer than 30 seconds
147
+
148
+ Whisper's built-in long-form mode relies on timestamp tokens, and NAVA was fine-tuned without them. Do not use that mode: on the 13 FLEURS clips over 30 seconds it scored 31.9% word error that way.
149
+
150
+ Cut long audio into pieces of at most 30 seconds at quiet points and transcribe each piece. `transcribe.py` does this, and with it the same 13 clips score 11.1%. A voice-activity splitter such as `vad_filter=True` in faster-whisper should serve the same purpose, but that path has not been tested with NAVA.
151
+
152
+ ### Numbers, dates and times
153
+
154
+ NAVA writes numbers the way they are spoken: «هزار و چهارصد و دو». The formatter in this repo turns them into digits after transcription:
155
+
156
+ ```python
157
+ from nimruz.itn import itn
158
+
159
+ itn("سال هزار و چهارصد و دو بیست و پنج درصد رشد داشتیم") # سال ۱۴۰۲ ۲۵ درصد رشد داشتیم
160
+ itn("ساعت ده و نیم دوازده مرداد میام") # ساعت ۱۰:۳۰ ۱۲ مرداد میام
161
+ itn("قیمتش دو میلیون و پونصد هزار تومنه", digits="ascii") # قیمتش 2500000 تومنه
162
+ ```
163
+
164
+ It is rule-based and conservative: lone small words such as «یک» and «نه» stay as words, and ordinals such as «بیست و پنجم» are left alone.
165
+
166
+ ### Beam search
167
+
168
+ Greedy decoding is the default everywhere above. Five beams lower word error by 0.1 to 0.3 points (11.6 / 16.9 / 8.6 / 13.1 on the four test sets) at about half the speed.
169
+
170
+ ## How the scores were computed
171
+
172
+ Persian sources disagree on word spacing: «می‌کنم», «می کنم» and «میکنم» are the same word. Scoring those as errors measures the spelling convention, not the recognition. So before comparing, reference and hypothesis both go through the same steps:
173
+
174
+ 1. Rewrite spacing variants to the standard half-space forms.
175
+ 2. Fold Arabic and Persian letter variants, and remove diacritics and punctuation.
176
+ 3. Spell out digits as words.
177
+ 4. Treat the half-space as a space.
178
+
179
+ The same table with step 1 switched off (word error rate, %):
180
+
181
+ | Model | C1Tech | PSRB | FLEURS fa | Common Voice 22 fa |
182
+ |---|---|---|---|---|
183
+ | **NAVA** | 17.0 | **17.2** | 9.7 | 20.6 |
184
+ | Reza2kn/Shenava-Koochik-v1.5 | 18.0 | 22.7 | **8.9** | **19.8** |
185
+ | mehdi-hf/nemotron-asr-streaming-farsi | 17.3 | 23.3 | 10.3 | 25.7 |
186
+ | givia/whisper-large-v3-turbo-persian | **16.4** | 26.1 | 10.3 | 23.4 |
187
+
188
+ Notes on the comparison:
189
+
190
+ - Numbers for the other models were measured here and can differ from their own cards, which use other normalizers and clip selections.
191
+ - Decoding was greedy for every Whisper model, with long clips cut at quiet points for NAVA and Whisper's long-form mode for the others. Shenava used its CTC head; Nemotron used its 1.12 s look-ahead setting.
192
+ - Shenava was used to filter NAVA's training data (see below), so the two are not fully independent. It was not used on any test set.
193
+
194
+ To score your own outputs the same way, write one JSON object per line with `ref` and `hyp` fields to `results/<model-name>/<test-set>.jsonl` and run:
195
+
196
+ ```bash
197
+ pip install jiwer
198
+ python score.py
199
+ ```
200
+
201
+ ## Training data
202
+
203
+ <picture>
204
+ <source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/data-dark.png">
205
+ <img src="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/data-light.png" alt="Hours used out of hours available per training source: YouTube subtitles 294 of 385, Filimo subtitles 176 of 244, Mana-TTS 60 of 103, Common Voice 49 of 372." width="800">
206
+ </picture>
207
+
208
+ | Source | Kind | Raw hours | Used hours |
209
+ |---|---|---|---|
210
+ | PerSets/youtube-persian-asr | YouTube subtitles, conversational | 385 | 294 (824 videos) |
211
+ | PerSets/filimo-persian-asr | Film and series subtitles | 244 | 176 (466 titles) |
212
+ | MahtaFetrat/Mana-TTS, parts 1 to 69 | Magazine readings, one speaker | 103 | 60 |
213
+ | Common Voice 25 fa, validated | Read sentences | 372 | 49 (1,120 speakers) |
214
+ | **Total** | | **1,104** | **579** |
215
+
216
+ How clips were chosen:
217
+
218
+ - **Agreement check.** Every clip was transcribed by an independent model (Shenava Koochik v1.5). Clips whose label differed from that transcript by more than 30% of characters were dropped. This removed about 120 hours, mostly subtitles that do not line up with the audio.
219
+ - **Text checks.** Clips with digits, Latin letters, URLs, stage directions, speaker labels or runs of repeated characters were dropped.
220
+ - **Speech-rate and audio checks.** Clips under 0.8 seconds, implausibly slow or fast for their text, or near silent were dropped.
221
+ - **Diversity caps.** At most 40 minutes per Common Voice speaker and 3 recordings of the same sentence; Mana-TTS capped at 60 hours so one voice does not dominate.
222
+ - **Test protection.** Common Voice test speakers and test sentences were excluded. Mana-TTS parts 70 to 77 (a public leaderboard's test set) were not used. 2% of YouTube and Filimo recordings were held out for development.
223
+
224
+ Training text was normalized to Persian letter forms, no diacritics, no punctuation, and one spacing convention (the standard half-space forms). The rules and word list for that are in `nimruz/ortho.py` and `assets/lexicon.json`.
225
+
226
+ ## Training
227
+
228
+ <picture>
229
+ <source media="(prefers-color-scheme: dark)" srcset="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/training-dark.png">
230
+ <img src="https://huggingface.co/Nimruz/NAVA/resolve/main/assets/training-light.png" alt="Line chart of development word error rate by training step. Conversational clips fall from 45.5% to 15.1%; FLEURS dev falls from 18.3% to 8.3%." width="800">
231
+ </picture>
232
+
233
+ | | |
234
+ |---|---|
235
+ | Base model | openai/whisper-large-v3-turbo |
236
+ | Trainable | Decoder and encoder layers 9 to 32 (conv stem and first 8 encoder layers frozen) |
237
+ | Steps | 10,386 optimizer steps, effective batch 32 (8 × 4 accumulation) |
238
+ | Audio seen | About 1,590 hours, roughly 2.75 passes over the data |
239
+ | Learning rate | 1e-5 peak, 400 warmup steps, cosine decay to 1e-6 |
240
+ | Optimizer | AdamW, betas (0.9, 0.98), weight decay 0.01, gradient clipping 1.0 |
241
+ | Precision | bfloat16 autocast |
242
+ | Augmentation | SpecAugment (time and feature masking, 5% each) |
243
+ | Sample packing | Neighbouring clips concatenated up to 30 s; 15% of samples left as single clips |
244
+ | Compute | 6.5 GPU-hours on one GPU (A100 for the first 500 steps, L40S after) |
245
+
246
+ Development word error went from 18.3% to 8.3% on FLEURS dev and from 45.5% to 15.1% on held-out conversational clips (250 clips each). The conversational score was 14.1% at step 9,000 and 15.1% at the end; the released weights are the final step, which scored better on the FLEURS test set.
247
+
248
+ ## Limitations
249
+
250
+ - **No punctuation.** Output is a plain stream of words.
251
+ - **Persian only.** English words are written in Persian script or mis-transcribed.
252
+ - **No timestamps**, and no built-in long-form decoding (see above).
253
+ - **Numbers are spelled out** unless the formatter is applied.
254
+ - **Spacing is not fully uniform.** Some forms are still written joined, such as «شدهاند» for «شده‌اند».
255
+ - **Behaviour on silence, music and non-speech audio has not been tested.** Whisper models can produce text where there is no speech.
256
+ - **One training run.** No second seed and no hyperparameter search.
257
+ - **Evaluation is on four public sets only**, and the Common Voice figure is from a 1,500-clip sample.
258
+
259
+ ## What's in this repo
260
+
261
+ | Path | What it is |
262
+ |---|---|
263
+ | `model.safetensors`, `config.json`, tokenizer files | The model, half precision |
264
+ | `transcribe.py` | Transcribe audio of any length and format |
265
+ | `nimruz/itn.py` | Spoken numbers, dates and times to digits |
266
+ | `nimruz/textnorm.py`, `nimruz/ortho.py`, `assets/lexicon.json` | Text normalization and spacing rules used for training text and scoring |
267
+ | `nimruz/scoring.py`, `score.py` | The scorer behind the tables above |
268
+ | `assets/*.png` | The charts on this page |
269
+
270
+ ## Licence and data
271
+
272
+ NAVA is released under the MIT licence, the same as its base model. The four training datasets are published under CC0. The YouTube and Filimo sets consist of audio from online video with its subtitles; check that this suits your use before deploying commercially.
273
+
274
+ ## Acknowledgements
275
+
276
+ Thanks to the authors of the training datasets, to the Shenava project for the model used in filtering, and to the authors of the FLEURS, Common Voice, C1Tech and PSRB test sets.
assets/benchmark-dark.png ADDED

Git LFS Details

  • SHA256: e3f0e4465926c810521b099542d81fb744598fa9a74b499d85be4db00a378ee2
  • Pointer size: 131 Bytes
  • Size of remote file: 105 kB
assets/benchmark-light.png ADDED

Git LFS Details

  • SHA256: 0922ff9629d696c69ca28134331e68e4e54ea1406fea2707615eed3eccaa03d9
  • Pointer size: 131 Bytes
  • Size of remote file: 106 kB
assets/data-dark.png ADDED
assets/data-light.png ADDED
assets/lexicon.json ADDED
The diff for this file is too large to render. See raw diff
 
assets/training-dark.png ADDED
assets/training-light.png ADDED
nimruz/__init__.py ADDED
File without changes
nimruz/itn.py ADDED
@@ -0,0 +1,513 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Rule-based Persian (Farsi) inverse text normalisation (ITN).
2
+
3
+ The ASR model writes numbers the way they are spoken ("هزار و چهارصد و دو");
4
+ ``itn`` rewrites cardinals, percentages, decimals, clock times, dates and
5
+ spoken digit strings as digits ("۱۴۰۲"). It is deliberately conservative:
6
+ anything that is not a well-formed number expression is left untouched,
7
+ byte for byte.
8
+
9
+ Standard library only; all tables and regexes are built once at import time
10
+ (no I/O). Invisible or easily-confused characters are written as ``\\uXXXX``
11
+ escapes with a comment naming them. Every yeh below is PERSIAN YEH U+06CC and
12
+ every kaf is KEHEH U+06A9 (enforced by a test); input with Arabic yeh/kaf is
13
+ folded before lookup.
14
+ """
15
+
16
+ from __future__ import annotations
17
+
18
+ import re
19
+
20
+ __all__ = ["itn", "words_to_int"]
21
+
22
+ ZWNJ = "‌" # ZERO WIDTH NON-JOINER (nim-fasele)
23
+
24
+ # ---------------------------------------------------------------------------
25
+ # Lexicon
26
+ # ---------------------------------------------------------------------------
27
+
28
+ # Word classes of the number grammar.
29
+ _Z, _O, _T, _D, _H, _S = range(6) # zero, 1-9, 10-19, 20-90, 100-900, scales
30
+
31
+ _AND = "و" # WAW, the conjunction "va"
32
+
33
+ _LEX: dict[str, tuple[int, int]] = {} # normalised word -> (value, class)
34
+
35
+
36
+ def _add(cls: int, value: int, *spellings: str) -> None:
37
+ for s in spellings:
38
+ _LEX[s] = (value, cls)
39
+
40
+
41
+ _add(_Z, 0, "صفر")
42
+ _add(_O, 1, "یک", "یه") # "یه" only ever converts inside a larger number
43
+ _add(_O, 2, "دو")
44
+ _add(_O, 3, "سه")
45
+ _add(_O, 4, "چهار", "چار")
46
+ _add(_O, 5, "پنج")
47
+ _add(_O, 6, "شش", "شیش")
48
+ _add(_O, 7, "هفت")
49
+ _add(_O, 8, "هشت")
50
+ _add(_O, 9, "نه")
51
+ _add(_T, 10, "ده")
52
+ _add(_T, 11, "یازده")
53
+ _add(_T, 12, "دوازده")
54
+ _add(_T, 13, "سیزده")
55
+ _add(_T, 14, "چهارده", "چارده")
56
+ _add(_T, 15, "پانزده", "پونزده")
57
+ _add(_T, 16, "شانزده", "شونزده")
58
+ _add(_T, 17, "هفده", "هیفده", "هیوده")
59
+ _add(_T, 18, "هجده", "هیجده", "هژده", "هیژده")
60
+ _add(_T, 19, "نوزده")
61
+ _add(_D, 20, "بیست", "بیس")
62
+ _add(_D, 30, "سی")
63
+ _add(_D, 40, "چهل")
64
+ _add(_D, 50, "پنجاه")
65
+ _add(_D, 60, "شصت")
66
+ _add(_D, 70, "هفتاد")
67
+ _add(_D, 80, "هشتاد")
68
+ _add(_D, 90, "نود")
69
+ _add(_H, 100, "صد", "یکصد")
70
+ _add(_H, 200, "دویست", "دویس", "دوصد")
71
+ _add(_H, 300, "سیصد", "سهصد")
72
+ _add(_H, 400, "چهارصد", "چارصد")
73
+ _add(_H, 500, "پانصد", "پونصد", "پنجصد")
74
+ _add(_H, 600, "ششصد", "شیشصد")
75
+ _add(_H, 700, "هفتصد")
76
+ _add(_H, 800, "هشتصد")
77
+ _add(_H, 900, "نهصد")
78
+ _add(_S, 10**3, "هزار")
79
+ _add(_S, 10**6, "میلیون", "ملیون")
80
+ _add(_S, 10**9, "میلیارد", "ملیارد")
81
+ _add(_S, 10**12, "تریلیون")
82
+
83
+ _HUNDRED = "صد"
84
+ _YE_COLLOQ = "یه" # colloquial "one"; also the indefinite article
85
+ _POINT = "ممیز"
86
+ _PERCENT = "درصد"
87
+ _SAAT = "ساعت"
88
+ _MINUTE = "دقیقه"
89
+ _HALF = "نیم"
90
+ _QUARTER = "ربع"
91
+ _TO = "به"
92
+ _DURATION_WORDS = frozenset({_QUARTER, _MINUTE, "ثانیه", _SAAT})
93
+
94
+ _MONTHS = frozenset({
95
+ # Solar Hijri
96
+ "فروردین", "اردیبهشت", "خرداد", "تیر", "مرداد", "امرداد", "شهریور",
97
+ "مهر", "آبان", "آذر", "دی", "بهمن", "اسفند",
98
+ # Gregorian, as written in Persian
99
+ "ژانویه", "فوریه", "مارس", "آوریل", "مه", "می", "ژوئن", "ژوئیه", "جولای",
100
+ "اوت", "آگوست", "سپتامبر", "اکتبر", "نوامبر", "دسامبر",
101
+ })
102
+ # Month names that are also common words (arrow, seal/kindness, "dey", fog,
103
+ # the verb prefix "mi"): a lone 1-9 day before them converts only if a year
104
+ # follows.
105
+ _AMBIGUOUS_MONTHS = frozenset({"تیر", "مهر", "دی", "مه", "می"})
106
+
107
+ # ---------------------------------------------------------------------------
108
+ # Character handling
109
+ # ---------------------------------------------------------------------------
110
+
111
+ # Arabic-script letters, tatweel and harakat (no digits, no punctuation).
112
+ _LETTER = (
113
+ "ء-غ" # hamza .. ghain
114
+ "ـ-ٟ" # tatweel, feh .. Arabic yeh, harakat
115
+ "ٰ-ۓ" # superscript alef .. yeh barree (Persian letters)
116
+ "ەۺ-ۼۿ"
117
+ )
118
+ _DIGIT = "0-9٠-٩۰-۹" # ASCII, Arabic-Indic, Persian
119
+ _WORD_PAT = f"[{_LETTER}]+(?:{ZWNJ}+[{_LETTER}]+)*"
120
+ # A digit token keeps its internal separators: 1,234 ۳٫۵ 10:30 1402/05/12
121
+ _DIGITS_PAT = f"[{_DIGIT}]+(?:[.,:/٫٬][{_DIGIT}]+)*" # + ARABIC DECIMAL / THOUSANDS SEPARATOR
122
+ _TOKEN_RE = re.compile(f"(?P<w>{_WORD_PAT})|(?P<d>{_DIGITS_PAT})")
123
+ # Gap allowed inside a number phrase: horizontal whitespace only.
124
+ _SPACE_GAP_RE = re.compile("[ \t  ]+") # space, tab, NBSP, NARROW NBSP
125
+
126
+ _NORM_TABLE: dict[int, str | None] = {
127
+ 0x064A: "ی", # ARABIC YEH -> PERSIAN YEH
128
+ 0x0649: "ی", # ALEF MAKSURA -> PERSIAN YEH
129
+ 0x0643: "ک", # ARABIC KAF -> PERSIAN KEHEH
130
+ 0x200C: None, # ZWNJ
131
+ 0x0640: None, # TATWEEL
132
+ 0x0670: None, # SUPERSCRIPT ALEF
133
+ }
134
+ for _cp in range(0x064B, 0x0660): # harakat
135
+ _NORM_TABLE[_cp] = None
136
+
137
+ _TO_FA = {ord(str(i)): chr(0x06F0 + i) for i in range(10)} # -> EXTENDED ARABIC-INDIC
138
+ _TO_FA.update({0x0660 + i: chr(0x06F0 + i) for i in range(10)})
139
+ _TO_ASCII = {0x06F0 + i: str(i) for i in range(10)}
140
+ _TO_ASCII.update({0x0660 + i: str(i) for i in range(10)})
141
+ _DECIMAL_SEP = {"fa": "٫", "ascii": "."} # ARABIC DECIMAL SEPARATOR
142
+
143
+
144
+ def _norm(word: str) -> str:
145
+ return word.translate(_NORM_TABLE)
146
+
147
+
148
+ def _lookup(norm: str) -> tuple[int, int, bool] | None:
149
+ """(value, class, glued_and) for a number word, else None."""
150
+ hit = _LEX.get(norm)
151
+ if hit is not None:
152
+ return hit[0], hit[1], False
153
+ # "هزارو", "صدو": conjunction glued on. Not for 1-9 ("سهو" = mistake).
154
+ if len(norm) > 2 and norm[-1] == _AND:
155
+ hit = _LEX.get(norm[:-1])
156
+ if hit is not None and hit[1] not in (_O, _Z):
157
+ return hit[0], hit[1], True
158
+ return None
159
+
160
+
161
+ def _merge_hundreds(a: str, b: str) -> tuple[int, int, bool] | None:
162
+ """"سی" + "صد" -> 300 when a hundreds word was split by a space."""
163
+ if not b.startswith(_HUNDRED):
164
+ return None
165
+ hit = _lookup(a + b)
166
+ return hit if hit is not None and hit[1] == _H else None
167
+
168
+
169
+ def _is_ordinal(norm: str) -> bool:
170
+ """"پنجم", "پنجمین", "سوم", "سی‌ام", "هزارمین" ..."""
171
+ if norm.endswith("مین"):
172
+ norm = norm[:-2]
173
+ if norm.endswith("ام"):
174
+ base = norm[:-2]
175
+ elif norm.endswith("م"):
176
+ base = norm[:-1]
177
+ else:
178
+ return False
179
+ return base in _LEX or base == "سو"
180
+
181
+
182
+ # ---------------------------------------------------------------------------
183
+ # Grammar
184
+ # ---------------------------------------------------------------------------
185
+
186
+ _AND_ITEM = None # marker for "va" in a flattened item list
187
+
188
+
189
+ def _can_follow(small: list[int], cls: int) -> bool:
190
+ if not small:
191
+ return cls in (_O, _T, _D, _H)
192
+ last = small[-1]
193
+ return (last == _H and cls in (_O, _T, _D)) or (last == _D and cls == _O)
194
+
195
+
196
+ def _parse(items: list) -> int | None:
197
+ """Value of a flattened item list [(value, class) | None-for-va], or None."""
198
+ if not items or items[0] is _AND_ITEM or items[-1] is _AND_ITEM:
199
+ return None
200
+ if len(items) == 1 and items[0][1] == _Z:
201
+ return 0
202
+ parts: list[list[tuple[int, int]]] = [[]]
203
+ for it in items:
204
+ if it is _AND_ITEM:
205
+ parts.append([])
206
+ else:
207
+ parts[-1].append(it)
208
+ total = 0
209
+ prev_scale: int | None = None
210
+ small: list[int] = []
211
+ small_val = 0
212
+ for part in parts:
213
+ if not part or len(part) > 2 or any(c == _Z for _, c in part):
214
+ return None
215
+ if len(part) == 2: # "<small> <scale>": "پنج هزار", "سیصد میلیون"
216
+ (v1, c1), (v2, c2) = part
217
+ if c2 != _S or not _can_follow(small, c1):
218
+ return None
219
+ small_val += v1
220
+ scale = v2
221
+ else:
222
+ v, c = part[0]
223
+ if c != _S:
224
+ if not _can_follow(small, c):
225
+ return None
226
+ small.append(c)
227
+ small_val += v
228
+ continue
229
+ if small or v != 1000: # only "هزار" stands alone
230
+ return None
231
+ small_val, scale = 1, v
232
+ if prev_scale is not None and scale >= prev_scale:
233
+ return None
234
+ total += small_val * scale
235
+ prev_scale, small, small_val = scale, [], 0
236
+ return total + small_val
237
+
238
+
239
+ def words_to_int(words: list[str]) -> int | None:
240
+ """Parse a cardinal phrase already split into tokens; None if invalid.
241
+
242
+ ``["هزار", "و", "چهارصد", "و", "دو"] -> 1402``. Accepts the colloquial
243
+ spellings, a glued "و" ("هزارو") and a split hundreds word (["سی", "صد"]).
244
+ A lone "یه" is not a number.
245
+ """
246
+ norms = [_norm(t) for w in words for t in w.split()]
247
+ if not norms:
248
+ return None
249
+ items: list = []
250
+ i = 0
251
+ n_units = 0
252
+ while i < len(norms):
253
+ w = norms[i]
254
+ hit = _merge_hundreds(w, norms[i + 1]) if i + 1 < len(norms) else None
255
+ if hit is not None:
256
+ i += 2
257
+ else:
258
+ i += 1
259
+ if w == _AND:
260
+ items.append(_AND_ITEM)
261
+ continue
262
+ hit = _lookup(w)
263
+ if hit is None:
264
+ return None
265
+ n_units += 1
266
+ items.append((hit[0], hit[1]))
267
+ if hit[2]:
268
+ items.append(_AND_ITEM)
269
+ if n_units == 1 and norms == [_YE_COLLOQ]:
270
+ return None
271
+ return _parse(items)
272
+
273
+
274
+ # ---------------------------------------------------------------------------
275
+ # Scanner
276
+ # ---------------------------------------------------------------------------
277
+
278
+ _NUM, _VA, _WORD, _DIGITS = range(4) # unit kinds
279
+
280
+
281
+ class _Unit:
282
+ __slots__ = ("kind", "start", "end", "norm", "value", "cls", "glued")
283
+
284
+ def __init__(self, kind, start, end, norm="", value=0, cls=-1, glued=False):
285
+ self.kind, self.start, self.end, self.norm = kind, start, end, norm
286
+ self.value, self.cls, self.glued = value, cls, glued
287
+
288
+
289
+ def _tokenize(text: str) -> list[_Unit]:
290
+ raw = [(m.start(), m.end(), m.lastgroup) for m in _TOKEN_RE.finditer(text)]
291
+ units: list[_Unit] = []
292
+ i = 0
293
+ while i < len(raw):
294
+ s, e, g = raw[i]
295
+ if g == "d":
296
+ units.append(_Unit(_DIGITS, s, e))
297
+ i += 1
298
+ continue
299
+ norm = _norm(text[s:e])
300
+ if i + 1 < len(raw) and raw[i + 1][2] == "w" and _SPACE_GAP_RE.fullmatch(text[e:raw[i + 1][0]]):
301
+ s2, e2, _ = raw[i + 1]
302
+ hit = _merge_hundreds(norm, _norm(text[s2:e2]))
303
+ if hit is not None:
304
+ units.append(_Unit(_NUM, s, e2, norm, *hit))
305
+ i += 2
306
+ continue
307
+ if norm == _AND:
308
+ units.append(_Unit(_VA, s, e, norm))
309
+ else:
310
+ hit = _lookup(norm)
311
+ if hit is None:
312
+ units.append(_Unit(_WORD, s, e, norm))
313
+ else:
314
+ units.append(_Unit(_NUM, s, e, norm, *hit))
315
+ i += 1
316
+ return units
317
+
318
+
319
+ class _Scanner:
320
+ def __init__(self, text: str, sep: str):
321
+ self.text = text
322
+ self.sep = sep
323
+ self.u = _tokenize(text)
324
+ self.n = len(self.u)
325
+ # linked[i]: unit i follows unit i-1 with only horizontal whitespace between.
326
+ self.linked = [False] + [
327
+ bool(_SPACE_GAP_RE.fullmatch(text[self.u[k - 1].end:self.u[k].start]))
328
+ for k in range(1, self.n)
329
+ ]
330
+ self.reps: list[tuple[int, int, str]] = []
331
+
332
+ # -- helpers -----------------------------------------------------------
333
+
334
+ def _is_word(self, k: int, words) -> bool:
335
+ return k < self.n and self.linked[k] and self.u[k].kind == _WORD and self.u[k].norm in words
336
+
337
+ def _is_num(self, k: int) -> bool:
338
+ return k < self.n and self.linked[k] and self.u[k].kind == _NUM
339
+
340
+ def _run_end(self, i: int) -> int:
341
+ """End of the maximal run of number words and "va" starting at i."""
342
+ m = i + 1
343
+ while m < self.n and self.linked[m] and self.u[m].kind in (_NUM, _VA):
344
+ m += 1
345
+ return m
346
+
347
+ def _ordinal_end(self, i: int) -> int | None:
348
+ """If the run at i ends "... و <ordinal>", the index after the ordinal."""
349
+ m = self._run_end(i)
350
+ if (m < self.n and self.linked[m] and self.u[m].kind == _WORD and _is_ordinal(self.u[m].norm)
351
+ and (self.u[m - 1].kind == _VA or self.u[m - 1].glued)):
352
+ return m + 1
353
+ return None
354
+
355
+ def _items(self, i: int, j: int) -> list:
356
+ items: list = []
357
+ for k in range(i, j):
358
+ un = self.u[k]
359
+ if un.kind == _VA:
360
+ items.append(_AND_ITEM)
361
+ else:
362
+ items.append((un.value, un.cls))
363
+ if un.glued:
364
+ items.append(_AND_ITEM)
365
+ return items
366
+
367
+ def _longest(self, i: int) -> tuple[int, int] | None:
368
+ """(end, value) of the longest grammatical cardinal starting at unit i."""
369
+ if i >= self.n or self.u[i].kind != _NUM:
370
+ return None
371
+ for j in range(self._run_end(i), i, -1):
372
+ if self.u[j - 1].kind == _VA:
373
+ continue
374
+ v = _parse(self._items(i, j))
375
+ if v is not None:
376
+ if j == i + 1 and self.u[i].norm == _YE_COLLOQ:
377
+ return None
378
+ return j, v
379
+ return None
380
+
381
+ def _digit_run(self, i: int) -> int:
382
+ k = i
383
+ while (k < self.n and (k == i or self.linked[k]) and self.u[k].kind == _NUM
384
+ and self.u[k].cls in (_Z, _O) and self.u[k].norm != _YE_COLLOQ):
385
+ k += 1
386
+ return k - i
387
+
388
+ def _rep(self, i: int, j: int, s: str) -> None:
389
+ self.reps.append((self.u[i].start, self.u[j - 1].end, s))
390
+
391
+ # -- patterns ----------------------------------------------------------
392
+
393
+ def _time(self, k: int) -> tuple[int, str | None] | None:
394
+ """Clock time after "ساعت"; (end, replacement-or-None-to-skip) or None."""
395
+ if not self._is_num(k):
396
+ return None
397
+ end = self._ordinal_end(k)
398
+ if end is not None:
399
+ return end, None
400
+ sp = self._longest(k)
401
+ if sp is None:
402
+ return None
403
+ j, h = sp
404
+ if self._is_word(j, (_QUARTER, _MINUTE)) and self._is_word(j + 1, (_TO,)):
405
+ # "یک ربع به ده": leave the whole expression as words
406
+ end = j + 2
407
+ if self._is_num(end):
408
+ sp2 = self._longest(end)
409
+ if sp2 is not None:
410
+ end = sp2[0]
411
+ return end, None
412
+ if h > 24 or self._is_word(j, _DURATION_WORDS):
413
+ return None
414
+ if j < self.n and self.linked[j] and self.u[j].kind == _VA:
415
+ if self._is_word(j + 1, (_HALF,)):
416
+ return j + 2, f"{h}:30"
417
+ if self._is_word(j + 1, (_QUARTER,)):
418
+ return j + 2, f"{h}:15"
419
+ if self._is_num(j + 1):
420
+ sp2 = self._longest(j + 1)
421
+ if sp2 is not None and self._is_word(sp2[0], (_MINUTE,)):
422
+ if sp2[1] > 59:
423
+ return None
424
+ return sp2[0] + 1, f"{h}:{sp2[1]:02d}"
425
+ return j, str(h)
426
+
427
+ def _frac(self, k: int) -> tuple[int, str] | None:
428
+ """Fractional part after "ممیز"."""
429
+ if not self._is_num(k):
430
+ return None
431
+ r = self._digit_run(k)
432
+ if r >= 2 or (r == 1 and self.u[k].cls == _Z):
433
+ return k + r, "".join(str(self.u[x].value) for x in range(k, k + r))
434
+ sp = self._longest(k)
435
+ return None if sp is None else (sp[0], str(sp[1]))
436
+
437
+ def _plain_converts(self, i: int, sp: tuple[int, int]) -> bool:
438
+ return not (sp[0] == i + 1 and sp[1] < 10)
439
+
440
+ def _number(self, i: int) -> int:
441
+ end = self._ordinal_end(i)
442
+ if end is not None:
443
+ return end
444
+ r = self._digit_run(i)
445
+ if r >= 3 and not self._is_num(i + r):
446
+ self._rep(i, i + r, "".join(str(self.u[x].value) for x in range(i, i + r)))
447
+ return i + r
448
+ sp = self._longest(i)
449
+ if sp is None:
450
+ return i + 1
451
+ j, v = sp
452
+ if self._is_word(j, (_POINT,)):
453
+ fr = self._frac(j + 1)
454
+ if fr is not None:
455
+ self._rep(i, fr[0], f"{v}{self.sep}{fr[1]}")
456
+ return fr[0]
457
+ if self._is_word(j, (_PERCENT,)):
458
+ self._rep(i, j, str(v))
459
+ return j
460
+ if 1 <= v <= 31 and self._is_word(j, _MONTHS):
461
+ month = self.u[j].norm
462
+ year = self._longest(j + 1) if self._is_num(j + 1) else None
463
+ if month not in _AMBIGUOUS_MONTHS or (year is not None and self._plain_converts(j + 1, year)):
464
+ self._rep(i, j, str(v))
465
+ return j
466
+ if self._plain_converts(i, sp):
467
+ self._rep(i, j, str(v))
468
+ return j
469
+
470
+ def run(self) -> list[tuple[int, int, str]]:
471
+ u, n, i = self.u, self.n, 0
472
+ while i < n:
473
+ un = u[i]
474
+ if un.kind == _NUM:
475
+ i = self._number(i)
476
+ continue
477
+ if un.kind == _WORD and un.norm == _SAAT:
478
+ t = self._time(i + 1)
479
+ if t is not None:
480
+ end, rep = t
481
+ if rep is not None:
482
+ self._rep(i + 1, end, rep)
483
+ i = end
484
+ continue
485
+ elif un.kind == _DIGITS and self._is_num(i + 1) and u[i + 1].cls == _S:
486
+ # "۲۰ هزار": mixed digits + scale words are left alone
487
+ i = self._run_end(i + 1)
488
+ continue
489
+ i += 1
490
+ return self.reps
491
+
492
+
493
+ def itn(text: str, digits: str = "fa") -> str:
494
+ """Rewrite spoken Persian number expressions in ``text`` as digits.
495
+
496
+ ``digits="fa"`` writes Persian digits (U+06F0..U+06F9) and U+066B as the
497
+ decimal separator; ``digits="ascii"`` writes 0-9 and ".". Digits already
498
+ in the text are converted to the requested style; nothing else that is
499
+ not a number expression is changed.
500
+ """
501
+ if digits not in _DECIMAL_SEP:
502
+ raise ValueError("digits must be 'fa' or 'ascii'")
503
+ if not text:
504
+ return text
505
+ reps = _Scanner(text, _DECIMAL_SEP[digits]).run()
506
+ out = []
507
+ pos = 0
508
+ for s, e, r in reps:
509
+ out.append(text[pos:s])
510
+ out.append(r)
511
+ pos = e
512
+ out.append(text[pos:])
513
+ return "".join(out).translate(_TO_FA if digits == "fa" else _TO_ASCII)
nimruz/ortho.py ADDED
@@ -0,0 +1,461 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Persian word-spacing standardiser for speech-recognition training targets.
2
+
3
+ Transcripts from different sources write the same word three ways: with a
4
+ ZWNJ (``\\u200c``, nim-fasele -- the standard), with a space, or joined.
5
+ ``SpacingFixer`` rewrites target text to ONE convention, the standard ZWNJ
6
+ forms, for a small set of high-frequency morphemes:
7
+
8
+ * ``verb_prefix_spaced`` ``mi X`` / ``nemi X`` -> ``mi<ZWNJ>X``
9
+ * ``verb_prefix_joined`` ``miX`` / ``nemiX`` -> ``mi<ZWNJ>X``
10
+ * ``plural_spaced`` ``X ha(y..)`` -> ``X<ZWNJ>ha(y..)``
11
+ * ``plural_joined`` ``Xha(y..)`` -> ``X<ZWNJ>ha(y..)`` (lexicon only)
12
+ * ``comparative_spaced`` ``X tar(in)`` -> ``X<ZWNJ>tar(in)`` (lexicon only)
13
+
14
+ Precision matters far more than recall: a wrong rewrite corrupts a training
15
+ label, a missed one leaves a harmless variant. Every rule therefore needs
16
+ positive evidence (a built-in verb grammar, or attestation in a lexicon built
17
+ from the corpus) and is blocked by curated exception lists.
18
+
19
+ ZWNJ after non-joining letters: the only source that uses ZWNJ (YouTube
20
+ subtitles) almost never writes it after a letter that does not join forward
21
+ (alef, dal, zal, reh, zain, jeh, waw ...): e.g. before "ha" 1-2 ZWNJ forms vs
22
+ 64-276 plain joins per letter. The two spellings render identically, so the
23
+ convention here is: ZWNJ after joining letters, plain join after non-joining
24
+ letters (``kshvr-ha`` -> joined, ``ketab-ha`` -> ZWNJ). The prefix "mi" ends
25
+ in yeh (joining), so it always takes a ZWNJ.
26
+
27
+ Input is expected to be the output of
28
+ ``textnorm.normalize_for_training(text, keep_punct=False)``. Standard
29
+ library only; all tables and regexes are built at import time (no I/O).
30
+ Every yeh is PERSIAN YEH U+06CC and every kaf is KEHEH U+06A9.
31
+ """
32
+
33
+ from __future__ import annotations
34
+
35
+ import json
36
+ import re
37
+ from collections import Counter
38
+
39
+ __all__ = ["build_lexicon", "save_lexicon", "load_lexicon", "SpacingFixer"]
40
+
41
+ ZWNJ = "‌" # ZERO WIDTH NON-JOINER (nim-fasele)
42
+
43
+ # Letters that never join to the following letter: a ZWNJ after them is
44
+ # invisible, and the convention here is to write a plain join instead.
45
+ _NON_JOINING = frozenset(
46
+ "ا" # ALEF
47
+ "آ" # ALEF WITH MADDA
48
+ "أ" # ALEF WITH HAMZA ABOVE
49
+ "إ" # ALEF WITH HAMZA BELOW
50
+ "ٱ" # ALEF WASLA
51
+ "د" # DAL
52
+ "ذ" # THAL
53
+ "ر" # REH
54
+ "ز" # ZAIN
55
+ "ژ" # JEH
56
+ "و" # WAW
57
+ "ؤ" # WAW WITH HAMZA
58
+ "ة" # TEH MARBUTA
59
+ "ء" # HAMZA
60
+ )
61
+
62
+ # A token the rules may touch: Arabic-script letters only (no digits, Latin, ZWNJ).
63
+ _FA_WORD_RE = re.compile("[ء-غف-يپچژکگی]+")
64
+
65
+
66
+ def _glue(left: str, right: str) -> str:
67
+ """Join two pieces with ZWNJ, or plainly after a non-joining letter."""
68
+ return left + right if left[-1] in _NON_JOINING else left + ZWNJ + right
69
+
70
+
71
+ def _words(s: str) -> frozenset:
72
+ return frozenset(s.split())
73
+
74
+
75
+ # ---------------------------------------------------------------------------
76
+ # Verb grammar (rules 1 and 2)
77
+ # ---------------------------------------------------------------------------
78
+
79
+ _MI = "می"
80
+ _NEMI = "نمی"
81
+
82
+ # Present stems, formal and colloquial (single-letter colloquial stems are
83
+ # handled separately with a restricted set of endings).
84
+ _PRESENT_STEMS = _words("""
85
+ کن شو گو گوی دون دان تون توان خوا خواه دار بین ده گیر زن خور ذار زار
86
+ گذار مون مان رس کش ساز خون خوان نویس فهم شناس بر پرس ترس خند
87
+ افت فت وفت ایست وایس ریز پذیر یاب بخش دوز سوز فروش خر پر چرخ جنگ کوش گرد گذر
88
+ گذر شور نشین شین خواب پوش چسب باش فرست بند بار پیچ لرز شکن شکون کوب رقص دو
89
+ چین شنو کار کاو خیز رو رون رسون رسان گذرون گذران ترسون ترسان چرخون چرخان
90
+ پوشون پوشان خوابون خوابان سوزون سوزان کشون شکاف مال پاش جوش درخش تاب تپ چک
91
+ پز پسند پرست پرداز پوس چپون انداز نداز اندیش آموز آمیز آرا گشا نما نمای نام
92
+ شمار شمر جو جوی خار لیس مک بوس سنج طلب کاه افزا افزای رنج رنجان لغز نگر
93
+ نگار جنب جنبون گز تراش ران پرور فشار فشر ارز میر بال شکف لرزون دزد بخشون
94
+ بر دوون پرون پیوند گوش چرخ خواهی ترکون ترک سپار سپر خرا کوچ شمر گرای
95
+ """)
96
+ # Past stems (also take the empty ending: 3rd person singular).
97
+ _PAST_STEMS = _words("""
98
+ کرد شد رفت گفت دونست دانست تونست توانست خواست داشت دید اومد داد گرفت
99
+ زد خورد ذاشت زاشت گذاشت موند ماند رسید کشید کشت ساخت خوند خواند نوشت فهمید
100
+ شناخت برد پرسید ترسید خندید افتاد فتاد وفتاد ایستاد وایساد ریخت
101
+ پذیرفت یافت بخشید دوخت سوخت فروخت خرید پرید چرخید جنگید کوشید گشت گذشت شست
102
+ نشست خوابید پوشید چسبید فرستاد بست بارید پیچید لرزید شکست کوبید رقصید دوید
103
+ چید شنید کاشت رسوند رساند گذروند گذراند انداخت نداخت اندیشید آموخت پنداشت
104
+ مرد زیست باخت پخت سوزوند سوزاند تابید تپید چکید درخشید جوشید مالید پاشید
105
+ جنبید لغزید نامید شمرد بالید خارید نمود کاست افزود طلبید رنجاند راند پرستید
106
+ پرداخت پسندید نگریست گریست گریخت شکافت کند جست بوسید لیسید مکید پوسید گزید
107
+ چرخوند ترسوند خوابوند بود شکوند پوشوند کشوند دزدید ارزید آمیخت گذروند رسید
108
+ """)
109
+ # Single-letter colloquial stems: ش (shodan) ر (raftan) گ (goftan) د (dadan)
110
+ # and آ/ا (amadan). Restricted endings keep nouns such as میش، میر، مید out.
111
+ _SHORT_STEMS_CLITIC = "گد" # may take an object clitic: میگمش، میدمت
112
+ _SHORT_STEMS_PLAIN = "شر"
113
+
114
+ _END_PRES = "م ی د ه یم ید ند ن ین یی ییم یید یند"
115
+ _END_PAST = "م ی یم ید ند ین ن ه"
116
+ _END_SHORT = "م ی ه یم ید ین ن"
117
+ _END_COME = "م ی د یم ید ین"
118
+ _CLITIC = "م ت ش مون تون شون"
119
+
120
+
121
+ def _alt(words) -> str:
122
+ return "|".join(sorted((re.escape(w) for w in words), key=len, reverse=True))
123
+
124
+
125
+ _VERB_BODY_RE = re.compile(
126
+ "(?:"
127
+ f"(?:{_alt(_PRESENT_STEMS)})(?:(?:{_alt(_END_PRES.split())})(?:{_alt(_CLITIC.split())})?"
128
+ f"|ت(?:{_alt(_CLITIC.split())}))"
129
+ f"|(?:{_alt(_PAST_STEMS)})(?:{_alt(_END_PAST.split())})?(?:{_alt(_CLITIC.split())})?"
130
+ f"|[{_SHORT_STEMS_CLITIC}](?:{_alt(_END_SHORT.split())})(?:{_alt(_CLITIC.split())})?"
131
+ f"|[{_SHORT_STEMS_PLAIN}](?:{_alt(_END_SHORT.split())})"
132
+ # impersonal / modal forms with no personal ending
133
+ "|توان|بایست|بایستی|باید"
134
+ ")"
135
+ )
136
+
137
+ # amadan / avardan: always spelled with ALEF WITH MADDA after the prefix
138
+ # (YouTube: "می<ZWNJ>آد" 30 times, "می<ZWNJ>اد" never), so a bare-alef body
139
+ # such as colloquial "میاد" is rewritten to "می<ZWNJ>آد".
140
+ _COME_BODY_RE = re.compile(
141
+ "\u0622(?:" # ALEF WITH MADDA
142
+ f"(?:{_alt(_END_COME.split())})"
143
+ f"|(?:ی|ر|ور)(?:{_alt(_END_PRES.split())})(?:{_alt(_CLITIC.split())})?"
144
+ f"|(?:ورد|مد)(?:{_alt(_END_PAST.split())})?(?:{_alt(_CLITIC.split())})?"
145
+ ")"
146
+ )
147
+
148
+ _PAST_TAIL_RE = re.compile(f"(?:{_alt(_PAST_STEMS)})$")
149
+
150
+ # Real words that begin with می / نمی and must never be split.
151
+ _MI_EXCEPTIONS = _words("""
152
+ می نمی میز میزان میزانی میان میانه میانی میانگین میانجی میانمار میدان میدون
153
+ میدانی میهن میهنی میهمان میهمانی میهمانان میوه میل میله میلاد میلادی میلیون
154
+ میلیونی میلیونر میلیونها میلیارد میلیاردی میلیاردها میلیمتر میکروب میکروفون
155
+ میکروفن میکروسکوپ میکرو میمون میمونها میخ میخک میخکوب میراث میرزا میرزای میش
156
+ میگو مینا مینی میت میسر میثاق میکاپ میکس میس میترا میر میرا میزبان میزبانی
157
+ میخانه میکده میگرن میگون میخواره میخوارگی میگسار میهنپرست مینو مینیاتور
158
+ میشل میلان میامی میا مید مین میم میه میسیون میسون میلر میلاس مینسک میرویس
159
+ میلتون میکائیل میکل میکی میگل میناب میبد میاندوآب میانرو میانداری میدانداری
160
+ میزگرد میزها میزی میزش میزو میوهها میوههای میوهای مینیبوس مینیمال میراثش
161
+ میلیاردش میلیاردو میانبر میانسال میانجیگری میقات میعاد میمنت میمیک میکرون
162
+ میلگرد میمیرم نمک نمکی نمایش نمایان نماز نمونه نمره نمد نمود نمودار نمناک
163
+ نمایشگاه نماینده نمایندگی نمین نمیر
164
+ """)
165
+
166
+ # Never join "mi" to these (function words, pronouns, interjections).
167
+ _MI_STOP_NEXT = _words("""
168
+ و که را رو به از در تا با هم یا اگر اگه ولی اما من تو او اون ما شما اونا آنها
169
+ این آن یه یک چی چه کی نه خب هی بله برای واسه هر همه
170
+ """)
171
+
172
+ _NUMBER_WORDS = _words("""
173
+ صفر یک یه دو سه چهار پنج شش هفت هشت نه ده یازده دوازده سیزده چهارده پانزده
174
+ پونزده شانزده شونزده هفده هجده هیجده نوزده بیست سی چهل پنجاه شصت هفتاد هشتاد
175
+ نود صد دویست سیصد چهارصد پانصد پو��صد ششصد هفتصد هشتصد نهصد هزار میلیون میلیارد
176
+ اول دوم سوم
177
+ """)
178
+ _DIGIT_RE = re.compile("[0-9٠-٩۰-۹]")
179
+
180
+ # ---------------------------------------------------------------------------
181
+ # Plural suffix (rules 3 and 4)
182
+ # ---------------------------------------------------------------------------
183
+
184
+ _PLURAL_SUFFIXES = (
185
+ "ها", "های", "هایی", "هایم", "هایت", "هایش", "هایمان", "هایتان", "هایشان",
186
+ "هام", "هات", "هاش", "هامون", "هاتون", "هاشون", "هارو",
187
+ )
188
+ _PLURAL_SET = frozenset(_PLURAL_SUFFIXES)
189
+ # Xهای / Xهام ... can also be Xه + ای / ام ... (indefinite, clitic, perfect).
190
+ _HEH_CLITIC_SUFFIXES = frozenset(
191
+ ("های", "هام", "هات", "هاش", "هایم", "هاید", "هامون", "هاتون", "هاشون")
192
+ )
193
+ _AMBIGUOUS_SPACED = frozenset(("ها", "هام", "هات", "هاش"))
194
+ # Longest first so "Xهایشان" is split as X + هایشان.
195
+ _PLURAL_SPLIT_RE = re.compile(f"^(.{{2,}}?)({_alt(_PLURAL_SUFFIXES)})$")
196
+
197
+ # Words that cannot be the stem of a spaced plural ("آره ها" = interjection).
198
+ _PLURAL_STOP_LEFT = _words("""
199
+ آره نه بله خب ها وای آها آهان اوه چی کی که و یا تا با از به در را رو هم اگه اگر
200
+ ولی اما من تو او ما شما اینه اونه بیا برو بگو ببین باشه نباشه هست نیست داره
201
+ دارم داری شه میشه نمیشه بشه کرده شده بود بوده دیگه حالا البته واقعا اصلا
202
+ خوبه بده آخه پس چرا یعنی مگه دیگر چون وقتی ببخشید بفرمایید بالا پایین کن
203
+ بکن نکن بره نره بیا نیا بشه برم بریم واسه برای مثل مثه مثلا اینجا اونجا
204
+ کجا پیش جلو عقب زیر بیرون تو توی
205
+ """)
206
+
207
+ # Real words that merely end in a plural-suffix string (rule 4).
208
+ _PLURAL_EXCEPTIONS = _words("""
209
+ تنها تنهای تنهایی تنهاییم تنهاییت تنهاییش بها بهای بهایی رها رهایی نها نهایی
210
+ نهایت بینهایت نهایتا انتها انتهای ابتدا منها منهای الها الهام اشتها اشتهای
211
+ ابهام اتهام اوهام ایهام سهام مهام استفهام شبهات جهات توجهات امهات دهات کراهات
212
+ مباهات شبهای تشبیهات تنبیهات نزاهت سفاهت کراهت فقاهت وجاهت بداهت سراهام
213
+ گواها آگاها شاها پادشاها الهی الهایم سپاها منتها منتهای منتهایش
214
+ """)
215
+
216
+ # ---------------------------------------------------------------------------
217
+ # Comparatives (rule 5)
218
+ # ---------------------------------------------------------------------------
219
+
220
+ _COMPARATIVE_SUFFIXES = ("تر", "ترین")
221
+
222
+
223
+ # ---------------------------------------------------------------------------
224
+ # Lexicon
225
+ # ---------------------------------------------------------------------------
226
+
227
+
228
+ def build_lexicon(texts) -> dict[str, int]:
229
+ """Token counts over an iterable of lines (whitespace tokens; ZWNJ stays inside)."""
230
+ c: Counter = Counter()
231
+ for line in texts:
232
+ c.update(line.split())
233
+ return dict(c)
234
+
235
+
236
+ def save_lexicon(lexicon, path) -> None:
237
+ with open(path, "w", encoding="utf-8") as f:
238
+ json.dump(dict(lexicon), f, ensure_ascii=False, sort_keys=True)
239
+
240
+
241
+ def load_lexicon(path) -> dict[str, int]:
242
+ with open(path, encoding="utf-8") as f:
243
+ return {str(k): int(v) for k, v in json.load(f).items()}
244
+
245
+
246
+ # ---------------------------------------------------------------------------
247
+ # Fixer
248
+ # ---------------------------------------------------------------------------
249
+
250
+ _MIN_STEM_COUNT = 3 # rule 4: X must occur standalone at least this often
251
+ _MIN_HEH_COUNT = 2 # rule 4: stem+heh attested this often makes Xهای ambiguous
252
+ _MIN_ZWNJ_EVIDENCE = 2 # rules 1/2: "mi<ZWNJ>X" attested at least this often
253
+
254
+
255
+ class SpacingFixer:
256
+ """Rewrite Persian text to the standard ZWNJ spacing convention.
257
+
258
+ ``lexicon`` (token -> count, see ``build_lexicon``) enables the
259
+ evidence-based rules; with ``None`` only the grammar/list based rules
260
+ (1, 2, 3) apply, rule 3 with stricter guards. Decisions are cached per token / token pair, so the
261
+ fixer is cheap to call on every sample in a data loader.
262
+ """
263
+
264
+ def __init__(self, lexicon: dict[str, int] | None = None):
265
+ self.lexicon = lexicon
266
+ self._tok_cache: dict[str, tuple[str, str] | None] = {}
267
+ self._pair_cache: dict[tuple[str, str, bool], tuple[str, str] | None] = {}
268
+
269
+ # -- evidence ------------------------------------------------------------
270
+
271
+ def _count(self, w: str) -> int:
272
+ return self.lexicon.get(w, 0) if self.lexicon is not None else 0
273
+
274
+ def _verb_body(self, prefix: str, body: str) -> str | None:
275
+ """Standard spelling of ``body`` (the part after می/نمی) if it is a
276
+ known verb form, else ``None``."""
277
+ if len(body) < 2 or not _FA_WORD_RE.fullmatch(body):
278
+ return None
279
+ # "ده" is far more often the verb (می ده = gives) than the number
280
+ # after "mi"; the month "May" is caught by the context guard instead.
281
+ if body in _MI_STOP_NEXT or (body in _NUMBER_WORDS and body != "ده"):
282
+ return None
283
+ if prefix + body in _MI_EXCEPTIONS:
284
+ return None
285
+ if body[0] in "اآ": # ALEF, ALEF WITH MADDA
286
+ madda = "آ" + body[1:]
287
+ if _COME_BODY_RE.fullmatch(madda):
288
+ return madda
289
+ if _VERB_BODY_RE.fullmatch(body):
290
+ return body
291
+ if self._count(prefix + ZWNJ + body) >= _MIN_ZWNJ_EVIDENCE:
292
+ return body
293
+ return None
294
+
295
+ def _is_verbish(self, tok: str) -> bool:
296
+ """A verb with the mi/nemi prefix (joined or ZWNJ)."""
297
+ for p in (_NEMI, _MI):
298
+ if tok.startswith(p):
299
+ body = tok[len(p):]
300
+ if body.startswith(ZWNJ):
301
+ return True
302
+ return self._verb_body(p, body) is not None
303
+ return False
304
+
305
+ def _plural_attested(self, stem: str) -> bool:
306
+ for suf in ("ها", "های", "هایی"):
307
+ if self._count(stem + ZWNJ + suf) or self._count(stem + suf):
308
+ return True
309
+ return False
310
+
311
+ # -- per-token rules -------------------------------------------------------
312
+
313
+ def _fix_token(self, tok: str) -> tuple[str, str] | None:
314
+ try:
315
+ return self._tok_cache[tok]
316
+ except KeyError:
317
+ pass
318
+ res = None
319
+ if _FA_WORD_RE.fullmatch(tok):
320
+ res = self._verb_joined(tok) or self._plural_joined(tok)
321
+ self._tok_cache[tok] = res
322
+ return res
323
+
324
+ def _verb_joined(self, tok: str) -> tuple[str, str] | None:
325
+ for p in (_NEMI, _MI):
326
+ if tok.startswith(p):
327
+ if tok in _MI_EXCEPTIONS:
328
+ return None
329
+ body = self._verb_body(p, tok[len(p):])
330
+ if body is not None:
331
+ return p + ZWNJ + body, "verb_prefix_joined"
332
+ return None
333
+ return None
334
+
335
+ def _plural_joined(self, tok: str) -> tuple[str, str] | None:
336
+ if self.lexicon is None or tok in _PLURAL_EXCEPTIONS:
337
+ return None
338
+ m = _PLURAL_SPLIT_RE.match(tok)
339
+ if not m:
340
+ return None
341
+ stem, suf = m.groups()
342
+ # After a non-joining letter the plain join already is the standard.
343
+ if stem[-1] in _NON_JOINING or stem[-1] == ZWNJ:
344
+ return None
345
+ if stem in _PLURAL_STOP_LEFT or self._is_verbish(stem):
346
+ return None
347
+ if self._count(stem) < _MIN_STEM_COUNT:
348
+ return None
349
+ # "خانهای" is خانه+ای (a house), not خان+های; "خانهام" is خانه+ام.
350
+ # For suffixes that can also be read as heh + clitic, skip whenever
351
+ # stem+heh is an attested word. For the others only a typo such as
352
+ # "بچها" (for بچه‌ها) is possible: skip if stem+heh is the commoner.
353
+ heh = self._count(stem + "\u0647") # HEH
354
+ if heh >= (_MIN_HEH_COUNT if suf in _HEH_CLITIC_SUFFIXES else max(1, self._count(stem))):
355
+ return None
356
+ # The heh+clitic reading is also possible for words the corpus never
357
+ # shows bare ("بیشهای" = بیشه‌ای): demand that the plain plural
358
+ # "Xها" or the ZWNJ spelling of this very token is attested.
359
+ # (Impossible after a final heh or yeh: "Xهه" / "Xیه" are not words.)
360
+ if suf in _HEH_CLITIC_SUFFIXES and stem[-1] not in "\u0647\u06cc" and not (
361
+ self._count(stem + "ها") or self._count(stem + ZWNJ + "ها")
362
+ or self._count(stem + ZWNJ + suf)
363
+ ):
364
+ return None
365
+ # "پرداختهام" is a perfect tense (پرداخته+ام), not پرداخت+هام.
366
+ if suf != "ها" and _PAST_TAIL_RE.search(stem):
367
+ return None
368
+ return stem + ZWNJ + suf, "plural_joined"
369
+
370
+ # -- pair rules ------------------------------------------------------------
371
+
372
+ def _merge(self, prev2: str | None, left: str, right: str, last: bool):
373
+ """Decide whether ``left right`` should become one token."""
374
+ key = (left, right, last if right == "ها" else False)
375
+ if left in (_MI, _NEMI):
376
+ # Context-dependent guards (month "May" after a number) first.
377
+ if prev2 is not None and (prev2 in _NUMBER_WORDS or _DIGIT_RE.search(prev2)):
378
+ return None
379
+ try:
380
+ return self._pair_cache[key]
381
+ except KeyError:
382
+ pass
383
+ res = self._merge_uncached(left, right, key[2])
384
+ self._pair_cache[key] = res
385
+ return res
386
+
387
+ def _merge_uncached(self, left: str, right: str, last: bool):
388
+ if left in (_MI, _NEMI):
389
+ body = self._verb_body(left, right)
390
+ if body is not None:
391
+ return left + ZWNJ + body, "verb_prefix_spaced"
392
+ return None
393
+ if right in _PLURAL_SET:
394
+ return self._plural_spaced(left, right, last)
395
+ if right in _COMPARATIVE_SUFFIXES:
396
+ return self._comparative_spaced(left, right)
397
+ return None
398
+
399
+ def _plural_spaced(self, left: str, suf: str, last: bool):
400
+ if len(left) < 2 or not _FA_WORD_RE.fullmatch(left.replace(ZWNJ, "")):
401
+ return None
402
+ if left[-1] == ZWNJ or left in _PLURAL_STOP_LEFT or self._is_verbish(left):
403
+ return None
404
+ if left.endswith(_PLURAL_SUFFIXES):
405
+ return None # already plural (keeps the rewrite idempotent)
406
+ if suf in _AMBIGUOUS_SPACED:
407
+ # Bare "ha" is also a colloquial interjection ("خوبه ها"), and the
408
+ # short possessives are easily mis-split ("واسه هات"). With a
409
+ # lexicon, demand that some plural of X is attested; without one,
410
+ # never merge a line-final "ha".
411
+ if self.lexicon is not None:
412
+ if not self._plural_attested(left):
413
+ return None
414
+ elif suf == "ها" and last:
415
+ return None
416
+ return _glue(left, suf), "plural_spaced"
417
+
418
+ def _comparative_spaced(self, left: str, suf: str):
419
+ if self.lexicon is None or not _FA_WORD_RE.fullmatch(left):
420
+ return None
421
+ if len(left) < 2 or left.endswith("تر") or self._count(left) < _MIN_STEM_COUNT:
422
+ return None
423
+ zw = self._count(left + ZWNJ + "تر") + self._count(left + ZWNJ + "ترین")
424
+ joined_both = self._count(left + "تر") and self._count(left + "ترین")
425
+ if not (zw or joined_both):
426
+ return None
427
+ return _glue(left, suf), "comparative_spaced"
428
+
429
+ # -- public ----------------------------------------------------------------
430
+
431
+ def _run(self, text: str, log: list | None) -> str:
432
+ toks = text.split()
433
+ if not toks:
434
+ return text
435
+ out: list[str] = []
436
+ for i, tok in enumerate(toks):
437
+ if out:
438
+ prev2 = out[-2] if len(out) >= 2 else None
439
+ res = self._merge(prev2, out[-1], tok, i == len(toks) - 1)
440
+ if res is not None:
441
+ if log is not None:
442
+ log.append((f"{out[-1]} {tok}", res[0], res[1]))
443
+ out[-1] = res[0]
444
+ continue
445
+ out.append(tok)
446
+ for i, tok in enumerate(out):
447
+ res = self._fix_token(tok)
448
+ if res is not None:
449
+ if log is not None:
450
+ log.append((tok, res[0], res[1]))
451
+ out[i] = res[0]
452
+ return " ".join(out)
453
+
454
+ def __call__(self, text: str) -> str:
455
+ return self._run(text, None)
456
+
457
+ def explain(self, text: str) -> list[tuple[str, str, str]]:
458
+ """``(before, after, rule_name)`` for every rewrite ``__call__`` makes."""
459
+ log: list[tuple[str, str, str]] = []
460
+ self._run(text, log)
461
+ return log
nimruz/scoring.py ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Scoring text: one canonical spelling for reference and hypothesis.
2
+
3
+ Persian sources disagree on word spacing ("می‌کنم" / "می کنم" / "میکنم", "آن‌ها" / "آنها").
4
+ Scoring those as errors measures the convention, not the recognition, so both sides are
5
+ first rewritten to the standard forms with the same rules used on the training text,
6
+ then folded by `normalize_for_scoring`.
7
+ """
8
+
9
+ from __future__ import annotations
10
+
11
+ import os
12
+ from functools import lru_cache
13
+ from pathlib import Path
14
+
15
+ from nimruz.ortho import SpacingFixer, load_lexicon
16
+ from nimruz.textnorm import normalize_for_scoring, normalize_for_training
17
+
18
+ LEXICON_PATHS = (
19
+ os.environ.get("NIMRUZ_LEXICON", ""),
20
+ str(Path(__file__).resolve().parent.parent / "assets" / "lexicon.json"),
21
+ "/data/train/lexicon.json",
22
+ )
23
+
24
+
25
+ @lru_cache(maxsize=1)
26
+ def fixer() -> SpacingFixer:
27
+ for p in LEXICON_PATHS:
28
+ if p and Path(p).exists():
29
+ return SpacingFixer(load_lexicon(p))
30
+ return SpacingFixer(None)
31
+
32
+
33
+ def canonical(text: str) -> str:
34
+ """Canonical scoring form: standard spacing, folded letters, spelled-out numbers, no punctuation."""
35
+ return normalize_for_scoring(fixer()(normalize_for_training(text or "", keep_punct=False)))
36
+
37
+
38
+ def error_rates(refs: list[str], hyps: list[str]) -> dict:
39
+ """WER / CER on canonical text, CER ignoring spaces, and WER without the spacing rules."""
40
+ import jiwer
41
+
42
+ pairs = [(canonical(r), canonical(h), normalize_for_scoring(r), normalize_for_scoring(h)) for r, h in zip(refs, hyps)]
43
+ pairs = [p for p in pairs if p[0]]
44
+ r, h = [p[0] for p in pairs], [p[1] for p in pairs]
45
+ return {
46
+ "n": len(pairs),
47
+ "wer": 100 * jiwer.wer(r, h),
48
+ "cer": 100 * jiwer.cer(r, h),
49
+ "ncer": 100 * jiwer.cer([x.replace(" ", "") for x in r], [x.replace(" ", "") or "-" for x in h]),
50
+ "wer_raw": 100 * jiwer.wer([p[2] for p in pairs], [p[3] for p in pairs]),
51
+ }
nimruz/textnorm.py ADDED
@@ -0,0 +1,465 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Persian (Farsi) text normalisation for speech recognition.
2
+
3
+ Two normalisers share a common base cleanup:
4
+
5
+ * ``normalize_for_scoring`` -- aggressive folding applied to both reference and
6
+ hypothesis before WER/CER, so that spellings a Persian reader considers the
7
+ same become identical strings.
8
+ * ``normalize_for_training`` -- conservative cleanup that keeps correct
9
+ orthography (hamza forms, ZWNJ, digits) for training targets.
10
+
11
+ Standard library only; all regexes and translation tables are built once at
12
+ import time (no I/O). Invisible or easily-confused characters are written as
13
+ ``\\uXXXX`` escapes with a comment naming them.
14
+ """
15
+
16
+ from __future__ import annotations
17
+
18
+ import re
19
+ import unicodedata
20
+
21
+ __all__ = [
22
+ "normalize_for_scoring",
23
+ "normalize_for_training",
24
+ "int_to_fa_words",
25
+ "spell_out_numbers",
26
+ "text_stats",
27
+ ]
28
+
29
+ ZWNJ = "\u200c" # ZERO WIDTH NON-JOINER (nim-fasele)
30
+
31
+ # ---------------------------------------------------------------------------
32
+ # Character classes (strings to be placed inside a regex [...] class)
33
+ # ---------------------------------------------------------------------------
34
+
35
+ # Arabic-script letters. U+0640 (tatweel) and U+064B-U+065F (harakat) are
36
+ # deliberately outside these ranges.
37
+ _AR_LETTERS = (
38
+ "\u0620-\u063f" # Arabic letters incl. standalone hamza U+0621
39
+ "\u0641-\u064a" # feh .. Arabic yeh U+064A
40
+ "\u066e\u066f" # dotless beh / qaf
41
+ "\u0671-\u06d3" # alef wasla .. yeh barree (incl. Persian peh/tcheh/jeh/gaf/kaf U+06A9/yeh U+06CC)
42
+ "\u06d5" # Arabic ae
43
+ "\u06ee\u06ef"
44
+ "\u06fa-\u06fc\u06ff"
45
+ "\u0750-\u077f" # Arabic Supplement letters
46
+ )
47
+ # For stats on raw text: also count presentation-form letters.
48
+ _AR_LETTERS_RAW = _AR_LETTERS + "\ufb50-\ufd3d\ufd50-\ufdfb\ufe70-\ufefc"
49
+ _LATIN_LETTERS = "A-Za-z\u00c0-\u00d6\u00d8-\u00f6\u00f8-\u024f" # excludes x U+00D7, / U+00F7
50
+ _ANY_DIGIT = "0-9\u0660-\u0669\u06f0-\u06f9" # ASCII, Arabic-Indic, Extended Arabic-Indic (Persian)
51
+ _MUSIC = "\u2669\u266a\u266b\u266c\U0001f3b5\U0001f3b6\U0001f3bc" # music notes / emoji
52
+
53
+
54
+ def _char_class(codepoints) -> str:
55
+ """Compress code points into a regex character-class body."""
56
+ cps = sorted(set(codepoints))
57
+ out = []
58
+ i = 0
59
+ while i < len(cps):
60
+ j = i
61
+ while j + 1 < len(cps) and cps[j + 1] == cps[j] + 1:
62
+ j += 1
63
+ a, b = re.escape(chr(cps[i])), re.escape(chr(cps[j]))
64
+ out.append(a if i == j else f"{a}-{b}")
65
+ i = j + 1
66
+ return "".join(out)
67
+
68
+
69
+ def _junk_codepoints():
70
+ """Punctuation, symbols and control/format chars (not whitespace, not ZWNJ)."""
71
+ scan = (
72
+ range(0x0000, 0x10000),
73
+ range(0x1D100, 0x1D200), # musical symbols
74
+ range(0x1F000, 0x1FB00), # emoji, pictographs, regional indicators
75
+ range(0xE0000, 0xE0080), # tag characters (emoji flags)
76
+ )
77
+ for r in scan:
78
+ for cp in r:
79
+ if 0xD800 <= cp <= 0xDFFF:
80
+ continue
81
+ ch = chr(cp)
82
+ if ch.isspace() or ch == ZWNJ:
83
+ continue
84
+ cat = unicodedata.category(ch)
85
+ if cat[0] in "PS" or cat in ("Cc", "Cf"):
86
+ yield cp
87
+ yield from range(0xFE00, 0xFE10) # variation selectors (emoji presentation)
88
+ yield from range(0x20D0, 0x2100) # combining marks for symbols (keycap U+20E3)
89
+
90
+
91
+ _JUNK = set(_junk_codepoints())
92
+ _KEEP_PUNCT = "\u060c\u061b\u061f.!:" # Arabic comma, Arabic semicolon, Arabic question mark, . ! :
93
+ _JUNK_RE = re.compile(f"[{_char_class(_JUNK)}]+")
94
+ _JUNK_EXCEPT_KEPT_RE = re.compile(
95
+ f"[{_char_class(_JUNK - {ord(c) for c in _KEEP_PUNCT})}]+"
96
+ )
97
+
98
+ # ---------------------------------------------------------------------------
99
+ # Base cleanup
100
+ # ---------------------------------------------------------------------------
101
+
102
+ # Arabic presentation forms A (U+FB50-U+FDFF) and B (U+FE70-U+FEFC): NFKC only these.
103
+ _PRES_RE = re.compile("[\ufb50-\ufdff\ufe70-\ufefc]+")
104
+
105
+
106
+ def _nfkc(m: re.Match) -> str:
107
+ return unicodedata.normalize("NFKC", m.group())
108
+
109
+
110
+ # Decomposed hamza / madda sequences -> precomposed (or plain heh).
111
+ _COMPOSE_MAP = {
112
+ "\u064a\u0654": "\u0626", # Arabic yeh + HAMZA ABOVE -> yeh with hamza
113
+ "\u06cc\u0654": "\u0626", # Persian yeh + HAMZA ABOVE -> yeh with hamza
114
+ "\u0649\u0654": "\u0626", # alef maksura + HAMZA ABOVE -> yeh with hamza
115
+ "\u0648\u0654": "\u0624", # waw + HAMZA ABOVE -> waw with hamza
116
+ "\u0627\u0653": "\u0622", # alef + MADDA ABOVE -> alef madda
117
+ "\u0627\u0654": "\u0623", # alef + HAMZA ABOVE -> alef hamza above
118
+ "\u0627\u0655": "\u0625", # alef + HAMZA BELOW -> alef hamza below
119
+ "\u0647\u0654": "\u0647", # heh + HAMZA ABOVE (ezafe) -> heh
120
+ "\u06d5\u0654": "\u0647", # ae + HAMZA ABOVE -> heh
121
+ }
122
+ _COMPOSE_RE = re.compile("|".join(_COMPOSE_MAP))
123
+
124
+ _BASE_TABLE: dict[int, str | None] = {
125
+ 0x064A: "\u06cc", # ARABIC YEH -> PERSIAN YEH
126
+ 0x0649: "\u06cc", # ALEF MAKSURA -> PERSIAN YEH
127
+ 0x0643: "\u06a9", # ARABIC KAF -> PERSIAN KEHEH
128
+ 0x06D5: "\u0647", # ARABIC AE -> HEH
129
+ 0x06C0: "\u0647", # HEH WITH YEH ABOVE (heh + hamza) -> HEH
130
+ 0x0640: None, # TATWEEL
131
+ 0x0670: None, # SUPERSCRIPT ALEF
132
+ 0x00AC: ZWNJ, # NOT SIGN, a common ZWNJ substitute
133
+ 0x00AD: None, # SOFT HYPHEN
134
+ 0x061C: None, # ARABIC LETTER MARK
135
+ 0x200B: None, # ZERO WIDTH SPACE
136
+ 0x200D: None, # ZERO WIDTH JOINER
137
+ 0x200E: None, # LEFT-TO-RIGHT MARK
138
+ 0x200F: None, # RIGHT-TO-LEFT MARK
139
+ 0x2060: None, # WORD JOINER
140
+ 0xFEFF: None, # BYTE ORDER MARK / ZWNBSP
141
+ }
142
+ for _cp in range(0x064B, 0x0660): # harakat, shadda, sukun, madda, hamza marks ...
143
+ _BASE_TABLE[_cp] = None
144
+ for _cp in (*range(0x0610, 0x061B), *range(0x06D6, 0x06DD), *range(0x06DF, 0x06E9),
145
+ *range(0x06EA, 0x06EE)): # Quranic annotation marks
146
+ _BASE_TABLE[_cp] = None
147
+ for _cp in (*range(0x202A, 0x202F), *range(0x2066, 0x206A)): # bidi embeddings/isolates
148
+ _BASE_TABLE[_cp] = None
149
+ for _cp in range(0xE000, 0xE004): # private-use code points we use as placeholders
150
+ _BASE_TABLE[_cp] = None
151
+ for _i in range(10): # all digit systems -> ASCII (per-mode handling later)
152
+ _BASE_TABLE[0x0660 + _i] = str(_i) # Arabic-Indic
153
+ _BASE_TABLE[0x06F0 + _i] = str(_i) # Extended Arabic-Indic (Persian)
154
+ for _cp in range(0x3001): # NBSP, thin/em/en spaces, etc. -> plain space
155
+ _ch = chr(_cp)
156
+ if _ch.isspace() and _ch not in " \t\n\r\x0b\x0c":
157
+ _BASE_TABLE[_cp] = "\n" if _ch in "\x85\u2028\u2029" else " "
158
+
159
+
160
+ def _base_cleanup(text: str) -> str:
161
+ text = _PRES_RE.sub(_nfkc, text)
162
+ text = _COMPOSE_RE.sub(lambda m: _COMPOSE_MAP[m.group()], text)
163
+ return text.translate(_BASE_TABLE)
164
+
165
+
166
+ # ---------------------------------------------------------------------------
167
+ # Non-speech tags
168
+ # ---------------------------------------------------------------------------
169
+
170
+ _BRACKET_TAG_RE = re.compile(r"\[[^\[\]\n]*\]")
171
+ _PAREN_RE = re.compile(r"\(([^()\n]*)\)")
172
+ _ANY_DIGIT_RE = re.compile(f"[{_ANY_DIGIT}]")
173
+ _MUSIC_SPAN_RE = re.compile(f"[{_MUSIC}][^\\n]*[{_MUSIC}]")
174
+ _MUSIC_CHAR_RE = re.compile(f"[{_MUSIC}]")
175
+ _SPEAKER_DASH_RE = re.compile(r"(?m)^[ \t]*[-\u2013\u2014]+(?=[ \t]|$)") # hyphen, en dash, em dash
176
+ _CHEVRON_RE = re.compile(r">{2,}")
177
+ _PAREN_TAG_MAX_WORDS = 3
178
+
179
+
180
+ def _is_paren_tag(inner: str) -> bool:
181
+ return len(inner.split()) <= _PAREN_TAG_MAX_WORDS and not _ANY_DIGIT_RE.search(inner)
182
+
183
+
184
+ def _paren_sub(m: re.Match) -> str:
185
+ return " " if _is_paren_tag(m.group(1)) else m.group()
186
+
187
+
188
+ def _remove_tags(text: str, parens: bool = True) -> str:
189
+ text = _BRACKET_TAG_RE.sub(" ", text)
190
+ if parens:
191
+ text = _PAREN_RE.sub(_paren_sub, text)
192
+ text = _MUSIC_SPAN_RE.sub(" ", text)
193
+ text = _SPEAKER_DASH_RE.sub(" ", text)
194
+ return _CHEVRON_RE.sub(" ", text)
195
+
196
+
197
+ # ---------------------------------------------------------------------------
198
+ # Numbers
199
+ # ---------------------------------------------------------------------------
200
+
201
+ # NOTE: every yeh below is PERSIAN YEH U+06CC and every kaf is KEHEH U+06A9
202
+ # (enforced by a test).
203
+ _ONES = ("", "یک", "دو", "سه", "چهار", "پنج", "شش", "هفت", "هشت", "نه")
204
+ _TEENS = ("ده", "یازده", "دوازده", "سیزده", "چهارده", "پانزده", "شانزده", "هفده", "هجده", "نوزده")
205
+ _TENS = ("", "", "بیست", "سی", "چهل", "پنجاه", "شصت", "هفتاد", "هشتاد", "نود")
206
+ _HUNDREDS = ("", "صد", "دویست", "سیصد", "چهارصد", "پانصد", "ششصد", "هفتصد", "هشتصد", "نهصد")
207
+ _SCALES = ("", "هزار", "میلیون", "میلیارد", "تریلیون")
208
+ _ZERO = "صفر"
209
+ _NEG = "منفی"
210
+ _POINT = "ممیز"
211
+ _PERCENT = "درصد"
212
+ _AND = " \u0648 "
213
+ _DIGIT_WORDS = (_ZERO,) + _ONES[1:]
214
+ _MAX_SPELL_DIGITS = 15
215
+
216
+
217
+ def _three(n: int) -> str:
218
+ parts = []
219
+ h, r = divmod(n, 100)
220
+ if h:
221
+ parts.append(_HUNDREDS[h])
222
+ if 10 <= r < 20:
223
+ parts.append(_TEENS[r - 10])
224
+ else:
225
+ t, o = divmod(r, 10)
226
+ if t:
227
+ parts.append(_TENS[t])
228
+ if o:
229
+ parts.append(_ONES[o])
230
+ return _AND.join(parts)
231
+
232
+
233
+ def int_to_fa_words(n: int) -> str:
234
+ """Persian cardinal words for ``-10**15 < n < 10**15``."""
235
+ if isinstance(n, bool) or not isinstance(n, int):
236
+ raise TypeError(f"expected int, got {type(n).__name__}")
237
+ if n < 0:
238
+ return f"{_NEG} {int_to_fa_words(-n)}"
239
+ if n >= 10**_MAX_SPELL_DIGITS:
240
+ raise ValueError("n must be < 10**15")
241
+ if n == 0:
242
+ return _ZERO
243
+ parts = []
244
+ idx = 0
245
+ while n:
246
+ n, g = divmod(n, 1000)
247
+ if g:
248
+ if idx == 1 and g == 1:
249
+ parts.append(_SCALES[1]) # 1000 -> "hezar", not "yek hezar"
250
+ elif idx:
251
+ parts.append(f"{_three(g)} {_SCALES[idx]}")
252
+ else:
253
+ parts.append(_three(g))
254
+ idx += 1
255
+ return _AND.join(reversed(parts))
256
+
257
+
258
+ _DIGITS_TO_ASCII = {0x0660 + i: str(i) for i in range(10)}
259
+ _DIGITS_TO_ASCII.update({0x06F0 + i: str(i) for i in range(10)})
260
+ _ASCII_TO_FA_DIGITS = {ord(str(i)): chr(0x06F0 + i) for i in range(10)}
261
+
262
+ _THOUSANDS_SEPS = ",\u066c" # comma, ARABIC THOUSANDS SEPARATOR
263
+ _DECIMAL_SEPS = ".\u066b/" # full stop, ARABIC DECIMAL SEPARATOR, slash
264
+ _NUM_CHAIN_RE = re.compile(f"[0-9]+(?:[{_THOUSANDS_SEPS}{_DECIMAL_SEPS}][0-9]+)*")
265
+ _NUM_SPLIT_RE = re.compile(f"([{_THOUSANDS_SEPS}{_DECIMAL_SEPS}])")
266
+
267
+
268
+ def _read_digits(s: str) -> str:
269
+ """Integer words, or digit-by-digit for leading zeros / very long runs."""
270
+ if len(s) > _MAX_SPELL_DIGITS or (len(s) > 1 and s[0] == "0"):
271
+ return " ".join(_DIGIT_WORDS[ord(c) - 48] for c in s)
272
+ return int_to_fa_words(int(s))
273
+
274
+
275
+ def _spell_chain(chain: str) -> str:
276
+ if chain.isdigit():
277
+ return _read_digits(chain)
278
+ parts = _NUM_SPLIT_RE.split(chain)
279
+ groups, seps = parts[0::2], parts[1::2]
280
+ has_dec = seps[-1] in _DECIMAL_SEPS
281
+ int_seps = seps[:-1] if has_dec else seps
282
+ n_int = len(int_seps) + 1
283
+ if all(s in _THOUSANDS_SEPS for s in int_seps) and (
284
+ not int_seps
285
+ or (len(groups[0]) <= 3 and groups[0][0] != "0"
286
+ and all(len(g) == 3 for g in groups[1:n_int]))
287
+ ):
288
+ words = _read_digits("".join(groups[:n_int]))
289
+ if has_dec:
290
+ words = f"{words} {_POINT} {_read_digits(groups[-1])}"
291
+ return words
292
+ # Not a well-formed number (dates, versions, lists): read each group.
293
+ out = [_read_digits(groups[0])]
294
+ for sep, g in zip(seps, groups[1:]):
295
+ out.append(sep)
296
+ out.append(_read_digits(g))
297
+ return " ".join(out)
298
+
299
+
300
+ def _spell_sub(m: re.Match) -> str:
301
+ s = m.string
302
+ words = _spell_chain(m.group())
303
+ if m.start() > 0 and s[m.start() - 1].isalnum():
304
+ words = " " + words
305
+ if m.end() < len(s) and s[m.end()].isalnum():
306
+ words += " "
307
+ return words
308
+
309
+
310
+ def spell_out_numbers(text: str) -> str:
311
+ """Replace every digit run (any digit system) with Persian words."""
312
+ text = text.translate(_DIGITS_TO_ASCII)
313
+ return _NUM_CHAIN_RE.sub(_spell_sub, text)
314
+
315
+
316
+ # ---------------------------------------------------------------------------
317
+ # Scoring
318
+ # ---------------------------------------------------------------------------
319
+
320
+ _SCORING_TABLE: dict[int, str | None] = {
321
+ 0x0623: "\u0627", # ALEF WITH HAMZA ABOVE -> ALEF
322
+ 0x0625: "\u0627", # ALEF WITH HAMZA BELOW -> ALEF
323
+ 0x0671: "\u0627", # ALEF WASLA -> ALEF
324
+ 0x0672: "\u0627", # ALEF WITH WAVY HAMZA ABOVE -> ALEF
325
+ 0x0673: "\u0627", # ALEF WITH WAVY HAMZA BELOW -> ALEF
326
+ 0x0624: "\u0648", # WAW WITH HAMZA ABOVE -> WAW
327
+ 0x0626: "\u06cc", # YEH WITH HAMZA ABOVE -> PERSIAN YEH
328
+ 0x0629: "\u0647", # TEH MARBUTA -> HEH
329
+ 0x0621: None, # standalone HAMZA
330
+ ord(ZWNJ): " ",
331
+ ord("%"): f" {_PERCENT} ",
332
+ 0x066A: f" {_PERCENT} ", # ARABIC PERCENT SIGN
333
+ }
334
+ # Anything that is not a Latin/Arabic-script letter or a space is dropped.
335
+ _SCORING_DROP_RE = re.compile(f"[^{_LATIN_LETTERS}{_AR_LETTERS} ]+")
336
+
337
+
338
+ def normalize_for_scoring(text: str) -> str:
339
+ """Aggressive normalisation for WER/CER (apply to reference and hypothesis)."""
340
+ if not text:
341
+ return ""
342
+ text = _base_cleanup(text)
343
+ # [..] tags and music spans are never speech; parentheses may be, so in
344
+ # scoring they are treated as plain punctuation (content kept).
345
+ text = _remove_tags(text, parens=False)
346
+ text = text.translate(_SCORING_TABLE)
347
+ text = spell_out_numbers(text)
348
+ text = text.lower()
349
+ text = _SCORING_DROP_RE.sub(" ", text)
350
+ return " ".join(text.split())
351
+
352
+
353
+ # ---------------------------------------------------------------------------
354
+ # Training
355
+ # ---------------------------------------------------------------------------
356
+
357
+ # "1,234,567" -> "1234567" (only well-formed 3-digit groups).
358
+ _THOUSANDS_RE = re.compile(f"(?<=[0-9])[{_THOUSANDS_SEPS}](?=[0-9]{{3}}(?![0-9]))")
359
+ # Number-internal marks are protected from punctuation removal via private-use placeholders.
360
+ _PROTECT_RE = re.compile(r"(?<=[0-9])(?:([.\u066b/:])(?=[0-9])|[ \t]*[%\u066a])")
361
+ _PROTECT_MAP = {".": "\ue000", "\u066b": "\ue000", "/": "\ue001", ":": "\ue002"}
362
+ _RESTORE_TABLE = {
363
+ 0xE000: "\u066b", # ARABIC DECIMAL SEPARATOR
364
+ 0xE001: "/",
365
+ 0xE002: ":",
366
+ 0xE003: "\u066a", # ARABIC PERCENT SIGN
367
+ }
368
+ _LATIN_TO_FA_PUNCT = {
369
+ ord(","): "\u060c", # ARABIC COMMA
370
+ ord(";"): "\u061b", # ARABIC SEMICOLON
371
+ ord("?"): "\u061f", # ARABIC QUESTION MARK
372
+ }
373
+ _ZWNJ_RUN_RE = re.compile(f"{ZWNJ}{{2,}}")
374
+ # ZWNJ is meaningful only between two Arabic-script letters.
375
+ _ZWNJ_STRAY_RE = re.compile(f"(?<![{_AR_LETTERS}]){ZWNJ}|{ZWNJ}(?![{_AR_LETTERS}])")
376
+ _PUNCT_RUN_RE = re.compile(f"\\s*([{re.escape(_KEEP_PUNCT)}]+)\\s*")
377
+ _REPEAT_CHAR_RE = re.compile(r"(.)\1+")
378
+ _LEADING_PUNCT_RE = re.compile(f"^[{re.escape(_KEEP_PUNCT)}\\s]+")
379
+
380
+
381
+ def _protect_sub(m: re.Match) -> str:
382
+ sep = m.group(1)
383
+ return _PROTECT_MAP[sep] if sep else "\ue003"
384
+
385
+
386
+ def _punct_run_sub(m: re.Match) -> str:
387
+ return _REPEAT_CHAR_RE.sub(r"\1", m.group(1)) + " "
388
+
389
+
390
+ def normalize_for_training(text: str, keep_punct: bool = False) -> str:
391
+ """Clean, orthographically consistent Persian target text."""
392
+ if not text:
393
+ return ""
394
+ text = _base_cleanup(text)
395
+ text = _remove_tags(text)
396
+ text = _THOUSANDS_RE.sub("", text)
397
+ text = _PROTECT_RE.sub(_protect_sub, text)
398
+ if keep_punct:
399
+ text = text.translate(_LATIN_TO_FA_PUNCT)
400
+ text = _JUNK_EXCEPT_KEPT_RE.sub(" ", text)
401
+ else:
402
+ text = _JUNK_RE.sub(" ", text)
403
+ text = _ZWNJ_RUN_RE.sub(ZWNJ, text)
404
+ text = _ZWNJ_STRAY_RE.sub("", text)
405
+ text = " ".join(text.split())
406
+ if keep_punct:
407
+ text = _PUNCT_RUN_RE.sub(_punct_run_sub, text).strip()
408
+ text = _LEADING_PUNCT_RE.sub("", text)
409
+ text = text.translate(_RESTORE_TABLE)
410
+ return text.translate(_ASCII_TO_FA_DIGITS)
411
+
412
+
413
+ # ---------------------------------------------------------------------------
414
+ # Stats (raw text, for data filtering)
415
+ # ---------------------------------------------------------------------------
416
+
417
+ _WS_RE = re.compile(r"\s+")
418
+ _AR_LETTER_RAW_RE = re.compile(f"[{_AR_LETTERS_RAW}]+")
419
+ _LATIN_LETTER_RE = re.compile(f"[{_LATIN_LETTERS}]+")
420
+ _URL_HANDLE_RE = re.compile(
421
+ r"(?:https?://|www\.)\S+"
422
+ r"|(?<![\w@])@[A-Za-z0-9_.]{2,}"
423
+ r"|(?<!\w)#[^\s#]+"
424
+ r"|\b[A-Za-z0-9-]+\.(?:com|ir|org|net|io|info|me|co|tv|app|dev|xyz)\b",
425
+ re.IGNORECASE,
426
+ )
427
+ _CHAR_RUN_RE = re.compile(r"(\S)\1*")
428
+
429
+
430
+ def _count(regex: re.Pattern, text: str) -> int:
431
+ return sum(m.end() - m.start() for m in regex.finditer(text))
432
+
433
+
434
+ def text_stats(text: str) -> dict:
435
+ """Cheap statistics on the raw text for filtering training data."""
436
+ n_nonspace = len(_WS_RE.sub("", text))
437
+ n_ar = _count(_AR_LETTER_RAW_RE, text)
438
+ n_lat = _count(_LATIN_LETTER_RE, text)
439
+ max_run = max((m.end() - m.start() for m in _CHAR_RUN_RE.finditer(text)), default=0)
440
+
441
+ max_rep = 0
442
+ prev, run = None, 0
443
+ for tok in text.split():
444
+ key = _JUNK_RE.sub("", tok).replace(ZWNJ, "").casefold()
445
+ if not key:
446
+ continue
447
+ run = run + 1 if key == prev else 1
448
+ prev = key
449
+ max_rep = max(max_rep, run)
450
+
451
+ has_tag = bool(_BRACKET_TAG_RE.search(text)) or any(
452
+ _is_paren_tag(m.group(1)) for m in _PAREN_RE.finditer(text)
453
+ )
454
+ return {
455
+ "n_chars": len(text),
456
+ "n_words": len(text.split()),
457
+ "persian_letter_ratio": n_ar / n_nonspace if n_nonspace else 0.0,
458
+ "latin_letter_ratio": n_lat / n_nonspace if n_nonspace else 0.0,
459
+ "n_digits": len(_ANY_DIGIT_RE.findall(text)),
460
+ "has_bracket_tag": has_tag,
461
+ "has_music_symbol": bool(_MUSIC_CHAR_RE.search(text)),
462
+ "has_url_or_handle": bool(_URL_HANDLE_RE.search(text)),
463
+ "max_char_run": max_run,
464
+ "max_word_repeat": max_rep,
465
+ }
score.py ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Score saved hypotheses: results/<model>/<testset>.jsonl -> error-rate table.
2
+
3
+ uv run --no-project --with jiwer python score.py [results_dir] [--show N]
4
+
5
+ Columns per test set: WER / CER / CER ignoring spaces (all on canonical spelling, see
6
+ nimruz/scoring.py), then in brackets the WER without the word-spacing rules.
7
+ """
8
+
9
+ from __future__ import annotations
10
+
11
+ import json
12
+ import sys
13
+ from pathlib import Path
14
+
15
+ import jiwer
16
+
17
+ from nimruz.scoring import canonical, error_rates
18
+
19
+
20
+ def main():
21
+ args = [a for a in sys.argv[1:] if not a.startswith("--")]
22
+ show = int(sys.argv[sys.argv.index("--show") + 1]) if "--show" in sys.argv else 0
23
+ root = Path(args[0] if args else "results")
24
+ table: dict[str, dict[str, dict]] = {}
25
+ rows_of = {}
26
+ for f in sorted(root.glob("*/*.jsonl")):
27
+ rows = [json.loads(l) for l in open(f, encoding="utf-8")]
28
+ table.setdefault(f.parent.name, {})[f.stem] = error_rates([r["ref"] for r in rows], [r["hyp"] for r in rows])
29
+ rows_of[(f.parent.name, f.stem)] = rows
30
+ sets = sorted({s for m in table.values() for s in m})
31
+ print("| model | " + " | ".join(sets) + " |")
32
+ print("|---|" + "---|" * len(sets))
33
+ key = lambda kv: sum(v["wer"] for v in kv[1].values()) / len(kv[1])
34
+ for model, res in sorted(table.items(), key=key):
35
+ cells = [f"{res[s]['wer']:.1f} / {res[s]['cer']:.1f} / {res[s]['ncer']:.1f} [{res[s]['wer_raw']:.1f}]" if s in res else "-" for s in sets]
36
+ print(f"| {model.replace('__', '/')} | " + " | ".join(cells) + " |")
37
+ if show:
38
+ for (model, s), rows in rows_of.items():
39
+ print(f"\n## {model} / {s}: worst {show}")
40
+ per = sorted(((jiwer.wer(canonical(r["ref"]) or "-", canonical(r["hyp"])), canonical(r["ref"]), canonical(r["hyp"])) for r in rows), reverse=True)[:show]
41
+ for e, a, b in per:
42
+ print(f"{e:.2f}\n ref: {a}\n hyp: {b}")
43
+
44
+
45
+ if __name__ == "__main__":
46
+ main()
transcribe.py ADDED
@@ -0,0 +1,96 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Transcribe Persian audio of any length with NAVA.
2
+
3
+ pip install torch transformers
4
+ python transcribe.py audio.mp3
5
+ python transcribe.py audio.mp3 --digits fa # numbers, times and dates in Persian digits
6
+ python transcribe.py audio.mp3 --digits ascii # ... in 0-9
7
+ python transcribe.py audio.mp3 --beams 5 # slightly more accurate, about half the speed
8
+
9
+ Audio is decoded with ffmpeg if it is installed, otherwise with librosa (`pip install librosa`).
10
+
11
+ NAVA transcribes one window of up to 30 seconds at a time and does not produce timestamps,
12
+ so Whisper's built-in long-form mode must not be used with it. This script cuts longer audio
13
+ at quiet points into pieces of at most 29 seconds and joins the text.
14
+ """
15
+
16
+ from __future__ import annotations
17
+
18
+ import argparse
19
+ import shutil
20
+ import subprocess
21
+
22
+ import numpy as np
23
+ import torch
24
+
25
+ SR = 16000
26
+
27
+
28
+ def load_audio(path: str) -> np.ndarray:
29
+ """Any audio file -> mono float32 at 16 kHz."""
30
+ if shutil.which("ffmpeg"):
31
+ raw = subprocess.run(
32
+ ["ffmpeg", "-v", "error", "-i", path, "-f", "f32le", "-ac", "1", "-ar", str(SR), "pipe:1"],
33
+ capture_output=True, check=True,
34
+ ).stdout
35
+ return np.frombuffer(raw, dtype=np.float32).copy()
36
+ import librosa
37
+
38
+ return librosa.load(path, sr=SR, mono=True)[0].astype(np.float32)
39
+
40
+
41
+ def split_long(wav: np.ndarray, max_s: float = 29.0, min_s: float = 12.0) -> list[np.ndarray]:
42
+ """Pieces of at most `max_s` seconds, each cut at the quietest 0.3 s between `min_s` and `max_s`
43
+ after the previous cut."""
44
+ if len(wav) <= int(30.0 * SR):
45
+ return [wav]
46
+ hop, win = int(0.02 * SR), int(0.3 * SR)
47
+ energy = np.convolve(wav.astype(np.float32) ** 2, np.ones(win, dtype=np.float32) / win, mode="same")[::hop]
48
+ pieces, start = [], 0
49
+ while len(wav) - start > int(max_s * SR):
50
+ lo, hi = (start + int(min_s * SR)) // hop, (start + int(max_s * SR)) // hop
51
+ cut = (lo + int(np.argmin(energy[lo:hi]))) * hop
52
+ pieces.append(wav[start:cut])
53
+ start = cut
54
+ pieces.append(wav[start:])
55
+ return pieces
56
+
57
+
58
+ @torch.inference_mode()
59
+ def transcribe(wav: np.ndarray, model, processor, beams: int = 1, batch_size: int = 8) -> str:
60
+ pieces = [p for p in split_long(wav) if len(p) > 0.1 * SR]
61
+ texts = []
62
+ for i in range(0, len(pieces), batch_size):
63
+ feats = processor.feature_extractor(pieces[i : i + batch_size], sampling_rate=SR, return_tensors="pt").input_features
64
+ ids = model.generate(feats.to(model.device, model.dtype), language="fa", task="transcribe",
65
+ num_beams=beams, max_new_tokens=440)
66
+ texts += [t.strip() for t in processor.batch_decode(ids, skip_special_tokens=True)]
67
+ return " ".join(t for t in texts if t)
68
+
69
+
70
+ def main():
71
+ from transformers import WhisperForConditionalGeneration, WhisperProcessor
72
+
73
+ ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
74
+ ap.add_argument("audio", nargs="+")
75
+ ap.add_argument("--model", default="Nimruz/NAVA")
76
+ ap.add_argument("--digits", choices=["none", "fa", "ascii"], default="none",
77
+ help="rewrite spoken numbers, times and dates in digits")
78
+ ap.add_argument("--beams", type=int, default=1)
79
+ args = ap.parse_args()
80
+
81
+ device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
82
+ dtype = torch.float16 if device == "cuda" else torch.float32
83
+ processor = WhisperProcessor.from_pretrained(args.model)
84
+ model = WhisperForConditionalGeneration.from_pretrained(args.model).to(device=device, dtype=dtype).eval()
85
+
86
+ for path in args.audio:
87
+ text = transcribe(load_audio(path), model, processor, beams=args.beams)
88
+ if args.digits != "none":
89
+ from nimruz.itn import itn # the nimruz/ folder next to this script
90
+
91
+ text = itn(text, digits=args.digits)
92
+ print(text)
93
+
94
+
95
+ if __name__ == "__main__":
96
+ main()