Training the app's Quran recitation model

A small speech model built only for Quran recitation: it hears a reciter and writes the Arabic letters (no harakat). The app uses it to follow your recitation and mark mistakes.

Model: Conformer with a CTC head, trained from scratch in PyTorch: 6 layers, pilot ~~10M parameters (~~10 MB), full ~~16M (~~16 MB). Deeper (16 layers) or wider (512) versions got stuck predicting the same letters for every clip; see "Editing the config" before enlarging it.
Output: the 29 Arabic letters plus a word separator. That is exactly what the tracker and the mistake hints compare, and the small alphabet keeps the model small and accurate.
Data: per-ayah recordings from everyayah.com.
- Hafs reciters are labelled with the Hafs Uthmani text.
- The 3 Warsh reciters are labelled with the Warsh Uthmani text. everyayah numbers even the Warsh files by Hafs ayahs, so the Warsh text is aligned to them word by word.
- Warsh clips are sampled more often (warsh_weight).
Long ayahs included: recordings up to 7.5 minutes (max_duration), e.g. al-Baqara 282 or a mujawwad Ayat al-Kursi. Training uses a memory-efficient attention kernel, so a 7-minute ayah needs ~4 GB (pilot) / ~7 GB (full) of GPU memory instead of running out.
Phone robustness: every clip is randomly degraded while training: room echo, noise, phone-like band limits, volume, speed and spectrogram masking.
Any reciting speed: half the clips are also made faster or slower at the same pitch (0.75x to 2.5x, tempo_range), from a slow tartil to a quick review (hadr).
On the phone: the exported model runs in the app's existing engine (sherpa-onnx). The training features are the exact Kaldi filterbank sherpa computes. export.py checks that sherpa and PyTorch produce the same text before copying the model into the app.

1. Set up (once)

From this folder (training/), in PowerShell:

python -m venv .venv
.venv\Scripts\activate
pip install torch --index-url https://download.pytorch.org/whl/cu130
pip install -r requirements.txt
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

The last line should print True NVIDIA GeForce RTX 5070 Ti Laptop GPU.

2. Smoke test (about 10 minutes)

A tiny model on a few surahs, to check everything is wired up:

python download.py --config config/smoke.yaml
python prepare.py  --config config/smoke.yaml
python train.py    --config config/smoke.yaml
python export.py   --config config/smoke.yaml

export.py should end with parity ... identical text in 10/10 (or close to it). The smoke model is copied into runs/smoke/app_assets, not into the app.

3. Pilot run

python download.py --config config/pilot.yaml     # ~17 reciters, ~15–20 GB, resumable
python prepare.py  --config config/pilot.yaml     # labels + splits; prints hours per reciter
python train.py    --config config/pilot.yaml     # prints an ETA; pause/resume any time (see below)
python evaluate.py --config config/pilot.yaml
python evaluate.py --config config/pilot.yaml --baseline   # the current NVIDIA model, for comparison
python export.py   --config config/pilot.yaml     # copies the model into the app

Then rebuild the app (gradlew assembleRelease). When app/src/main/assets/asr/quran/ holds a model, the app uses it right away and no longer offers the NVIDIA download.

Training in sessions

Training can be split into as many sessions as you like (2 hours now, 8 hours tonight…). Running the same train.py command again continues exactly where it stopped: same batch, optimizer state, learning rate schedule and data order.

Pause whenever you like: Ctrl+C in the training window, or the dashboard's Pause training button. It saves and exits within seconds (no validation on the way out).
Pause by itself: --stop-after 2 (hours) or --stop-at 07:30, e.g. python train.py --config config/full.yaml --stop-at 07:30 before going to sleep.
Accidents (closing the window, a power cut, Windows restarting for updates): it also saves every checkpoint_minutes (10), so at most that much training is lost.
Sleep/hibernate just freezes training; it carries on when the laptop wakes. Plug the laptop in: on battery the GPU slows down a lot.

The dashboard and the log count hours of audio and training time across all sessions.

Adding data or epochs to a running training

train.py notices when the training data or the number of epochs changed since its checkpoint, and continues from it: a new epoch starts with the new data and the learning rate warms up again. Always pause training first, then:

python download.py --config config/full.yaml   # e.g. after setting data.iqraeval: true
python prepare.py  --config config/full.yaml
# raise train.epochs in config/full.yaml if you want more epochs
python train.py    --config config/full.yaml   # "the training data changed: continuing from step ..."

Iqra'Eval (data.iqraeval: true): ordinary people reading sentences aloud (IqraEval/Iqra_train). Only the sentences whose text is Quran are kept (about a quarter), labelled with that Quran text. download.py fetches the 30 parquet files (~12 GB) one by one, keeps the Quranic clips and deletes each file; it resumes where it stopped. Few hours compared with the reciters, so it is sampled more often: iqraeval_share (0.10) is its share of each epoch's audio; train.py prints the resulting factor. The dataset card states no licence.

tlog (data.tlog_shards: 200): phone recordings by ordinary people in the Tarteel app (tarteel-ai/tlog, CC-BY-4.0). Run hf auth login and accept the dataset's terms on its page first. download.py fetches the shards (~165 MB each, resumable; Hugging Face's CDN is slow from here, ~5 MB/s in total). Their labels come from the file names and are often wrong, so prepare.py transcribes every clip once with the current model (on the GPU: pause training first) and keeps a clip only if what it hears matches its ayah, or a neighbouring ayah or two consecutive ones, with an optional basmala/isti'adha (tlog_max_error). The shards of eval_tlog.py's test set are never used for training.

Mix: ordinary_share: 0.5 makes ordinary people (Iqra'Eval + tlog) half of each epoch's audio and the professional reciters the other half; train.py prints the sampling factor. Epochs get longer (each now holds twice as much audio), so lower epochs if the ETA is too long.

Phone test: python eval_tlog.py scores the model (and the NVIDIA model) on 146 unseen tlog clips.

Watching training: the dashboard

In a second PowerShell window (same folder, venv activated):

python dashboard.py

It opens http://localhost:8008 and updates itself every 3 seconds while train.py runs:

Status: step, epoch, ETA and the clock time it should finish. It warns you if train.py has stopped writing.
Charts:
- loss per letter, with TensorBoard-style smoothing and an optional log scale;
- validation error (letters, words, skeleton, Hafs vs Warsh);
- learning rate and gradient norm;
- throughput, and the GPU's peak memory against its 12 GB.
- Drag across any chart to zoom (double-click to reset). Hovering shows the values on every chart at once.
GPU live: utilisation, memory, temperature and power from nvidia-smi. CPU and RAM too, after pip install psutil (useful to see whether num_workers keeps the GPU fed).
What the model hears: the same 12 validation ayahs at every evaluation, the model's text next to the reference with its mistakes marked. Press play to watch it learn to read.
Data and model: hours per reciter, and the run's settings.
Several runs: they appear as chips at the top. Ctrl+click (or "Compare runs") overlays them, e.g. pilot vs full.

It only reads files, so start and stop it whenever you like. To check on training from your phone on the same Wi-Fi: python dashboard.py --host 0.0.0.0, then open http://<this PC's IP>:8008 (allow Python through the Windows firewall when asked).

4. Full run

The same commands with --config config/full.yaml. It downloads ~~40 reciters (~~40–45 GB in total; files from the pilot are reused) and trains the ~16 MB model.

Reading the results

evaluate.py scores reciters the model never trained on, plus held-out Warsh surahs:

column	meaning
letters	letter error rate (CER)
words	word error rate (WER)
skeleton	letters without alif and hamza: what the tracker actually compares. Fairest when comparing with the NVIDIA model, whose spelling convention differs (الصراط vs الصرط)

On clean studio audio the NVIDIA model is already very good on Hafs (about 1% skeleton error). Your model should aim to match that while doing much better on Warsh and on phone recordings, at a quarter of the size.

Test on your own voice (recommended). Record a few ayahs on your phone. Save each one as data/audio/warsh/my_phone/SSSAAA.mp3, numbered like everyayah (Hafs numbering, e.g. 002255.mp3 for Ayat al-Kursi). Add warsh/my_phone to test_reciters, then re-run prepare.py and evaluate.py.

After exporting, ../tools/recitation_fixtures.py can regenerate the app's recitation-tracking test recordings with your model (see its docstring). Then run the app's RecitationTrackerTest.

Editing the config

Everything is in config/*.yaml. Copy a file to try variations; each run keeps its own folder.

Dataset size:
- hafs_reciters / warsh_reciters choose who trains the model.
- max_hours_per_reciter caps each reciter's hours.
- surahs restricts training to a subset of surahs.
- warsh_weight sets how often Warsh is seen.
Model: d_model, num_layers and num_heads set the size. Roughly, parameters ≈ 23 × d_model² × num_layers, and the int8 file is about 1 MB per million parameters. Grow it carefully: at 16 layers, or 512 wide, the model never left the stage where it predicts the same letters for every clip (loss flat around 2.6 per letter, word error ~100%). If the dashboard shows that after ~3,000 steps, stop and go back to a smaller model.
Augmentation starts after augment.start_after_steps (3,000): the model first learns from clean audio, then gets the noisy versions. Augmenting from the first step kept it from learning.
Training:
- epochs and lr (with warmup_steps).
- max_batch_seconds: audio per batch, counting padding. The log shows the GPU's peak memory; lower this if it nears 12 GB or the log says a batch was skipped for lack of memory.
- num_workers: CPU processes preparing audio; each prepares about 300 seconds of audio per second, so the default keeps the GPU fed.
- Speed measured on this laptop with the pilot model: about 1,000 hours of audio per hour of training (the first log lines print an ETA for your settings).
Augmentation: probabilities and ranges in augment / spec_augment.

Outputs, in runs/<name>/:

log.csv: loss, validation error, gradient norm, GPU memory and speed over time.
status.json / samples.json: live state and transcriptions, for the dashboard.
best.pt: the checkpoint with the lowest validation error.
step_*.pt: the last few checkpoints, which export.py averages.
export/: the ONNX models and tokens.txt.

Files

file	step
`download.py`	fetch recordings (resumable, parallel)
`prepare.py`	labels (Hafs/Warsh), basmala/isti'adha detection, splits, `tokens.txt`
`train.py`	training with automatic resume, bf16, validation and checkpoints
`dashboard.py`	live training dashboard in the browser (`dashboard.html`)
`evaluate.py`	letter/word/skeleton error per reciter; `--onnx` for the exported model, `--baseline` for NVIDIA
`export.py`	ONNX export, int8 quantization, sherpa-onnx parity check, copy into the app
`quran_asr/`	text normalization, Warsh↔Hafs alignment, features, augmentation, model, data loading
`tests/test_features.py`	checks the features against Kaldi (needs `pip install torchaudio`)

Licences: recordings are from everyayah.com, for personal use. The texts come from KFGQPC (Warsh) and Tanzil (Hafs, via fawazahmed0/quran-api). The NVIDIA model used for basmala detection is CC-BY-4.0.