# Training the app's Quran recitation model

A small speech model built only for Quran recitation: it hears a reciter and writes the Arabic
letters (no harakat). The app uses it to follow your recitation and mark mistakes.

- **Model**: Conformer with a CTC head, trained from scratch in PyTorch: 6 layers, pilot ~10M
  parameters (~10 MB), full ~16M (~16 MB). Deeper (16 layers) or wider (512) versions got stuck
  predicting the same letters for every clip; see "Editing the config" before enlarging it.
- **Output**: the 29 Arabic letters plus a word separator. That is exactly what the tracker and the
  mistake hints compare, and the small alphabet keeps the model small and accurate.
- **Data**: per-ayah recordings from [everyayah.com](https://everyayah.com).
  - Hafs reciters are labelled with the Hafs Uthmani text.
  - The 3 Warsh reciters are labelled with the Warsh Uthmani text. everyayah numbers even the
    Warsh files by Hafs ayahs, so the Warsh text is aligned to them word by word.
  - Warsh clips are sampled more often (`warsh_weight`).
- **Long ayahs included**: recordings up to 7.5 minutes (`max_duration`), e.g. al-Baqara 282 or a
  mujawwad Ayat al-Kursi. Training uses a memory-efficient attention kernel, so a 7-minute ayah
  needs ~4 GB (pilot) / ~7 GB (full) of GPU memory instead of running out.
- **Phone robustness**: every clip is randomly degraded while training: room echo, noise,
  phone-like band limits, volume, speed and spectrogram masking.
- **Any reciting speed**: half the clips are also made faster or slower at the same pitch
  (0.75x to 2.5x, `tempo_range`), from a slow tartil to a quick review (hadr).
- **On the phone**: the exported model runs in the app's existing engine (sherpa-onnx). The training
  features are the exact Kaldi filterbank sherpa computes. `export.py` checks that sherpa and
  PyTorch produce the same text before copying the model into the app.

## 1. Set up (once)

From this folder (`training/`), in PowerShell:

```
python -m venv .venv
.venv\Scripts\activate
pip install torch --index-url https://download.pytorch.org/whl/cu130
pip install -r requirements.txt
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"
```

The last line should print `True NVIDIA GeForce RTX 5070 Ti Laptop GPU`.

## 2. Smoke test (about 10 minutes)

A tiny model on a few surahs, to check everything is wired up:

```
python download.py --config config/smoke.yaml
python prepare.py  --config config/smoke.yaml
python train.py    --config config/smoke.yaml
python export.py   --config config/smoke.yaml
```

`export.py` should end with `parity ... identical text in 10/10` (or close to it). The smoke
model is copied into `runs/smoke/app_assets`, not into the app.

## 3. Pilot run

```
python download.py --config config/pilot.yaml     # ~17 reciters, ~15–20 GB, resumable
python prepare.py  --config config/pilot.yaml     # labels + splits; prints hours per reciter
python train.py    --config config/pilot.yaml     # prints an ETA; pause/resume any time (see below)
python evaluate.py --config config/pilot.yaml
python evaluate.py --config config/pilot.yaml --baseline   # the current NVIDIA model, for comparison
python export.py   --config config/pilot.yaml     # copies the model into the app
```

Then rebuild the app (`gradlew assembleRelease`). When `app/src/main/assets/asr/quran/` holds a
model, the app uses it right away and no longer offers the NVIDIA download.

## Training in sessions

Training can be split into as many sessions as you like (2 hours now, 8 hours tonight…). Running the
same `train.py` command again continues exactly where it stopped: same batch, optimizer state,
learning rate schedule and data order.

- **Pause whenever you like**: Ctrl+C in the training window, or the dashboard's **Pause training**
  button. It saves and exits within seconds (no validation on the way out).
- **Pause by itself**: `--stop-after 2` (hours) or `--stop-at 07:30`, e.g.
  `python train.py --config config/full.yaml --stop-at 07:30` before going to sleep.
- **Accidents** (closing the window, a power cut, Windows restarting for updates): it also saves
  every `checkpoint_minutes` (10), so at most that much training is lost.
- Sleep/hibernate just freezes training; it carries on when the laptop wakes. Plug the laptop in:
  on battery the GPU slows down a lot.

The dashboard and the log count hours of audio and training time across all sessions.

## Adding data or epochs to a running training

`train.py` notices when the training data or the number of epochs changed since its checkpoint,
and continues from it: a new epoch starts with the new data and the learning rate warms up again.
Always pause training first, then:

```
python download.py --config config/full.yaml   # e.g. after setting data.iqraeval: true
python prepare.py  --config config/full.yaml
# raise train.epochs in config/full.yaml if you want more epochs
python train.py    --config config/full.yaml   # "the training data changed: continuing from step ..."
```

**Iqra'Eval** (`data.iqraeval: true`): ordinary people reading sentences aloud
([IqraEval/Iqra_train](https://huggingface.co/datasets/IqraEval/Iqra_train)). Only the sentences whose
text is Quran are kept (about a quarter), labelled with that Quran text. `download.py` fetches the 30
parquet files (~12 GB) one by one, keeps the Quranic clips and deletes each file; it resumes where it
stopped. Few hours compared with the reciters, so it is sampled more often: `iqraeval_share` (0.10)
is its share of each epoch's audio; `train.py` prints the resulting factor. The dataset card states
no licence.

**tlog** (`data.tlog_shards: 200`): phone recordings by ordinary people in the Tarteel app
([tarteel-ai/tlog](https://huggingface.co/datasets/tarteel-ai/tlog), CC-BY-4.0). Run `hf auth login`
and accept the dataset's terms on its page first. `download.py` fetches the shards (~165 MB each,
resumable; Hugging Face's CDN is slow from here, ~5 MB/s in total). Their labels come from the file
names and are often wrong, so `prepare.py` transcribes every clip once with the current model (on the
GPU: pause training first) and keeps a clip only if what it hears matches its ayah, or a neighbouring
ayah or two consecutive ones, with an optional basmala/isti'adha (`tlog_max_error`). The shards of
`eval_tlog.py`'s test set are never used for training.

**Mix**: `ordinary_share: 0.5` makes ordinary people (Iqra'Eval + tlog) half of each epoch's audio and
the professional reciters the other half; `train.py` prints the sampling factor. Epochs get longer
(each now holds twice as much audio), so lower `epochs` if the ETA is too long.

**Phone test**: `python eval_tlog.py` scores the model (and the NVIDIA model) on 146 unseen tlog clips.

## Watching training: the dashboard

In a second PowerShell window (same folder, venv activated):

```
python dashboard.py
```

It opens http://localhost:8008 and updates itself every 3 seconds while `train.py` runs:

- **Status**: step, epoch, ETA and the clock time it should finish. It warns you if `train.py` has stopped writing.
- **Charts**:
  - loss per letter, with TensorBoard-style smoothing and an optional log scale;
  - validation error (letters, words, skeleton, Hafs vs Warsh);
  - learning rate and gradient norm;
  - throughput, and the GPU's peak memory against its 12 GB.
  - Drag across any chart to zoom (double-click to reset). Hovering shows the values on every chart at once.
- **GPU live**: utilisation, memory, temperature and power from `nvidia-smi`. CPU and RAM too, after
  `pip install psutil` (useful to see whether `num_workers` keeps the GPU fed).
- **What the model hears**: the same 12 validation ayahs at every evaluation, the model's text next
  to the reference with its mistakes marked. Press play to watch it learn to read.
- **Data and model**: hours per reciter, and the run's settings.
- **Several runs**: they appear as chips at the top. Ctrl+click (or "Compare runs") overlays them,
  e.g. pilot vs full.

It only reads files, so start and stop it whenever you like. To check on training from your phone
on the same Wi-Fi: `python dashboard.py --host 0.0.0.0`, then open `http://<this PC's IP>:8008`
(allow Python through the Windows firewall when asked).

## 4. Full run

The same commands with `--config config/full.yaml`. It downloads ~40 reciters (~40–45 GB in total;
files from the pilot are reused) and trains the ~16 MB model.

## Reading the results

`evaluate.py` scores reciters the model never trained on, plus held-out Warsh surahs:

| column | meaning |
|---|---|
| letters | letter error rate (CER) |
| words | word error rate (WER) |
| skeleton | letters without alif and hamza: what the tracker actually compares. Fairest when comparing with the NVIDIA model, whose spelling convention differs (الصراط vs الصرط) |

On clean studio audio the NVIDIA model is already very good on Hafs (about 1% skeleton error).
Your model should aim to match that while doing much better on Warsh and on phone recordings, at
a quarter of the size.

**Test on your own voice (recommended).** Record a few ayahs on your phone. Save each one as
`data/audio/warsh/my_phone/SSSAAA.mp3`, numbered like everyayah (Hafs numbering, e.g. `002255.mp3`
for Ayat al-Kursi). Add `warsh/my_phone` to `test_reciters`, then re-run `prepare.py` and `evaluate.py`.

After exporting, `../tools/recitation_fixtures.py` can regenerate the app's recitation-tracking test
recordings with your model (see its docstring). Then run the app's `RecitationTrackerTest`.

## Editing the config

Everything is in `config/*.yaml`. Copy a file to try variations; each run keeps its own folder.

- **Dataset size**:
  - `hafs_reciters` / `warsh_reciters` choose who trains the model.
  - `max_hours_per_reciter` caps each reciter's hours.
  - `surahs` restricts training to a subset of surahs.
  - `warsh_weight` sets how often Warsh is seen.
- **Model**: `d_model`, `num_layers` and `num_heads` set the size. Roughly, parameters ≈ 23 ×
  `d_model`² × `num_layers`, and the int8 file is about 1 MB per million parameters. Grow it
  carefully: at 16 layers, or 512 wide, the model never left the stage where it predicts the same
  letters for every clip (loss flat around 2.6 per letter, word error ~100%). If the dashboard
  shows that after ~3,000 steps, stop and go back to a smaller model.
- **Augmentation** starts after `augment.start_after_steps` (3,000): the model first learns from
  clean audio, then gets the noisy versions. Augmenting from the first step kept it from learning.
- **Training**:
  - `epochs` and `lr` (with `warmup_steps`).
  - `max_batch_seconds`: audio per batch, counting padding. The log shows the GPU's peak memory;
    lower this if it nears 12 GB or the log says a batch was skipped for lack of memory.
  - `num_workers`: CPU processes preparing audio; each prepares about 300 seconds of audio per
    second, so the default keeps the GPU fed.
  - Speed measured on this laptop with the pilot model: about 1,000 hours of audio per hour of
    training (the first log lines print an ETA for your settings).
- **Augmentation**: probabilities and ranges in `augment` / `spec_augment`.

Outputs, in `runs/<name>/`:
- `log.csv`: loss, validation error, gradient norm, GPU memory and speed over time.
- `status.json` / `samples.json`: live state and transcriptions, for the dashboard.
- `best.pt`: the checkpoint with the lowest validation error.
- `step_*.pt`: the last few checkpoints, which `export.py` averages.
- `export/`: the ONNX models and `tokens.txt`.

## Files

| file | step |
|---|---|
| `download.py` | fetch recordings (resumable, parallel) |
| `prepare.py` | labels (Hafs/Warsh), basmala/isti'adha detection, splits, `tokens.txt` |
| `train.py` | training with automatic resume, bf16, validation and checkpoints |
| `dashboard.py` | live training dashboard in the browser (`dashboard.html`) |
| `evaluate.py` | letter/word/skeleton error per reciter; `--onnx` for the exported model, `--baseline` for NVIDIA |
| `export.py` | ONNX export, int8 quantization, sherpa-onnx parity check, copy into the app |
| `quran_asr/` | text normalization, Warsh↔Hafs alignment, features, augmentation, model, data loading |
| `tests/test_features.py` | checks the features against Kaldi (needs `pip install torchaudio`) |

**Licences**: recordings are from everyayah.com, for personal use. The texts come from
KFGQPC (Warsh) and Tanzil (Hafs, via fawazahmed0/quran-api). The NVIDIA model used for basmala
detection is CC-BY-4.0.