nemotron-3.5-asr-arabic-dialectal

Fine-tune of nvidia/nemotron-3.5-asr-streaming-0.6b (638M) for multi-dialect Arabic speech recognition (undiacritized output).

Private / internal model. Evaluate on your own data before production use.

Results (932-clip held-out test set)

	WER	CER
Base (nemotron-3.5-asr-streaming-0.6b, zero-shot)	0.592	0.252
This model (fine-tuned)	0.422	n/a

Cache-aware STREAMING RNNT with native Arabic. Direct fine-tune (no vocab swap). Edge is low-latency streaming, not peak WER.

Model comparison — all Arabic ASR models

Same 932-clip held-out test set, same clean_text scoring (strip tashkil + tags, keep punctuation + dialect spelling) — so every row is directly comparable.

Model	Params	Zero-shot WER	Fine-tuned WER	CER (best)
whisper-large-v3-turbo 🏆	809M	0.590	0.344	0.115
cohere-transcribe-arabic	2.0B	0.457	0.357	0.137
whisper-medium	769M	0.717	0.358	0.123
nemotron-3.5-asr (streaming)	638M	0.592	0.422	—
whisper-small	244M	~0.77	0.428	0.151
qwen3-asr-0.6b	938M	0.756	0.676	0.408
qwen3-asr-1.7b	1.7B	—	training	—

• Best fine-tuned: whisper-large-v3-turbo (WER 0.344), with cohere-transcribe-arabic a close second (0.357).
• Best zero-shot: cohere-transcribe-arabic (0.457, Arabic-specialized). A full fine-tune (all ~2B params, low LR) now improves it to 0.357; an earlier 32 GB LoRA attempt had instead degraded it (0.510, overfit) — full-parameter tuning with best-checkpoint selection was the fix.
• Streaming / low-latency: nemotron-3.5-asr.
• Parakeet-TDT-0.6b-v3 was tried but abandoned (European-only pretraining; cross-lingual transfer to Arabic converged far too slowly, WER ~0.93 after 3k steps).

Dataset

Fine-tuned on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private).

• ~40,000 train / 1,000 test clips, 16 kHz mono, single-channel.
• Multi-dialect Arabic: Levantine (Lebanese/Syrian), Maghrebi (Moroccan/Algerian/Tunisian), Egyptian, Gulf/Saudi, Sudanese, Iraqi, and MSA.
• Augmented: each clean clip is expanded with variants (noise, music, speed, voice, reverb/codec) via an augmentation column. Derived from oddadmix/dialectal-arabic-lahgtna-v2-smaller.

Preprocessing (applied to targets)

• Stripped tashkil/harakat (diacritics) and tatweel; removed non-verbal tags ([laughter], [exhale], [inhale], [mumble], [cough], timestamps, …).
• Kept dialectal consonants (گ ڨ چ پ ژ) — they encode real phonemes — and sentence punctuation. Output is undiacritized.
• Filtered: dropped clips > 30 s (~10%; Whisper's encoder limit) and a few corrupt rows (12k-token transcripts). ~62% of rows had harakat, ~28% had non-verbal tags before cleaning. → 36,769 train / 932 test after filtering.

Training

• Base: nvidia/nemotron-3.5-asr-streaming-0.6b · full fine-tune (no LoRA)
• Effective batch 16 · LR 1e-4 (warmup 500, cosine) · 6000 steps · best @ step 6000 · ~200 min
• Hardware: a single 32 GB GPU, bf16 autocast (fp32 loss for the RNNT numba kernel)
• Metric: WER/CER on cleaned references (NeMo val_wer with the ar prompt)

Fine-tuning (reproduce)

The exact fine-tuning code is bundled in this repo (train_nemotron.py, export_nemo_data.py, normalize.py, eval_nemotron.py) plus requirements.txt. See FINETUNE.md for the full walkthrough + lessons learned. Trained on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private) — swap in any HF audio dataset with audio + text columns. normalize.py is the shared text cleaning (strip tashkil + non-verbal tags, keep dialectal letters گ ڨ چ).

pip install -r requirements.txt        # NeMo main + Blackwell pins

python export_nemo_data.py --out data/nemo    # wav + NeMo manifests (per-sample lang=ar)

NUMBA_CUDA_USE_NVIDIA_BINDING=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  python train_nemotron.py --run_name my-run --precision 32 --batch_duration 40

python eval_nemotron.py --model runs/my-run/.../checkpoints/my-run.nemo

Notes: --precision 32 is required (the numba RNNT loss can't take bf16), and NUMBA_CUDA_USE_NVIDIA_BINDING=1 lets numba compile for sm_120. The model is prompt-conditioned, so the manifests carry a per-sample lang=ar field.

Usage

# needs NeMo main:  pip install "nemo_toolkit[asr] @ git+https://github.com/NVIDIA/NeMo.git@main"
import nemo.collections.asr as nemo_asr
m = nemo_asr.models.ASRModel.from_pretrained("oddadmix/nemotron-3.5-asr-arabic-dialectal")
print(m.transcribe(["clip.wav"], prompt={"target_lang": "ar"})[0].text)

Learnings & notes

• Data cleaning matters most: keeping dialectal letters + stripping tashkil/tags (rather than aggressive letter-folding) was key for a multi-dialect model.
• Whisper fine-tunes furthest; the offline (full-context) models beat the streaming Nemotron on pure WER, though Nemotron has the best zero-shot (native Arabic) and offers streaming.
• Blackwell (sm_120) GPU gotchas for the NeMo/Nemotron model: numba's RNNT loss needs numba==0.61.2 + NUMBA_CUDA_USE_NVIDIA_BINDING=1 + cuda-python<13 to emit sm_120 code, and fp32 (the numba loss can't take bf16). Nemotron 3.5 requires NeMo main (its EncDecRNNTBPEModelWithPrompt class isn't in any release) and a per-sample lang=ar prompt.

Limitations

• Trained on augmented, partly synthetic multi-dialect data; real-world dialect coverage varies (Maghrebi is the hardest).
• Output is undiacritized and lower-cased for Latin tokens.
• Streaming/cache-aware model — WER trades off against latency.