|
Download README.md from EchoesML/Georgian-NeMo: direct link, hf CLI and curl.
- Browser
- Download file 13.5 kB
-
https://huggingface.co/EchoesML/Georgian-NeMo/resolve/main/README.md
- Command line
-
hf download hf://EchoesML/Georgian-NeMo/README.md
-
curl -L -o README.md https://huggingface.co/EchoesML/Georgian-NeMo/resolve/main/README.md
13.5 kB
| title: Georgian ASR Demo | |
| emoji: ๐๏ธ | |
| colorFrom: red | |
| colorTo: gray | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| # ๐ฌ๐ช Georgian ASR โ End-to-End Speech Recognition System | |
| > Automatic Speech Recognition system for the Georgian language, built from the ground up using state-of-the-art deep learning techniques, custom data pipelines, and domain-adapted language models. | |
| --- | |
| ## Overview | |
| Georgian (แฅแแ แแฃแแ) is a South Caucasian language spoken by approximately 3.7 million people. Despite its rich linguistic heritage and unique script (แแฎแแแ แฃแแ), Georgian remains severely underserved in the NLP and speech recognition space. This project addresses that gap by building a robust, general-purpose Georgian ASR system capable of handling diverse acoustic conditions โ from clean studio recordings to noisy phone call audio. | |
| The system achieves competitive Word Error Rate (WER) on Georgian conversational speech and is released for research and educational purposes only. | |
| --- | |
| ## Architecture | |
| ### Core Model โ NeMo FastConformer Hybrid (RNNT + CTC) | |
| The acoustic model is built on NVIDIA's **FastConformer-Hybrid-Large** architecture, fine-tuned from the pretrained checkpoint `nvidia/stt_ka_fastconformer_hybrid_large_pc`. | |
| #### What is FastConformer? | |
| FastConformer is an evolution of the Conformer architecture introduced in *"Conformer: Convolution-augmented Transformer for Speech Recognition"* (arxiv.org/abs/2005.08100). It combines: | |
| - **Multi-Head Self-Attention** โ captures long-range temporal dependencies in speech, allowing the model to understand context across the entire utterance | |
| - **Depthwise Separable Convolution** โ captures local acoustic patterns (phoneme-level features) efficiently | |
| - **Feed-Forward Layers** โ non-linear transformations between attention and convolution blocks | |
| - **Subsampling** โ reduces sequence length by 8x using strided convolutions, making attention computationally feasible on long sequences | |
| FastConformer specifically improves over standard Conformer by using **8x subsampling** instead of 4x and **global attention with local windows**, reducing compute by approximately 2.5x while maintaining accuracy. | |
| #### Hybrid RNNT + CTC Decoder | |
| The model uses a **hybrid decoding architecture**: | |
| **RNNT (Recurrent Neural Network Transducer)** โ the primary decoder used at inference time. RNNT jointly models acoustic and language probability, producing output tokens conditioned on both the audio encoding and previously predicted tokens. It handles variable-length input-output alignment natively through a special blank token mechanism, making it significantly more accurate than CTC for conversational speech. | |
| **CTC (Connectionist Temporal Classification)** โ used as an auxiliary loss during training only (weight 0.3). CTC enforces alignment structure on the encoder, stabilizing early training and improving convergence speed. At inference, CTC is discarded. | |
| The joint loss during training is: | |
| ``` | |
| Total Loss = 0.7 ร RNNT_Loss + 0.3 ร CTC_Loss | |
| ``` | |
| --- | |
| ### Language Model โ KenLM n-gram | |
| KenLM (Heafield, 2011 โ *"KenLM: Faster and Smaller Language Model Queries"*, arxiv.org/abs/1106.1234) is used for shallow fusion rescoring at inference time. An n-gram language model trained on Georgian text corpora assigns probability to word sequences, boosting acoustically ambiguous but linguistically plausible outputs. | |
| The KenLM model was trained on public Georgian text from multiple domains including financial documents, legal texts, news articles. | |
| **Shallow fusion** combines the acoustic model score with the language model score at decode time: | |
| ``` | |
| Score(y|x) = log P_acoustic(y|x) + ฮป ร log P_LM(y) | |
| ``` | |
| Where ฮป is a tunable interpolation weight. | |
| --- | |
| ### Voice Activity Detection โ Silero VAD | |
| **Silero VAD** (github.com/snakers4/silero-vad) is used for audio segmentation before ASR inference. It is a lightweight temporal convolutional network (~1MB) that classifies 32ms audio frames as speech or silence with high accuracy across languages. | |
| The VAD pipeline: | |
| 1. Loads full-length audio | |
| 2. Detects speech segment boundaries | |
| 3. Splits into chunks at natural silence boundaries | |
| 4. Passes cleaned segments to the ASR model | |
| Silero VAD is language-agnostic โ speech vs silence detection relies on universal acoustic properties (harmonic structure, energy distribution) that generalize across all human languages without fine-tuning. | |
| --- | |
| ### Noise Reduction โ Spectral Gating | |
| Pre-processing applies **spectral noise reduction** (Sainburg et al., 2020 โ based on Boll 1979's spectral subtraction) via the `noisereduce` library: | |
| 1. Estimate noise spectral profile from silent segments | |
| 2. Compute per-frequency noise threshold | |
| 3. Apply sigmoid spectral gate โ attenuate bins where signal โ noise floor | |
| 4. Reconstruct via inverse STFT | |
| This improves transcription accuracy on noisy phone call audio without any model retraining. | |
| --- | |
| ## Data Pipeline | |
| ### Data Collection | |
| Training data was collected from diverse Georgian audio sources to maximize acoustic and linguistic coverage: | |
| | Source | Type | Approximate Size | | |
| |--------|------|-----------------| | |
| | Mozilla Common Voice (Georgian) | Read speech, multiple speakers | ~4,200 utterances | | |
| | Public Georgian Data | ~7,300 utterances | | |
| | Phone call recordings | Telephony, noisy, short utterances | ~7,700 utterances | | |
| **Total: ~20,000 unique original utterances, ~108 hours with augmentation** | |
| ### Data Labeling | |
| Manual labeling was performed for original recordings. Gemini 2.5 Flash API was used for pseudo-labeling of new data, with a confidence filtering pipeline: | |
| 1. Run current ASR model on unlabeled audio โ model transcription | |
| 2. Run Gemini API on same audio โ Gemini transcription | |
| 3. Compute symmetric Character Error Rate between the two | |
| 4. Auto-accept entries with symmetric CER โค 10% (using Gemini text as label) | |
| 5. Manual review for entries with CER 10-30% | |
| 6. Discard entries with CER > 30% | |
| This approach reduced manual labeling effort by approximately 60-70% while maintaining label quality. | |
| **KenLM vs RNNT โ empirical comparison:** | |
| A domain-specific **KenLM n-gram language model** was trained on Georgian | |
| text from financial documents, legal texts, and news articles, and applied | |
| via shallow fusion with the CTC decoder head as an alternative decoding | |
| strategy. Despite KenLM providing meaningful language model support to CTC, | |
| **RNNT greedy decoding still outperformed CTC+KenLM** on the test set. | |
| This confirms that RNNT's joint acoustic-language modeling โ where the | |
| decoder conditions on both audio representations and previously predicted | |
| tokens simultaneously โ is more powerful than post-hoc language model | |
| rescoring for Georgian conversational speech. KenLM remains available as | |
| a lightweight alternative for latency-constrained deployments where RNNT | |
| beam search is too slow. | |
| ### Audio Preprocessing | |
| Raw audio goes through the following pipeline before training: | |
| ``` | |
| Raw WAV | |
| โ Resample to 16kHz mono (librosa) | |
| โ VAD-based chunking (Silero VAD) | |
| - Min chunk: 0.3s | |
| - Max chunk: 15.0s | |
| - Split on silence โฅ 200ms | |
| โ Duration filtering | |
| - Remove: duration < 0.3s or > 15.0s | |
| โ Add to manifest | |
| ``` | |
| ### Augmentation Strategy | |
| A **hybrid static + dynamic augmentation** approach was used: | |
| **Static augmentations** (pre-generated, saved as separate files): | |
| | Augmentation | Purpose | Ratio | | |
| |---|---|---| | |
| | Telephone bandwidth filter (300-3400Hz) | Simulate phone call audio | 1:1 | | |
| | Codec noise (MP3/GSM compression) | Simulate compressed audio | 1:1 | | |
| | Room Impulse Response convolution | Simulate reverberant environments | 1:1 | | |
| | SNR noise mixing (MUSAN/DNS) | Simulate background noise | 1:1 | | |
| **Dynamic augmentations** (applied on-the-fly during training via NeMo preprocessor): | |
| | Augmentation | Parameters | Probability | | |
| |---|---|---| | |
| | Gain perturbation | -10 to +5 dBFS | 50% | | |
| | Speed perturbation | 0.9x to 1.1x | 50% | | |
| **SpecAugment** (applied on mel spectrogram during training): | |
| - Frequency masking: 2 masks, max width 27 bins | |
| - Time masking: 10 masks, max width 5% of utterance | |
| The philosophy: static augmentations teach domain-specific acoustic robustness. Dynamic augmentations introduce infinite variety so the model never sees the exact same file twice across epochs. | |
| --- | |
| ## Training Configuration | |
| ```python | |
| Model: nvidia/stt_ka_fastconformer_hybrid_large_pc (fine-tuned) | |
| Optimizer: AdamW (weight_decay=1e-3) | |
| LR: 1e-4 with CosineAnnealing (warmup=1000 steps, min_lr=1e-6) | |
| Batch size: 32 (effective 64 with gradient accumulation ร 2) | |
| Precision: BF16 mixed precision | |
| GPU: NVIDIA L4 (24GB) | |
| Max epochs: 50 (early stopping patience=8 on val_wer) | |
| CTC weight: 0.3 (auxiliary loss during training only) | |
| ``` | |
| ### Validation Strategy | |
| Validation set: 400 unique original utterances (no augmented variants), distributed to mirror training domain proportions: | |
| Zero leakage verified โ no training stem appears in validation in any augmented form. | |
| ### Checkpoint Averaging | |
| After training, top-5 checkpoints (by val_wer) are averaged using uniform weight averaging: | |
| ```python | |
| averaged_weights[key] = sum(ckpt[key] for ckpt in top5) / 5 | |
| ``` | |
| This smooths the loss landscape and consistently gives 0.1-0.3% absolute WER improvement over the single best checkpoint. | |
| --- | |
| ## Results | |
| | Model | WER | CER | | |
| |---|---|---| | |
| | This model (fine-tuned) | 12.32% | 3.89% | | |
| | NVIDIA Base (starting point) | 26.70% | 10.38% | | |
| | Azure Speech Service | 49.29% | 40.93% | | |
| Evaluated on 575 clips from publicly available Georgian government | |
| and broadcaster content โ National Bank press conferences, NCDC | |
| health briefings, etc. | |
| 54% relative WER improvement over base. 75% over Azure. | |
| --- | |
| ### Fleurs KA Benchmark (google/fleurs, ka_ge test set) | |
| | Model | WER | CER | | |
| |---|---|---| | |
| | This model (Last_Average.ckpt) | **11.45%** | **4.06%** | | |
| | NVIDIA Base (nvidia/stt_ka_fastconformer_hybrid_large_pc) | 13.63% | 4.39% | | |
| | Whisper Large-v3 (forced language=ka) | 76.33% | 37.05% | | |
| Notably, the base NVIDIA model was trained directly on Fleurs Georgian | |
| train/dev splits โ yet our fine-tuned model still outperforms it on the | |
| Fleurs test set, demonstrating genuine generalization rather than | |
| benchmark overfitting. | |
| ## Key Technical Decisions | |
| **Why FastConformer over Whisper?** | |
| FastConformer's hybrid RNNT+CTC architecture trains more stably on limited data than Whisper's encoder-decoder approach. RNNT handles streaming inference naturally and has lower latency for real-time applications. For a low-resource language like Georgian, fine-tuning a pretrained NVIDIA model proved more effective than fine-tuning Whisper. | |
| **Why static + dynamic augmentation?** | |
| Static augmentations (telephone, codec, RIR, noise) teach the model specific acoustic degradation patterns. Dynamic augmentations (gain, speed) prevent memorization โ the model never hears the exact same file twice across epochs. Separating these concerns gives cleaner control over what the model learns. | |
| **Why KenLM over neural LM rescoring?** | |
| KenLM is fast, interpretable, and requires minimal Georgian text to train effectively. Neural LM rescoring (e.g. with GPT-style models) would give marginally better results but adds significant inference latency. For a production system targeting call center transcription, latency matters. | |
| **Why confidence filtering for pseudo-labels?** | |
| Fully manual labeling of 20,000+ utterances is not feasible for a solo project. Using the ASR model and Gemini as dual references and keeping only high-agreement entries (symmetric CER โค 15%) gives near-manual label quality at a fraction of the cost. The two-reference approach catches cases where both systems hallucinate on the same audio (rare but possible) better than single-model filtering. | |
| --- | |
| ## References | |
| 1. Gulati et al. (2020). *Conformer: Convolution-augmented Transformer for Speech Recognition.* arxiv.org/abs/2005.08100 | |
| 2. Graves et al. (2012). *Sequence Transduction with Recurrent Neural Networks.* arxiv.org/abs/1211.3711 โ Original RNNT paper. | |
| 3. Graves et al. (2006). *Connectionist Temporal Classification.* โ Original CTC paper, ICML 2006. | |
| 4. Heafield (2011). *KenLM: Faster and Smaller Language Model Queries.* Proceedings of the Sixth Workshop on Statistical Machine Translation. | |
| 5. Park et al. (2019). *SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition.* arxiv.org/abs/1904.08779 | |
| 6. Boll (1979). *Suppression of acoustic noise in speech using spectral subtraction.* IEEE Transactions on Acoustics, Speech, and Signal Processing. | |
| 7. Sainburg et al. (2020). *Finding, visualizing, and quantifying latent structure across diverse animal acoustic repertoires.* PLOS Computational Biology. โ Spectral gating formalization used in noisereduce. | |
| 8. Ko et al. (2015). *Audio augmentation for speech recognition.* Interspeech 2015. โ Speed perturbation augmentation. | |
| 9. Snyder et al. (2015). *MUSAN: A Music, Speech, and Noise Corpus.* arxiv.org/abs/1510.08484 โ Noise corpus used for SNR augmentation. | |
| --- | |
| ## Author | |
| Built by โ Senior AI/ML Engineer | |
| *This project demonstrates end-to-end ML engineering for a low-resource language: data collection and pseudo-labeling at scale, augmentation pipeline design, model fine-tuning, evaluation framework, and production inference pipeline.* | |
| --- | |
| *For research and demonstration purposes.* |