| --- |
| language: en |
| tags: |
| - speaker-verification |
| - speaker-embedding |
| - indic |
| - wavlm |
| - ecapa-tdnn |
| - lora |
| license: mit |
| --- |
| |
| # Pi-Indic-Speaker-Embed-v1 |
|
|
| A 512-dimensional speaker embedding model optimized for **Indic languages** (Hindi, Bengali, Marathi, Telugu, and 8 more). Built on top of WavLM Large + ECAPA-TDNN with LoRA fine-tuning, designed for cross-lingual voice cloning and speaker verification tasks. |
|
|
| ## Model Description |
|
|
| | Component | Details | |
| |-----------|---------| |
| | **Frontend** | WavLM Large (frozen base, 315M params) + LoRA adapters on layers 9–24 | |
| | **LoRA scheme** | Layers 9–16: rank 8; Layers 17–24: rank 16; on all 6 projections (q, k, v, out, FFN up, FFN down) | |
| | **Encoder** | ECAPA-TDNN (3 Res2Net blocks, model_scale=8, ndim=1024, output=1536) | |
| | **Pooling** | Channel-Attention Statistics Pooling (→ 3072-dim) | |
| | **Projector** | RawNet3 (BatchNorm + Linear → 512-dim) | |
| | **Embedding dim** | 512 (L2-normalized) | |
| | **Total trainable params** | ~24M (LoRA 3.5M + ECAPA 18M + Pooling 0.8M + Projector 1.6M) | |
| | **Checkpoint size** | 96 MB (WavLM base downloaded separately from `microsoft/wavlm-large`) | |
| |
| ### Training Details |
| |
| - **Warm-started** from [`espnet/voxcelebs12_ecapa_wavlm_joint`](https://huggingface.co/espnet/voxcelebs12_ecapa_wavlm_joint) (8th epoch) |
| - **Training data**: Kathbath (12 Indic languages, 960 speakers) + VoxCeleb1 (full) + VoxCeleb2 (~2k sampled) |
| - **Loss**: AAM-Softmax with sub-centers (K=3, k_top=5, margin=0.2, scale=30) — discarded after training |
| - **Optimizer**: Adam (lr=1e-4, weight_decay=1e-5), ExponentialLR (gamma=0.87055) |
| - **Training**: 30 real epochs (150 mini-epochs), early stopped at epoch 29 (best at epoch 3) |
| - **Checkpoint**: Average of top-5 epochs by validation EER (epochs 3, 5, 6, 14, 20) |
| |
| ### Performance |
| |
| | Evaluation Set | EER | minDCF | Description | |
| |---------------|------|--------|-------------| |
| | Kathbath Validation (seen conditions, unseen speakers) | **1.586%** | 0.0640 | 79 speakers, 4 Indic languages | |
| | IndicVoices-R Test (unseen conditions, unseen speakers) | **4.003%** | 0.1699 | 64 speakers, 4 Indic languages | |
| | SOTA baseline (voxcelebs12, VoxCeleb-trained) | 3.999% / 5.161% | 0.1741 / 0.2248 | Same eval sets for comparison | |
| |
| The averaged checkpoint (`valid.eer.ave_5best.pth`) consistently outperforms single-epoch checkpoints on unseen data, confirming that weight averaging reduces overfitting. |
|
|
| ## Installation |
|
|
| ```bash |
| pip install espnet s3prl transformers torch soundfile librosa numpy huggingface_hub |
| ``` |
|
|
| ## Quick Start |
|
|
| ### Python API |
|
|
| ```python |
| from inference import PiIndicSpeakerEmbed |
| |
| # Downloads model files from HuggingFace automatically |
| model = PiIndicSpeakerEmbed.from_pretrained("Paytmlabs/Pi-indic-speaker-embed-v1") |
| |
| # Extract a single embedding |
| embedding = model.extract("audio.wav") |
| print(embedding.shape) # → (512,) |
| |
| # Extract embeddings for multiple files |
| embeddings = model.extract_batch(["speaker1.wav", "speaker2.wav", "speaker3.wav"]) |
| print(embeddings.shape) # → (3, 512) |
| ``` |
|
|
| ### Command Line |
|
|
| ```bash |
| # Single file |
| python inference.py --audio audio.wav |
| |
| # Multiple files, save to .npy |
| python inference.py --audio spk1.wav spk2.wav spk3.wav --output embeddings.npy |
| ``` |
|
|
| ### Speaker Verification (cosine similarity) |
|
|
| ```python |
| from inference import PiIndicSpeakerEmbed |
| import numpy as np |
| |
| model = PiIndicSpeakerEmbed.from_pretrained("Paytmlabs/Pi-indic-speaker-embed-v1") |
| |
| emb1 = model.extract("speaker_a_utt1.wav") |
| emb2 = model.extract("speaker_a_utt2.wav") |
| emb3 = model.extract("speaker_b_utt1.wav") |
| |
| # Cosine similarity (higher = more similar) |
| sim_same = np.dot(emb1, emb2) # same speaker, different utterance |
| sim_diff = np.dot(emb1, emb3) # different speakers |
| |
| print(f"Same speaker similarity: {sim_same:.4f}") |
| print(f"Diff speaker similarity: {sim_diff:.4f}") |
| # Same speaker similarity: 0.85+ (typically) |
| # Diff speaker similarity: 0.40-0.60 (typically) |
| ``` |
|
|
| ## How It Works |
|
|
| ``` |
| Raw audio (16kHz) |
| → 3 random crops × 96,000 samples (6s each) [Preprocessor] |
| → WavLM Large (frozen) + LoRA (layers 9-24) [Frontend] |
| → 1024-dim frame features @ 50Hz [multilayer weighted sum] |
| → utterance mean normalization [utterance_mvn] |
| → ECAPA-TDNN (Conv1d + 3 Res2Net blocks) [Encoder] |
| → 1536-dim frame-level embeddings |
| → Channel-Attention Statistics Pooling [Pooling] |
| → 3072-dim utterance embedding (weighted mean ⊕ weighted std) |
| → RawNet3 Projector (BatchNorm + Linear) [Projector] |
| → 512-dim speaker embedding |
| → L2 normalize [Inference] |
| → 512-dim unit vector on hypersphere |
| ``` |
|
|
| The AAM-Softmax classifier (used during training) is **discarded** after training. At inference, only the encoder + pooling + projector are used to produce embeddings. This means the model can verify **any speaker**, including those never seen during training. |
|
|
| ## Files |
|
|
| | File | Description | |
| |------|-------------| |
| | `valid.eer.ave_5best_stripped.pth` | Stripped checkpoint (LoRA + ECAPA + pooling + projector, 96 MB) | |
| | `config.yaml` | Model architecture configuration | |
| | `inference.py` | Self-contained inference script | |
| | `lora_s3prl_frontend.py` | Custom WavLM + LoRA frontend | |
| | `lora_wavlm.py` | LoRA injection utilities | |
| | `spk_preprocessor_lora.py` | Custom preprocessor | |
|
|
| ## Citation |
|
|
| If you use this model, please cite: |
|
|
| ```bibtex |
| @misc{pi-indic-speaker-embed-v1, |
| title={Pi-Indic-Speaker-Embed-v1: A 512-dim Speaker Embedding Model for Indic Languages}, |
| author={Paytmlabs}, |
| year={2026}, |
| url={https://huggingface.co/Paytmlabs/Pi-indic-speaker-embed-v1} |
| } |
| ``` |
|
|
| ## License |
|
|
| MIT License. The WavLM Large base model follows its own license (`microsoft/wavlm-large`). |
|
|
