---
language: en
tags:
  - speaker-verification
  - speaker-embedding
  - indic
  - wavlm
  - ecapa-tdnn
  - lora
license: mit
---

# Pi-Indic-Speaker-Embed-v1

A 512-dimensional speaker embedding model optimized for **Indic languages** (Hindi, Bengali, Marathi, Telugu, and 8 more). Built on top of WavLM Large + ECAPA-TDNN with LoRA fine-tuning, designed for cross-lingual voice cloning and speaker verification tasks.

## Model Description

| Component | Details |
|-----------|---------|
| **Frontend** | WavLM Large (frozen base, 315M params) + LoRA adapters on layers 9–24 |
| **LoRA scheme** | Layers 9–16: rank 8; Layers 17–24: rank 16; on all 6 projections (q, k, v, out, FFN up, FFN down) |
| **Encoder** | ECAPA-TDNN (3 Res2Net blocks, model_scale=8, ndim=1024, output=1536) |
| **Pooling** | Channel-Attention Statistics Pooling (→ 3072-dim) |
| **Projector** | RawNet3 (BatchNorm + Linear → 512-dim) |
| **Embedding dim** | 512 (L2-normalized) |
| **Total trainable params** | ~24M (LoRA 3.5M + ECAPA 18M + Pooling 0.8M + Projector 1.6M) |
| **Checkpoint size** | 96 MB (WavLM base downloaded separately from `microsoft/wavlm-large`) |

### Training Details

- **Warm-started** from [`espnet/voxcelebs12_ecapa_wavlm_joint`](https://huggingface.co/espnet/voxcelebs12_ecapa_wavlm_joint) (8th epoch)
- **Training data**: Kathbath (12 Indic languages, 960 speakers) + VoxCeleb1 (full) + VoxCeleb2 (~2k sampled)
- **Loss**: AAM-Softmax with sub-centers (K=3, k_top=5, margin=0.2, scale=30) — discarded after training
- **Optimizer**: Adam (lr=1e-4, weight_decay=1e-5), ExponentialLR (gamma=0.87055)
- **Training**: 30 real epochs (150 mini-epochs), early stopped at epoch 29 (best at epoch 3)
- **Checkpoint**: Average of top-5 epochs by validation EER (epochs 3, 5, 6, 14, 20)

### Performance

| Evaluation Set | EER | minDCF | Description |
|---------------|------|--------|-------------|
| Kathbath Validation (seen conditions, unseen speakers) | **1.586%** | 0.0640 | 79 speakers, 4 Indic languages |
| IndicVoices-R Test (unseen conditions, unseen speakers) | **4.003%** | 0.1699 | 64 speakers, 4 Indic languages |
| SOTA baseline (voxcelebs12, VoxCeleb-trained) | 3.999% / 5.161% | 0.1741 / 0.2248 | Same eval sets for comparison |

The averaged checkpoint (`valid.eer.ave_5best.pth`) consistently outperforms single-epoch checkpoints on unseen data, confirming that weight averaging reduces overfitting.

## Installation

```bash
pip install espnet s3prl transformers torch soundfile librosa numpy huggingface_hub
```

## Quick Start

### Python API

```python
from inference import PiIndicSpeakerEmbed

# Downloads model files from HuggingFace automatically
model = PiIndicSpeakerEmbed.from_pretrained("Paytmlabs/Pi-indic-speaker-embed-v1")

# Extract a single embedding
embedding = model.extract("audio.wav")
print(embedding.shape)  # → (512,)

# Extract embeddings for multiple files
embeddings = model.extract_batch(["speaker1.wav", "speaker2.wav", "speaker3.wav"])
print(embeddings.shape)  # → (3, 512)
```

### Command Line

```bash
# Single file
python inference.py --audio audio.wav

# Multiple files, save to .npy
python inference.py --audio spk1.wav spk2.wav spk3.wav --output embeddings.npy
```

### Speaker Verification (cosine similarity)

```python
from inference import PiIndicSpeakerEmbed
import numpy as np

model = PiIndicSpeakerEmbed.from_pretrained("Paytmlabs/Pi-indic-speaker-embed-v1")

emb1 = model.extract("speaker_a_utt1.wav")
emb2 = model.extract("speaker_a_utt2.wav")
emb3 = model.extract("speaker_b_utt1.wav")

# Cosine similarity (higher = more similar)
sim_same = np.dot(emb1, emb2)    # same speaker, different utterance
sim_diff = np.dot(emb1, emb3)    # different speakers

print(f"Same speaker similarity:  {sim_same:.4f}")
print(f"Diff speaker similarity:  {sim_diff:.4f}")
# Same speaker similarity:  0.85+ (typically)
# Diff speaker similarity:  0.40-0.60 (typically)
```

## How It Works

```
Raw audio (16kHz)
  → 3 random crops × 96,000 samples (6s each)          [Preprocessor]
  → WavLM Large (frozen) + LoRA (layers 9-24)          [Frontend]
      → 1024-dim frame features @ 50Hz                [multilayer weighted sum]
  → utterance mean normalization                       [utterance_mvn]
  → ECAPA-TDNN (Conv1d + 3 Res2Net blocks)             [Encoder]
      → 1536-dim frame-level embeddings
  → Channel-Attention Statistics Pooling              [Pooling]
      → 3072-dim utterance embedding (weighted mean ⊕ weighted std)
  → RawNet3 Projector (BatchNorm + Linear)            [Projector]
      → 512-dim speaker embedding
  → L2 normalize                                       [Inference]
      → 512-dim unit vector on hypersphere
```

The AAM-Softmax classifier (used during training) is **discarded** after training. At inference, only the encoder + pooling + projector are used to produce embeddings. This means the model can verify **any speaker**, including those never seen during training.

## Files

| File | Description |
|------|-------------|
| `valid.eer.ave_5best_stripped.pth` | Stripped checkpoint (LoRA + ECAPA + pooling + projector, 96 MB) |
| `config.yaml` | Model architecture configuration |
| `inference.py` | Self-contained inference script |
| `lora_s3prl_frontend.py` | Custom WavLM + LoRA frontend |
| `lora_wavlm.py` | LoRA injection utilities |
| `spk_preprocessor_lora.py` | Custom preprocessor |

## Citation

If you use this model, please cite:

```bibtex
@misc{pi-indic-speaker-embed-v1,
  title={Pi-Indic-Speaker-Embed-v1: A 512-dim Speaker Embedding Model for Indic Languages},
  author={Paytmlabs},
  year={2026},
  url={https://huggingface.co/Paytmlabs/Pi-indic-speaker-embed-v1}
}
```

## License

MIT License. The WavLM Large base model follows its own license (`microsoft/wavlm-large`).