Pi-Indic-Speaker-Embed-v1

A 512-dimensional speaker embedding model optimized for Indic languages (Hindi, Bengali, Marathi, Telugu, and 8 more). Built on top of WavLM Large + ECAPA-TDNN with LoRA fine-tuning, designed for cross-lingual voice cloning and speaker verification tasks.

Model Description

Component	Details
Frontend	WavLM Large (frozen base, 315M params) + LoRA adapters on layers 9–24
LoRA scheme	Layers 9–16: rank 8; Layers 17–24: rank 16; on all 6 projections (q, k, v, out, FFN up, FFN down)
Encoder	ECAPA-TDNN (3 Res2Net blocks, model_scale=8, ndim=1024, output=1536)
Pooling	Channel-Attention Statistics Pooling (→ 3072-dim)
Projector	RawNet3 (BatchNorm + Linear → 512-dim)
Embedding dim	512 (L2-normalized)
Total trainable params	~24M (LoRA 3.5M + ECAPA 18M + Pooling 0.8M + Projector 1.6M)
Checkpoint size	96 MB (WavLM base downloaded separately from microsoft/wavlm-large)

Training Details

• Warm-started from espnet/voxcelebs12_ecapa_wavlm_joint (8th epoch)
• Training data: Kathbath (12 Indic languages, 960 speakers) + VoxCeleb1 (full) + VoxCeleb2 (~2k sampled)
• Loss: AAM-Softmax with sub-centers (K=3, k_top=5, margin=0.2, scale=30) — discarded after training
• Optimizer: Adam (lr=1e-4, weight_decay=1e-5), ExponentialLR (gamma=0.87055)
• Training: 30 real epochs (150 mini-epochs), early stopped at epoch 29 (best at epoch 3)
• Checkpoint: Average of top-5 epochs by validation EER (epochs 3, 5, 6, 14, 20)

Performance

Evaluation Set	EER	minDCF	Description
Kathbath Validation (seen conditions, unseen speakers)	1.586%	0.0640	79 speakers, 4 Indic languages
IndicVoices-R Test (unseen conditions, unseen speakers)	4.003%	0.1699	64 speakers, 4 Indic languages
SOTA baseline (voxcelebs12, VoxCeleb-trained)	3.999% / 5.161%	0.1741 / 0.2248	Same eval sets for comparison

The averaged checkpoint (valid.eer.ave_5best.pth) consistently outperforms single-epoch checkpoints on unseen data, confirming that weight averaging reduces overfitting.

Installation

pip install espnet s3prl transformers torch soundfile librosa numpy huggingface_hub

Quick Start

Python API

from inference import PiIndicSpeakerEmbed

# Downloads model files from HuggingFace automatically
model = PiIndicSpeakerEmbed.from_pretrained("Paytmlabs/Pi-indic-speaker-embed-v1")

# Extract a single embedding
embedding = model.extract("audio.wav")
print(embedding.shape)  # → (512,)

# Extract embeddings for multiple files
embeddings = model.extract_batch(["speaker1.wav", "speaker2.wav", "speaker3.wav"])
print(embeddings.shape)  # → (3, 512)

Command Line

# Single file
python inference.py --audio audio.wav

# Multiple files, save to .npy
python inference.py --audio spk1.wav spk2.wav spk3.wav --output embeddings.npy

Speaker Verification (cosine similarity)

from inference import PiIndicSpeakerEmbed
import numpy as np

model = PiIndicSpeakerEmbed.from_pretrained("Paytmlabs/Pi-indic-speaker-embed-v1")

emb1 = model.extract("speaker_a_utt1.wav")
emb2 = model.extract("speaker_a_utt2.wav")
emb3 = model.extract("speaker_b_utt1.wav")

# Cosine similarity (higher = more similar)
sim_same = np.dot(emb1, emb2)    # same speaker, different utterance
sim_diff = np.dot(emb1, emb3)    # different speakers

print(f"Same speaker similarity:  {sim_same:.4f}")
print(f"Diff speaker similarity:  {sim_diff:.4f}")
# Same speaker similarity:  0.85+ (typically)
# Diff speaker similarity:  0.40-0.60 (typically)

How It Works

Raw audio (16kHz)
  → 3 random crops × 96,000 samples (6s each)          [Preprocessor]
  → WavLM Large (frozen) + LoRA (layers 9-24)          [Frontend]
      → 1024-dim frame features @ 50Hz                [multilayer weighted sum]
  → utterance mean normalization                       [utterance_mvn]
  → ECAPA-TDNN (Conv1d + 3 Res2Net blocks)             [Encoder]
      → 1536-dim frame-level embeddings
  → Channel-Attention Statistics Pooling              [Pooling]
      → 3072-dim utterance embedding (weighted mean ⊕ weighted std)
  → RawNet3 Projector (BatchNorm + Linear)            [Projector]
      → 512-dim speaker embedding
  → L2 normalize                                       [Inference]
      → 512-dim unit vector on hypersphere

The AAM-Softmax classifier (used during training) is discarded after training. At inference, only the encoder + pooling + projector are used to produce embeddings. This means the model can verify any speaker, including those never seen during training.

Files

File	Description
valid.eer.ave_5best_stripped.pth	Stripped checkpoint (LoRA + ECAPA + pooling + projector, 96 MB)
config.yaml	Model architecture configuration
inference.py	Self-contained inference script
lora_s3prl_frontend.py	Custom WavLM + LoRA frontend
lora_wavlm.py	LoRA injection utilities
spk_preprocessor_lora.py	Custom preprocessor

Citation

If you use this model, please cite:

@misc{pi-indic-speaker-embed-v1,
  title={Pi-Indic-Speaker-Embed-v1: A 512-dim Speaker Embedding Model for Indic Languages},
  author={Paytmlabs},
  year={2026},
  url={https://huggingface.co/Paytmlabs/Pi-indic-speaker-embed-v1}
}

License

MIT License. The WavLM Large base model follows its own license (microsoft/wavlm-large).