Curry / README.md
massabaali's picture
Curry W2V-BERT 2.0 MFA speaker embedding model (VoxCeleb1-O EER 0.38%)
65c8a73 verified
|
Raw History Blame Contribute Delete
7.43 kB
---
license: cc-by-nc-sa-4.0
language:
- multilingual
tags:
- speaker-verification
- speaker-recognition
- speaker-embedding
- audio
- w2v-bert-2.0
- curriculum-learning
base_model: facebook/w2v-bert-2.0
pipeline_tag: audio-classification
---
# Curry
<p align="center">
<b>CURriculum Ranking Loss for Large-Scale Speaker Verification</b><br>
<i>What and When to Learn</i>
</p>
<p align="center">
<a href="https://arxiv.org/abs/2603.24432"><img src="https://img.shields.io/badge/arXiv-2603.24432-b31b1b.svg" alt="arXiv"></a>
<a href="https://huggingface.co/cmu-mlsp/Curry"><img src="https://img.shields.io/badge/πŸ€—-Model-yellow.svg" alt="HuggingFace"></a>
<a href="https://creativecommons.org/licenses/by-nc-sa/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg" alt="License"></a>
</p>
<p align="center">
πŸ“– Read our <a href="https://arxiv.org/abs/2603.24432">Paper</a> Β·
πŸ’¬ Questions? <a href="mailto:mbaali@cs.cmu.edu">Contact Us</a>
</p>
---
## Introduction
**Curry** (CURriculum Ranking) is an adaptive loss for speaker verification. Fixed-margin losses such as
ArcFace treat every training sample the same, so mislabeled or heavily degraded utterances inject noisy
gradients. Curry estimates each sample's difficulty online from its Sub-center ArcFace confidence,
sorts samples into **easy / medium / hard** tiers using running batch statistics (no extra labels), and
learns tier weights that move training from stable identity learning, to manifold refinement, to
boundary sharpening.
This repository hosts the Curry speaker-embedding model from the paper: a fully fine-tuned
**W2V-BERT 2.0** frontend with an MFA backend, trained on **500K+ speaker identities** (VoxCeleb1/2,
VoxBlink2, CommonVoice), the largest-scale speaker verification system trained to date. It reaches
**0.38% EER on VoxCeleb1-O** and **1.60% EER on SITW**, an **86.8%** and **60.0%** relative EER reduction
over the Sub-center ArcFace baseline.
## Quick Start
### Installation
```bash
pip install torch torchaudio transformers safetensors
```
### Usage
```python
from transformers import AutoModel
import torch
import torchaudio
# Load model (weights + custom code are in this repo)
model = AutoModel.from_pretrained("cmu-mlsp/Curry", trust_remote_code=True)
model.eval()
# Load audio (16 kHz, mono)
waveform, sr = torchaudio.load("audio.wav")
waveform = waveform.mean(dim=0, keepdim=True)
if sr != 16000:
waveform = torchaudio.functional.resample(waveform, sr, 16000)
# Extract the 192-d speaker embedding
with torch.no_grad():
outputs = model(waveform)
speaker_embedding = outputs.embeddings # [batch, 192]
frame_features = outputs.last_hidden_state # [batch, time, 1024]
```
### Speaker Verification Example
```python
import torch.nn.functional as F
def verify_speaker(model, audio1, audio2, threshold=0.35):
"""Check if two 16 kHz waveforms [1, T] are from the same speaker."""
with torch.no_grad():
emb1 = model(audio1).embeddings
emb2 = model(audio2).embeddings
similarity = F.cosine_similarity(emb1, emb2)
return similarity.item() > threshold, similarity.item()
same_speaker, score = verify_speaker(model, waveform1, waveform2)
print(f"Same speaker: {same_speaker}, Score: {score:.4f}")
# or simply, from file paths:
score = model.verify("spk1_a.wav", "spk1_b.wav")
```
The threshold of 0.35 is the EER operating point on VoxCeleb1-O after eval-set mean subtraction; tune it
for your domain.
### Evaluate on VoxCeleb1-O
```bash
python inference.py --trials veri_test.txt --root /path/to/VoxCeleb1/wav/
```
---
## Performance on Benchmarks
### Speaker Verification (EER % ↓ / minDCF ↓)
| Loss | VoxCeleb1-O EER | VoxCeleb1-O minDCF | SITW EER | SITW minDCF |
|------|:---------------:|:------------------:|:--------:|:-----------:|
| Sub-center ArcFace | 2.87 | 0.45 | 4.00 | 0.27 |
| **Curry (this model)** | **0.38** | **0.04** | **1.60** | **0.07** |
The VoxCeleb1-O numbers were re-measured from the released `model.safetensors` with `inference.py`
(cosine scoring, eval-set mean subtraction, minDCF at p_target = 0.001) and match Table 1 of the paper.
### EARS demographic subgroups (EER % ↓)
| Subgroup | Sub-center ArcFace | **Curry** |
|----------|:-----:|:-----:|
| Female (59 spk) | 6.23 | **1.09** |
| Male (43 spk) | 9.95 | **1.84** |
See the paper for the full age Γ— gender breakdown.
---
## Architecture
| Component | Configuration |
|-----------|---------------|
| Frontend | [W2V-BERT 2.0](https://huggingface.co/facebook/w2v-bert-2.0), 24 Conformer layers, 1024-d, **fully fine-tuned** |
| Input | 160-bin log-mel (25 ms window / 10 ms hop, stride 2), computed internally from 16 kHz waveform |
| Layer aggregation | Learnable softmax-weighted sum over all 25 hidden states |
| Backend | MFA residual adapter 1024 β†’ 128 β†’ 1024 |
| Pooling | Attentive Statistics Pooling (2048-d) |
| Head | BatchNorm β†’ Linear(2048, 192) β†’ BatchNorm |
| Embedding | 192-d, cosine scoring |
| Parameters | 581 M |
The training classifier (Sub-center ArcFace, K = 3 sub-centers, 385,986 classes) is not shipped; only
the embedding network is.
---
## Training Details
| Setting | Value |
|---------|-------|
| Loss | Curry over Sub-center ArcFace + inter-top-k (s = 32, K = 3, top-k = 5, mp = 0.06), code in `curry_loss.py` |
| Difficulty tiers | easy: s > ΞΌΜ‚ + ΟƒΜ‚, hard: s < ΞΌΜ‚ βˆ’ ΟƒΜ‚, medium otherwise; ΞΌΜ‚, ΟƒΜ‚ running stats with momentum 0.01 |
| Curriculum | Phase I easy only β†’ Phase II + medium β†’ Phase III all tiers, then tier logits learnable |
| Margin schedule | 0.2 β†’ 0.3 β†’ 0.35 across phases |
| Data | VoxCeleb1 + VoxCeleb2 (7K spk), VoxBlink2 (111K spk), CommonVoice (340K+ spk); 3 s crops, Gaussian noise |
| Optimizer | AdamW, wd 1e-4; lr 5e-6 (frontend), 5e-5 (backend, classifier), 1e-3 (tier logits) |
| Schedule | Linear warm-up + cosine decay |
| Batch | 128 per GPU, effective 1024 |
| Hardware | 8Γ— NVIDIA H100 |
| Precision | fp16 mixed |
---
## Use Cases
- βœ… **Speaker Verification** β€” Verify if two audio samples are from the same speaker
- βœ… **Speaker Identification** β€” Enrol speakers and match by cosine similarity
- βœ… **Speaker Diarization** β€” Cluster segments by speaker embedding
- βœ… **Forensic Audio Analysis** β€” Robust embeddings on degraded, large-scale, multilingual data
---
## Files
| File | Purpose |
|------|---------|
| `model.safetensors` | embedding-network weights (fp32) |
| `config.json` | model config (includes the embedded W2V-BERT 2.0 config, no extra downloads) |
| `configuration_curry.py`, `modeling_curry.py` | `trust_remote_code` model definition |
| `curry_loss.py` | the Curry loss used for training (reference) |
| `inference.py` | pair scoring and VoxCeleb1-O evaluation |
| `veri_test.txt` | VoxCeleb1-O trial list |
---
## Citation
If you find Curry useful, please cite our paper:
```bibtex
@article{baali2026curry,
title={What and When to Learn: {CURriculum} Ranking Loss for Large-Scale Speaker Verification},
author={Baali, Massa and Bisht, Sarthak and Singh, Rita and Raj, Bhiksha},
journal={arXiv preprint arXiv:2603.24432},
year={2026}
}
```
---
## Authors
<p align="center">
<b>Massa Baali</b>, Sarthak Bisht, Rita Singh, Bhiksha Raj<br>
Carnegie Mellon University<br>
πŸ“§ <a href="mailto:mbaali@cs.cmu.edu">mbaali@cs.cmu.edu</a>
</p>