|
Download README.md from cmu-mlsp/Curry: direct link, hf CLI and curl.
- Browser
- Download file 7.43 kB
-
https://huggingface.co/cmu-mlsp/Curry/resolve/main/README.md
- Command line
-
hf download hf://cmu-mlsp/Curry/README.md
-
curl -L -o README.md https://huggingface.co/cmu-mlsp/Curry/resolve/main/README.md
7.43 kB
| license: cc-by-nc-sa-4.0 | |
| language: | |
| - multilingual | |
| tags: | |
| - speaker-verification | |
| - speaker-recognition | |
| - speaker-embedding | |
| - audio | |
| - w2v-bert-2.0 | |
| - curriculum-learning | |
| base_model: facebook/w2v-bert-2.0 | |
| pipeline_tag: audio-classification | |
| # Curry | |
| <p align="center"> | |
| <b>CURriculum Ranking Loss for Large-Scale Speaker Verification</b><br> | |
| <i>What and When to Learn</i> | |
| </p> | |
| <p align="center"> | |
| <a href="https://arxiv.org/abs/2603.24432"><img src="https://img.shields.io/badge/arXiv-2603.24432-b31b1b.svg" alt="arXiv"></a> | |
| <a href="https://huggingface.co/cmu-mlsp/Curry"><img src="https://img.shields.io/badge/π€-Model-yellow.svg" alt="HuggingFace"></a> | |
| <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/"><img src="https://img.shields.io/badge/License-CC%20BY--NC--SA%204.0-lightgrey.svg" alt="License"></a> | |
| </p> | |
| <p align="center"> | |
| π Read our <a href="https://arxiv.org/abs/2603.24432">Paper</a> Β· | |
| π¬ Questions? <a href="mailto:mbaali@cs.cmu.edu">Contact Us</a> | |
| </p> | |
| --- | |
| ## Introduction | |
| **Curry** (CURriculum Ranking) is an adaptive loss for speaker verification. Fixed-margin losses such as | |
| ArcFace treat every training sample the same, so mislabeled or heavily degraded utterances inject noisy | |
| gradients. Curry estimates each sample's difficulty online from its Sub-center ArcFace confidence, | |
| sorts samples into **easy / medium / hard** tiers using running batch statistics (no extra labels), and | |
| learns tier weights that move training from stable identity learning, to manifold refinement, to | |
| boundary sharpening. | |
| This repository hosts the Curry speaker-embedding model from the paper: a fully fine-tuned | |
| **W2V-BERT 2.0** frontend with an MFA backend, trained on **500K+ speaker identities** (VoxCeleb1/2, | |
| VoxBlink2, CommonVoice), the largest-scale speaker verification system trained to date. It reaches | |
| **0.38% EER on VoxCeleb1-O** and **1.60% EER on SITW**, an **86.8%** and **60.0%** relative EER reduction | |
| over the Sub-center ArcFace baseline. | |
| ## Quick Start | |
| ### Installation | |
| ```bash | |
| pip install torch torchaudio transformers safetensors | |
| ``` | |
| ### Usage | |
| ```python | |
| from transformers import AutoModel | |
| import torch | |
| import torchaudio | |
| # Load model (weights + custom code are in this repo) | |
| model = AutoModel.from_pretrained("cmu-mlsp/Curry", trust_remote_code=True) | |
| model.eval() | |
| # Load audio (16 kHz, mono) | |
| waveform, sr = torchaudio.load("audio.wav") | |
| waveform = waveform.mean(dim=0, keepdim=True) | |
| if sr != 16000: | |
| waveform = torchaudio.functional.resample(waveform, sr, 16000) | |
| # Extract the 192-d speaker embedding | |
| with torch.no_grad(): | |
| outputs = model(waveform) | |
| speaker_embedding = outputs.embeddings # [batch, 192] | |
| frame_features = outputs.last_hidden_state # [batch, time, 1024] | |
| ``` | |
| ### Speaker Verification Example | |
| ```python | |
| import torch.nn.functional as F | |
| def verify_speaker(model, audio1, audio2, threshold=0.35): | |
| """Check if two 16 kHz waveforms [1, T] are from the same speaker.""" | |
| with torch.no_grad(): | |
| emb1 = model(audio1).embeddings | |
| emb2 = model(audio2).embeddings | |
| similarity = F.cosine_similarity(emb1, emb2) | |
| return similarity.item() > threshold, similarity.item() | |
| same_speaker, score = verify_speaker(model, waveform1, waveform2) | |
| print(f"Same speaker: {same_speaker}, Score: {score:.4f}") | |
| # or simply, from file paths: | |
| score = model.verify("spk1_a.wav", "spk1_b.wav") | |
| ``` | |
| The threshold of 0.35 is the EER operating point on VoxCeleb1-O after eval-set mean subtraction; tune it | |
| for your domain. | |
| ### Evaluate on VoxCeleb1-O | |
| ```bash | |
| python inference.py --trials veri_test.txt --root /path/to/VoxCeleb1/wav/ | |
| ``` | |
| --- | |
| ## Performance on Benchmarks | |
| ### Speaker Verification (EER % β / minDCF β) | |
| | Loss | VoxCeleb1-O EER | VoxCeleb1-O minDCF | SITW EER | SITW minDCF | | |
| |------|:---------------:|:------------------:|:--------:|:-----------:| | |
| | Sub-center ArcFace | 2.87 | 0.45 | 4.00 | 0.27 | | |
| | **Curry (this model)** | **0.38** | **0.04** | **1.60** | **0.07** | | |
| The VoxCeleb1-O numbers were re-measured from the released `model.safetensors` with `inference.py` | |
| (cosine scoring, eval-set mean subtraction, minDCF at p_target = 0.001) and match Table 1 of the paper. | |
| ### EARS demographic subgroups (EER % β) | |
| | Subgroup | Sub-center ArcFace | **Curry** | | |
| |----------|:-----:|:-----:| | |
| | Female (59 spk) | 6.23 | **1.09** | | |
| | Male (43 spk) | 9.95 | **1.84** | | |
| See the paper for the full age Γ gender breakdown. | |
| --- | |
| ## Architecture | |
| | Component | Configuration | | |
| |-----------|---------------| | |
| | Frontend | [W2V-BERT 2.0](https://huggingface.co/facebook/w2v-bert-2.0), 24 Conformer layers, 1024-d, **fully fine-tuned** | | |
| | Input | 160-bin log-mel (25 ms window / 10 ms hop, stride 2), computed internally from 16 kHz waveform | | |
| | Layer aggregation | Learnable softmax-weighted sum over all 25 hidden states | | |
| | Backend | MFA residual adapter 1024 β 128 β 1024 | | |
| | Pooling | Attentive Statistics Pooling (2048-d) | | |
| | Head | BatchNorm β Linear(2048, 192) β BatchNorm | | |
| | Embedding | 192-d, cosine scoring | | |
| | Parameters | 581 M | | |
| The training classifier (Sub-center ArcFace, K = 3 sub-centers, 385,986 classes) is not shipped; only | |
| the embedding network is. | |
| --- | |
| ## Training Details | |
| | Setting | Value | | |
| |---------|-------| | |
| | Loss | Curry over Sub-center ArcFace + inter-top-k (s = 32, K = 3, top-k = 5, mp = 0.06), code in `curry_loss.py` | | |
| | Difficulty tiers | easy: s > ΞΌΜ + ΟΜ, hard: s < ΞΌΜ β ΟΜ, medium otherwise; ΞΌΜ, ΟΜ running stats with momentum 0.01 | | |
| | Curriculum | Phase I easy only β Phase II + medium β Phase III all tiers, then tier logits learnable | | |
| | Margin schedule | 0.2 β 0.3 β 0.35 across phases | | |
| | Data | VoxCeleb1 + VoxCeleb2 (7K spk), VoxBlink2 (111K spk), CommonVoice (340K+ spk); 3 s crops, Gaussian noise | | |
| | Optimizer | AdamW, wd 1e-4; lr 5e-6 (frontend), 5e-5 (backend, classifier), 1e-3 (tier logits) | | |
| | Schedule | Linear warm-up + cosine decay | | |
| | Batch | 128 per GPU, effective 1024 | | |
| | Hardware | 8Γ NVIDIA H100 | | |
| | Precision | fp16 mixed | | |
| --- | |
| ## Use Cases | |
| - β **Speaker Verification** β Verify if two audio samples are from the same speaker | |
| - β **Speaker Identification** β Enrol speakers and match by cosine similarity | |
| - β **Speaker Diarization** β Cluster segments by speaker embedding | |
| - β **Forensic Audio Analysis** β Robust embeddings on degraded, large-scale, multilingual data | |
| --- | |
| ## Files | |
| | File | Purpose | | |
| |------|---------| | |
| | `model.safetensors` | embedding-network weights (fp32) | | |
| | `config.json` | model config (includes the embedded W2V-BERT 2.0 config, no extra downloads) | | |
| | `configuration_curry.py`, `modeling_curry.py` | `trust_remote_code` model definition | | |
| | `curry_loss.py` | the Curry loss used for training (reference) | | |
| | `inference.py` | pair scoring and VoxCeleb1-O evaluation | | |
| | `veri_test.txt` | VoxCeleb1-O trial list | | |
| --- | |
| ## Citation | |
| If you find Curry useful, please cite our paper: | |
| ```bibtex | |
| @article{baali2026curry, | |
| title={What and When to Learn: {CURriculum} Ranking Loss for Large-Scale Speaker Verification}, | |
| author={Baali, Massa and Bisht, Sarthak and Singh, Rita and Raj, Bhiksha}, | |
| journal={arXiv preprint arXiv:2603.24432}, | |
| year={2026} | |
| } | |
| ``` | |
| --- | |
| ## Authors | |
| <p align="center"> | |
| <b>Massa Baali</b>, Sarthak Bisht, Rita Singh, Bhiksha Raj<br> | |
| Carnegie Mellon University<br> | |
| π§ <a href="mailto:mbaali@cs.cmu.edu">mbaali@cs.cmu.edu</a> | |
| </p> | |