README.md

---
license: apache-2.0
tags:
- audio
- speaker-verification
- asv
- speaker-recognition
- speaker-identification
- pytorch
- redimnet2
- voice-biometrics
language:
- en
pipeline_tag: audio-classification
---

# ReDimNet2+: Speaker Verification Model

ReDimNet2+ is a robustness-oriented adaptation of the compact ReDimNet2-B6 backbone for Automatic Speaker Verification (ASV), trained for robustness to channel and codec mismatch (compression, telephony codecs, noise, reverberation). The model uses a ReDimNet2 backbone with a Mel-spectrogram frontend and produces 192-dimensional speaker embeddings.

- **Code (training, eval, Docker submission pipeline):** [github.com/lab260ru/redimnet2-plus](https://github.com/lab260ru/redimnet2-plus)
- **Paper:** *ReDimNet2+ for Robust Speaker Verification under Channel and Codec Shift* (manuscript submitted to ICASSP 2027)

## Results

Evaluated with random 4-second windows and cosine similarity on L2-normalized embeddings (VoxCeleb1, EER %; EER<sub>p</sub> pools O/E/H; EER<sub>ph</sub> is a 26-condition codec/waveform stress test). See the paper for the full protocol and ablations.

| System | EER<sub>o</sub> | EER<sub>e</sub> | EER<sub>h</sub> | EER<sub>p</sub> | EER<sub>ph</sub> |
|---|---:|---:|---:|---:|---:|
| ReDimNet2 baseline (public ckpt.) | 1.601 | 1.725 | 3.039 | 2.420 | 7.214 |
| WeSpeaker CAM++ (best evaluated WeSpeaker ckpt.) | 0.787 | 0.928 | 1.824 | — | — |
| **ReDimNet2+ LMFT (this checkpoint)** | **0.351** | **0.523** | **1.055** | **0.824** | **1.991** |

## Model Architecture

- **Backbone**: ReDimNet2-B6
- **Embedding Dimension**: 192
- **Input**: Mel-spectrograms (64 mel bins, 10ms hop)
- **Sample Rate**: 16 kHz
- **Frontend**: Log-Mel spectrogram with 64 mel bands
- **Pooling**: ASTP (Attentive Statistics Aware Pooling)
- **Normalization**: L2 normalization on embeddings

## Usage

### Installation

```bash
pip install torch torchaudio soundfile omegaconf
```

### Loading the Model

```python
import torch
from huggingface_hub import hf_hub_download
from omegaconf import OmegaConf

# Download model weights
model_path = hf_hub_download("lab260/redimnet2-plus", "pytorch_model_fsdp.bin")

# Load checkpoint
state_dict = torch.load(model_path, map_location="cpu")

# Strip FSDP wrapper prefixes
state_dict = {
    k.replace("_orig_mod.", "").replace("module.", ""): v
    for k, v in state_dict.items()
}

# Model configuration
model_config = {
    "encoder": {
        "_target_": "asv.models.redimnet2.ReDimNet2Encoder",
        "model_name": "b6",
        "pretrained": False,
        "train_type": "lm",
        "dataset": "vox2",
        "strict_load": True,
        "model_overrides": {
            "out_channels": 224,
            "return_2d_output": False
        }
    },
    "bridge": {
        "_target_": "asv.models.bridge.IdentityBridge"
    },
    "classifier": {
        "_target_": "asv.models.classifier.IdentityClassifier"
    }
}
```

### Extracting Embeddings

```python
import torchaudio.transforms as T

def load_audio(path, sr=16000):
    """Load audio file."""
    audio, sr = torchaudio.load(path)
    if audio.shape[0] > 1:
        audio = audio.mean(dim=0)  # Convert to mono
    return audio, sr

def compute_mel_spec(audio, sr=16000):
    """Compute mel-spectrogram."""
    mel_transform = T.MelSpectrogram(
        sample_rate=sr,
        n_fft=512,
        hop_length=160,
        n_mels=64,
        normalized=False
    )
    mel = mel_transform(audio.unsqueeze(0))
    mel = torch.log(mel + 1e-6)
    return mel

def extract_embedding(model, audio_path, device="cpu"):
    """Extract speaker embedding from audio."""
    # Load and preprocess
    audio, sr = load_audio(audio_path)
    mel = compute_mel_spec(audio, sr)

    # Add channel dim
    if mel.ndim == 3:
        mel = mel.unsqueeze(1)

    mel = mel.to(device)
    model = model.to(device)
    model.eval()

    with torch.no_grad():
        embedding = model(mel)
        embedding = torch.nn.functional.normalize(embedding, dim=1)

    return embedding.cpu().squeeze(0)

# Example usage
embedding = extract_embedding(model, "audio.wav")
print(f"Embedding shape: {embedding.shape}")  # [192]
```

### Speaker Verification

```python
def cosine_similarity(e1, e2):
    """Calculate cosine similarity between embeddings."""
    return torch.dot(e1, e2).item()

def verify_speaker(model, audio1_path, audio2_path, threshold=0.79):
    """Verify if two audio samples are from the same speaker."""
    emb1 = extract_embedding(model, audio1_path)
    emb2 = extract_embedding(model, audio2_path)

    similarity = cosine_similarity(emb1, emb2)
    is_same = similarity >= threshold

    return {
        "similarity": similarity,
        "is_same_speaker": is_same,
        "confidence": abs(similarity - threshold)
    }
```

## Model Details

- **File Format**: PyTorch state_dict (binary)
- **File Size**: ~100 MB
- **Framework**: PyTorch 2.11+
- **Key Prefix**: `_orig_mod.` (should be stripped during loading)

## Training

Full recipe and code: [github.com/lab260ru/redimnet2-plus](https://github.com/lab260ru/redimnet2-plus). Summary:
- **Initialization**: public ReDimNet2-B6 checkpoint, backbone architecture unchanged.
- **Data**: 7 corpora, ~63.9k speakers, ~4.6M utterances, ~8,675 h — VoxBlink2, VoxCeleb2, 3D-Speaker, CN-Celeb/CN-Celeb2, TidyVoice, KeSpeech.
- **Augmentation**: filters, additive noise (colored, MUSAN, RawBoost-style), simulated RIRs, speed perturbation, and codec simulation (MP3, Opus, AAC, FLAC, Vorbis, G.723.1, IMA ADPCM, G.722, A-law, μ-law, Speex, AMR-NB at 8/16 kHz), plus SpecAugment/CutMix.
- **Loss Function**: SphereFace2 (one-vs-all)
- **Optimizer**: AdamW, BFloat16, Accelerate + FSDP on 6 GPUs, gradient clipping at 20
- **Recipe**: staged multi-corpus adaptation followed by large-margin fine-tuning (LMFT) on 6-second windows (margin 0.3)

This checkpoint (`pytorch_model_fsdp.bin`) is the final **ReDimNet2+ LMFT** model from the paper.

## Retrieval reranking

The GitHub repo also ships a training-free, graph-based reranking stage (mean-chain + k-NN graph reranking with hubness correction) for open-set retrieval on top of these embeddings — see [`submission/reranking/`](https://github.com/lab260ru/redimnet2-plus/tree/main/submission/reranking). It raises VoxBlink2-subset Pr@k from 0.7413 to 0.7687 without retraining.

## Limitations

- Model expects 16 kHz audio input
- Maximum audio duration: 60 seconds
- EER roughly doubles from VoxCeleb1-E to -H (hard impostors), and the 26-condition codec/waveform stress test (EER<sub>ph</sub>) stays above the clean metrics even after LMFT — see the paper's limitations section
- External comparisons in the paper are shared-protocol, not equal-data comparisons

## Citation

```bibtex
@unpublished{borodin2027redimnet2plus,
  title  = {{ReDimNet2+} for Robust Speaker Verification under Channel and Codec Shift},
  author = {Borodin, Kirill and Kudryavtsev, Vasiliy and Mkrchian, Grach},
  year   = {2026},
  note   = {Manuscript submitted to ICASSP 2027}
}
```

## License

This model is released under the Apache 2.0 license.

## Contact

- Email: kborodin.research@gmail.com
- Telegram: [@korallll_ai](https://t.me/korallll_ai)