VOSSA: Voiceprint Optimization for Streaming Speech Architectures

Pretrained checkpoints for VOSSA, a real-time streaming voice conversion model (Interspeech 2026, Best Student Paper Award).

Paper · arXiv · Code · Demo page

Model description

VOSSA converts speech from a source speaker to a target speaker in real time, from a short reference recording of the target. Instead of relying on an external speaker-verification model, it extracts the speaker embedding from intermediate layers of its own streaming content encoder, pools it with attentive statistics pooling, and trains it jointly with the voice conversion objectives.

• Content encoder: causal CNN + 8-layer transformer, 50 Hz frames, 2 s look-back, factorized VQ trained on HuBERT k-means units (frozen).
• Speaker embedding: from the last CNN layer and every other transformer layer of the content encoder.
• Decoder: time-varying timbre module conditioned on the speaker embedding, mel synthesizer, and a neural vocoder.
• Audio: 16 kHz mono in and out.

Files

File	Size	Used for
config.yaml	<1 KB	names the checkpoints below; read by the loader
vossa_epoch=52-val_loss=0.6670.ckpt	1.09 GB	full VOSSA model (includes content encoder and vocoder weights)
mimivocoder.ckpt	839 MB	vocoder, needed to build the model
mimi_pasrv2_withquant_la4.ckpt	451 MB	content encoder, needed for training only

The checkpoints are PyTorch Lightning checkpoints and include optimizer states for resuming training.

Usage

Install the code from GitHub:

git clone https://github.com/Morris88826/VOSSA.git
cd VOSSA
pip install torch==2.8.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt

Offline conversion (checkpoints download automatically on first use):

import librosa, torch
from src.pretrained import load_vossa

model = load_vossa()
src, _ = librosa.load("source.wav", sr=16000)
tgt, _ = librosa.load("target_reference.wav", sr=16000)

with torch.no_grad():
    to_tensor = lambda a: torch.from_numpy(a).view(1, 1, -1).to(model.device)
    converted = model(to_tensor(src), to_tensor(tgt)).view(-1)[: len(src)].cpu().numpy()

Streaming conversion (60 ms chunks, any input block size):

from src.streaming import VossaStreamer

streamer = VossaStreamer(model, chunk_frames=3, lookahead=0)
session = streamer.session(tgt)          # 16 kHz reference of the target speaker
for block in microphone_blocks:
    play(session.push(block))
play(session.flush())

See the GitHub repository for notebooks and more details.

Citation

@inproceedings{tseng26c_interspeech,
  title     = {{VOSSA: Voiceprint Optimization for Streaming Speech Architectures}},
  author    = {Mu-Ruei Tseng and Waris Quamer and Ghady Nasrallah and Ricardo Gutierrez-Osuna},
  year      = {2026},
  booktitle = {{Interspeech 2026}},
  pages     = {4716--4720},
  doi       = {10.21437/Interspeech.2026-2763},
  issn      = {2958-1796},
}