Quentin-Le-Tellier's picture
Add the WavLM converter and the model card
8753e02 verified
|
Raw History Blame Contribute Delete
2.55 kB
---
license: mit
pipeline_tag: audio-to-audio
tags:
- audio
- speech
- vocal-intensity
- speech-synthesis
---
# Vocal intensity conversion with continuous SPL control via adversarial training
Quentin Le Tellier, Albert Rilliard, Olivier Perrotin, Marc Evrard
Submitted to ICASSP 2027.
**[Code](https://github.com/vers-project/vocal-intensity-conversion)** ·
**[Audio samples](https://vers-project.github.io/vocal-intensity-conversion/)**
These are the trained converters of the paper. A converter renders a recorded
utterance at a target vocal intensity, given in dB SPL at 1 m, while preserving
its content and speaker identity. It transforms the features of a frozen speech
encoder, and a pretrained vocoder resynthesizes the speech.
## Models
| folder | encoder/decoder pair | parameters |
|---|---|---|
| `converter-wavlm` | WavLM-Large layer 6 + kNN-VC HiFi-GAN | 2.66 M |
Each folder holds the converter weights (`model.safetensors`) and the
configuration needed to rebuild it (`config.yaml`). The encoder and vocoder are
not included: the code downloads them from their original sources.
## Usage
```bash
git clone https://github.com/vers-project/vocal-intensity-conversion.git
cd vocal-intensity-conversion
uv run --extra cpu --extra hub scripts/convert.py \
--model vers-project/vocal-intensity-conversion \
--subfolder converter-wavlm \
--input speech.wav \
--target-db 70.0 \
--output speech_70dB.wav
```
The first run downloads WavLM-Large and the kNN-VC HiFi-GAN (about 1.3 GB)
into `~/.cache/vic`.
## Training
The converter is a Transformer encoder (six layers, width 128, four heads),
trained as a conditional cycle-consistent GAN without parallel data, on the
training split of [AVID](https://doi.org/10.5281/zenodo.8331897), the Aalto Vocal
Intensity Database (CC BY 4.0). It was trained for 11 400 iterations, as reported
in the paper.
## Limitations
The converter was trained on AVID only. Target intensities outside the range
seen in training, 42.8 to 76.1 dB SPL, were not evaluated.
## Third-party components
- WavLM-Large (Chen et al., 2022) and the HiFi-GAN vocoder of kNN-VC
(Baas et al., 2023), both from https://github.com/bshall/knn-vc (MIT).
## Citation
```bibtex
@misc{letellier2027vocal,
title = {Vocal Intensity Conversion with Continuous {SPL} Control via Adversarial Training},
author = {Le Tellier, Quentin and Rilliard, Albert and Perrotin, Olivier and Evrard, Marc},
year = {2026},
note = {Submitted to ICASSP 2027}
}
```
## Licence
MIT.