danmor/separated-vocal-restorer

Separated vocal restorer

A small network that restores vocals damaged by source separation, in the log-mel domain of the PC-NSF-HiFiGAN singing vocoder. It is part of SongShift, a tool that changes the key and tempo of whole songs, where it is an experimental option: SongShift downloads this file by itself the first time it is enabled (songshift shift song.mp3 -s 2 --restorer, or "Restore the separated voice" in the web app).

What it does

Vocals separated from a mix keep their overall level but lose low-level detail: breath, consonants, the noise between harmonics. The network takes the log-mel spectrogram of the separated vocal and of the original mix (which still contains what the separator removed) and predicts the clean vocal's log-mel spectrogram as a residual correction. SongShift applies it before re-synthesising the voice at the new pitch.

• Architecture: 1-D conv in → 8 gated residual blocks (dilations 1, 2, 4, 8, twice) → 1×1 head initialised to zero; 3.9 M parameters (config.json).
• Input / output: 128-bin natural-log mel frames of the PC-NSF-HiFiGAN front end (44.1 kHz, n_fft 2048, hop 512, 40–16000 Hz).

Training

• Clean vocals: VocalSet 1.1 (Wilkins et al., 2018; 20 singers, CC BY 4.0).
• Each recording mixed with accompaniment from MUSDB18 preview clips (training half of the clips) at a random SNR in [−6, +6] dB and separated with htdemucs.
• L1 loss on log-mel, 256-frame crops, batch 32, AdamW with OneCycle (peak 1e-3), 8,000 steps.

Evaluation

On the English songs of CSD (a singer never seen in training), mixed with held-out MUSDB18 clips, separated, then pitch-shifted to the key the singer actually sang in the other version (74 cases; DTW-aligned mel-cepstral distortion against the real recording, lower is better):

	MCD
neural vocals, no restorer	44.46
with this restorer	40.40 (better in 100% of cases)
clean (unseparated) input, upper bound	~33.9

In full songs the improvement is hard to hear, because the detail it restores is still present in the other stems; SongShift therefore leaves it off by default. Details and commands: EXPERIMENTS.md (E9) in the SongShift repository.

Licence

CC BY-NC-SA 4.0: the training mixes use MUSDB18 material, which is for non-commercial use.