shakeda/speech-restoration-models

speech-restoration models

Trained weights from github.com/avivshaked/speech-restoration. The code, the evaluation and the full write-up are there. All three models work on mono speech at 44.1 kHz.

Folder	Model	Size
restorer-daps/	Resemble Enhance fine-tuned on DAPS device recordings (last checkpoint, step 37,458 of the fine-tune)	1.42 GB
producer-cleanup/	2.79M-parameter spectral U-Net trained from scratch: unprocessed studio take to the engineer's post-produced version, including breath removal	11 MB
producer-natural/	Same architecture and target, trained with the de-breathed take as input, so its training never teaches breath removal	11 MB

Restorer

Resemble Enhance (MIT) fine-tuned on DAPS recordings made on iPads and iPhones in real rooms, with the studio take (cleanraw) as target. Speakers f8, f10, m8 and m10 were held out of training.

Evaluation on those four speakers in all 12 DAPS device and room conditions, one script each, cut to about 30 s at sentence boundaries (48 clips). WER compares Whisper large-v3's transcript of each output with its transcript of the studio take.

System	Mean WER	WER, two balcony conditions (8 clips)	WER, other rooms (40 clips)	Speaker similarity
untreated input	0.028	0.054	0.023	0.970
off-the-shelf Resemble Enhance	0.089	0.351	0.036	0.968
this model	0.038	0.126	0.020	0.976

Fine-tuning cut the off-the-shelf model's WER by 0.051 (paired bootstrap 95% interval 0.023 to 0.084). It did not make the speech easier to transcribe than the untreated recording: indoors the two are level, and on the balcony this model is worse on all 8 clips. This checkpoint was not selected on any evaluation clips.

Limits: fine-tuned only on DAPS conditions (background noise, room reverberation, device colouration). It does not restore bandwidth loss, clipping or codec damage. Listening checks were done by one unblinded listener.

Producers

Trained on DAPS studio takes with the same four speakers held out. Evaluation on the 20 held-out clips (four speakers, five scripts, about 30 s each), against the engineer's version as target:

System	Log-magnitude L1 to target	Loudness change	Crest factor change
unprocessed input	0.792	+0.0 LU	+0.0 dB
gain matched to the target's loudness	0.660	+4.3 LU	-0.0 dB
best static EQ plus gain (fitted on training speakers)	0.602	+3.5 LU	+0.8 dB
producer-cleanup	0.424	+4.3 LU	-2.6 dB
producer-natural	0.583	+4.6 LU	-1.5 dB
engineer's version		+4.3 LU	-3.2 dB

producer-cleanup was closer to the target than all three baselines on every clip. None of the baselines compresses, and the distance is also the training loss. Each producer's checkpoint was chosen on 4 of these 20 clips; on the other 16, producer-cleanup scored 0.430 against 0.605 to 0.793 for the baselines and was still closer on every clip. producer-natural is meant to keep the breaths the target removes, so its distance to that target is larger by design. WER stayed at or below 0.033 for every system.

Loading

git clone https://github.com/avivshaked/speech-restoration.git && cd speech-restoration
uv sync
uv run python scripts/fetch_resemble_enhance.py   # the restorer loads on top of the released model
uv run python -c "from huggingface_hub import snapshot_download; snapshot_download('shakeda/speech-restoration-models', local_dir='models')"

from voice_cleanup.infer.pretrained import load_producer, load_restorer

restorer = load_restorer("models/restorer-daps/model.safetensors")
producer = load_producer("models/producer-cleanup/model.safetensors")

scripts/eval_heldout.py in the repository reproduces the tables above. In the repository's locked environment, set torch.backends.cudnn.enabled = False before running the restorer on a GPU; the GitHub README explains why.

Licence

These weights are released under CC BY-NC 4.0 because they were trained on DAPS, which is CC BY-NC 4.0:

Gautham J. Mysore, "Can We Automatically Transform Speech Recorded on Common Consumer Devices in Real-World Environments into Professional Production Quality Speech? - A Dataset, Insights, and Challenges", IEEE Signal Processing Letters, Vol. 22, No. 8, August 2015.

The restorer is derived from the Resemble Enhance release, copyright (c) 2023 Resemble AI, used under the MIT licence. The code in the GitHub repository is Apache-2.0.

SHA-256

7a1c81b873364216a7467e94c7131ebae0ab0852162f83e727c0a01a6e98ad00  restorer-daps/model.safetensors
887274d1f07f0a50d765b07ed04eee1148ae9956e97f64ad35495987adf2eca4  producer-cleanup/model.safetensors
48bb49d6bf1c46517287f67b98f24aab50dcca4b2a0f44dfca1ce979c434341f  producer-natural/model.safetensors