README.md

metadata

title: Universal speech enhancement
emoji: 🎧
colorFrom: indigo
colorTo: blue
sdk: gradio
python_version: '3.12'
sdk_version: 6.27.0
app_file: app.py
pinned: false
license: apache-2.0
short_description: One model for noise, reverb, any mics, any rate
tags:
  - espnet
  - speech-enhancement
  - audio-to-audio
  - uses
models:
  - espnet/Wangyou_Zhang_universal_train_enh_uses_refch0_2mem_raw

Universal speech enhancement

The source of the Hugging Face Space espnet/universal-se, one of the demos kept in this repository. USES is a single enhancement model for every condition the task usually splits into separate ones — noise, reverberation, one microphone or several, and any sample rate — trained by the recipe in the directory above this one. It is the model espnet enhance uses by default.

What the app reads from the checkpoint rather than from this file:

from the checkpoint	how the app uses it
the training categories (1ch_48k, 2ch_16k, …)	the rates in the "Process at" menu
the largest channel count those categories name	a warning when a file has more
num_spk	one output player per speaker, so a separation checkpoint reached through ENH_MODEL_TAG grows the column instead of dropping speakers

Multi-channel files keep their channels on the way in, because a model that can use them should be given them; the model returns one channel.

The hosted demo enhances up to 30 seconds and asks ZeroGPU for a matching slice of GPU time; longer audio is trimmed, with a warning saying so. The model itself has no such limit, so run the app yourself for longer recordings.

The card pins python_version: "3.12". espnet requires 3.12 or 3.13, and a Space image with an older interpreter installs no espnet at all - pip reports "No matching distribution found" and the build fails.

requirements.txt asks for espnet[enh], not plain espnet: building this model builds the training criteria with it, and the SI-SNR loss imports fast-bss-eval. Without the extra the model does not load at all.

Running and publishing

pip install -r requirements.txt gradio
python app.py                     # http://127.0.0.1:7860

DEVICE decides where the models run and takes precedence over everything else, so DEVICE=cpu is a local CPU test even on a machine with a GPU. With DEVICE unset, the choice is CUDA when the runtime is ZeroGPU (SPACES_ZERO_GPU, the marker Hugging Face sets — asking torch.cuda.is_available() there answers False, because the GPU is attached only while a @spaces.GPU function runs) or when torch reports a GPU, and CPU otherwise.

To publish, from the ESPnet checkout:

hf auth login
hf upload espnet/universal-se egs2/universal_se_v1/enh1/demo . --repo-type space

The Space itself has to exist as ZeroGPU hardware (zero-a10g), which is a setting on the Space rather than something in these files. This model is the slowest of the five demos on a CPU — around three times real time — so the GPU is what makes it usable.

Citation

@inproceedings{zhang2023uses,
  title={Toward Universal Speech Enhancement for Diverse Input Conditions},
  author={Zhang, Wangyou and Saijo, Kohei and Wang, Zhong-Qiu and
          Watanabe, Shinji and Qian, Yanmin},
  booktitle={Proc. ASRU},
  year={2023}
}