| --- |
| license: cc-by-nc-4.0 |
| tags: |
| - speech-restoration |
| - speech-enhancement |
| - bandwidth-extension |
| - telephony |
| - call-center |
| - w2v-bert |
| - lora |
| - dac |
| base_model: facebook/w2v-bert-2.0 |
| pipeline_tag: audio-to-audio |
| --- |
| |
| # CallEnhancer |
|
|
| CallEnhancer restores call-centre speech. It takes narrowband, codec'd, noisy phone audio, such as 8 kHz G.711 or GSM calls, |
| and returns clean speech at 48 kHz. |
|
|
| This is **version 2**. Version 1 stays available at the [`v1` tag](https://huggingface.co/Scicom-intl/CallEnhancer/tree/v1). |
|
|
| ## What changed in v2 |
|
|
| - **The full band.** v2's high-band distance to clean speech is 5.5 dB on our 1,289 call-centre clips, against 6.9 dB for v1, and 3.9 dB against 5.3 dB on 58 real calls. Clean studio recordings score 2.0 to 2.3 dB. |
| - **Intelligibility.** On the 58 real calls v2 reads 30.8 % nCER and v1 30.1 %. The mean per-call difference is +1.1 points (95 % CI [-0.9, +3.5]), within noise. v2 is better on 27 calls and worse on 30. |
| - **Deletions.** v2 deletes less speech. It mutes 10.9 % of speech against 13.2 % for v1, at 40.5 % against 41.9 % silence filled: more than a change in loudness balance explains. |
| - **Two parties in one channel.** v2 keeps a quieter second voice. With the two sides of our 58 real calls mixed into one channel, v2 dropped 4.4 % of the quieter party's speech and v1 4.1 %. v2 still drops more than a quarter of it in 4 of 56 calls, v1 in 1. |
| - **Naturalness.** About the same. SOTA-MOS is 2.77 for v2 and 2.70 for v1 (300 clips). Two runs of one recipe differ by 0.07 to 0.08. |
| - **Same size, same files.** A w2v-BERT 2.0 encoder with a LoRA adapter and a 188M DAC decoder. The weights keep v1's paths, so |
| existing download code keeps working. |
|
|
| ## Comparison |
|
|
| We ran CallEnhancer v2, CallEnhancer v1 and three open restoration models on the same call-centre audio, through the same scripts. |
|
|
|  |
|
|
| Each curve is a system's median spectrum, relative to its own 1 to 4 kHz band, on speech-active frames. Black is clean speech. |
| The number in the legend is the high-band distance (HBD): the RMS gap to the black curve from 4 to 20 kHz. Lower is closer to |
| clean speech. |
|
|
| | model | output | HBD, 1,289 clips (dB) ↓ | HBD, 58 real calls (dB) ↓ | nCER, 58 real calls (%) ↓ | speech muted (%) ↓ | silence filled (%) | SOTA-MOS ↑ | DNSMOS overall ↑ | |
| |---|---|---:|---:|---:|---:|---:|---:|---:| |
| | **CallEnhancer v2 (this release)** | 48 kHz | 5.5 | 3.9 | 30.8 | **10.9** | 40.5 | **2.77** | 2.97 | |
| | CallEnhancer v1 | 48 kHz | 6.9 | 5.3 | **30.1** | 13.2 | 41.9 | 2.70 | 2.76 | |
| | sidon-v0.1 | 48 kHz | 4.4 | 5.2 | 30.8 | 28.2 | 20.9 | 2.64 | **3.17** | |
| | VoiceFixer | 44.1 kHz | **3.0** | **2.1** | 50.0 | 27.1 | 52.9 | 2.19 | 2.78 | |
| | Resemble Enhance | 44.1 kHz | 4.8 | 8.6 | 48.8 | 19.5 | 15.7 | 2.28 | 3.05 | |
| | input (telephone audio) | 8 to 16 kHz | 51.3 | 49.2 | 48.8 | | | | | |
|
|
| Bold marks the best system in each column. Silence filled has no best (see How we measured). |
|
|
|  |
|
|
|  |
|
|
|  |
|
|
|  |
|
|
| ### How we measured |
|
|
| - **Test audio.** 1,289 call-centre clips (English, Malay and Mandarin, 9.4 hours) and 58 real calls with human transcripts, one |
| party per channel. Both are customer data; we publish aggregates only. |
| - **Systems.** Each restores the same files with its default settings. 44.1 kHz outputs are resampled to 48 kHz. |
| - **nCER.** [Malaysian Whisper](https://huggingface.co/mesolitica/Malaysian-whisper-large-v3-turbo-v3) against the human transcripts, |
| after LLM normalisation, mean of 3 runs. Each party is restored on its own channel. |
| - **Speech muted.** The share of 20 ms speech frames that the output drops by more than 6 dB. A band-pass that deletes nothing |
| scores 5.9 %. |
| - **Silence filled.** The share of silent input frames that the output raises by more than 6 dB. A counter-metric, not a score: a |
| change in loudness balance trades one against the other, so compare muting at similar silence filled. The shaded band in the |
| deletions figure is v1 with only its balance changed. |
| - **One channel.** Both sides of each real call mixed into one file at their recorded levels (11.4 dB apart at the median). Speech muted is read on the quieter party, at 8 kHz. |
| - **SOTA-MOS** ([HighRateMOS](https://huggingface.co/Scicom-intl/HighRateMOS-VoiceMOS2025)) on 300 clips at 48 kHz. |
| - **DNSMOS** P.835 overall, the median over the clips it can score (1,243 to 1,283 of 1,289). |
| - **Silent outputs.** Resemble Enhance returns silence (under -60 dBFS) on 41 clips; the other systems never do. Speech muted counts these as deletions. |
|
|
| ## Quick start |
|
|
| ```bash |
| pip install torch torchaudio "transformers>=4.56" "descript-audio-codec>=1.0.0" soundfile "huggingface_hub>=0.34" |
| |
| hf download Scicom-intl/CallEnhancer infer_callcentre.py expand_decoder.py \ |
| fe_callcentre/fe_adapter_full.pt decoder_callcentre/decoder_only.pt --local-dir CallEnhancer |
| cd CallEnhancer |
| |
| python infer_callcentre.py --input your_call.wav --out-dir out \ |
| --fe-adapter fe_callcentre/fe_adapter_full.pt \ |
| --decoder decoder_callcentre/decoder_only.pt --device cuda |
| ``` |
|
|
| - `out/your_call_restored48k.wav` is the restored speech at 48 kHz. |
| - `out/your_call_orig48k.wav` is the input, upsampled with no model, for an A/B listen. |
| - `--input` takes a file or a directory (`.wav`, `.flac`, `.mp3`, `.ogg`, `.opus`, `.m4a`). Stereo calls are restored per channel. |
| One channel holding both parties works too: v2 keeps the quieter voice. |
| - The default restores in 4-second windows with a 2-second crossfade. Keep it: the encoder's feature normalisation depends on |
| clip length, and a single pass over a long file drops speech. Use `--device cpu` without a GPU. |
| - v1 stays at the tag `v1`. To run it, keep this script and fetch only v1's weights: |
| `hf download Scicom-intl/CallEnhancer fe_callcentre/fe_adapter_full.pt decoder_callcentre/decoder_only.pt --revision v1 --local-dir v1`, |
| then pass them to `--fe-adapter` and `--decoder`. v1's own script restores in a single pass by default. |
|
|
| ### Python |
|
|
| ```python |
| import numpy as np, soundfile as sf, torch, torchaudio |
| from huggingface_hub import hf_hub_download |
| from transformers import AutoFeatureExtractor |
| import infer_callcentre as ic # from the download above, next to expand_decoder.py |
| |
| torch.set_float32_matmul_precision("medium") # as the command line sets it: same output, bit for bit |
| dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") |
| fe = ic.load_fe(hf_hub_download("Scicom-intl/CallEnhancer", "fe_callcentre/fe_adapter_full.pt"), dev) |
| dec, out_sr = ic.load_decoder(hf_hub_download("Scicom-intl/CallEnhancer", "decoder_callcentre/decoder_only.pt"), dev) |
| proc = AutoFeatureExtractor.from_pretrained("facebook/w2v-bert-2.0") |
| |
| x, sr = sf.read("your_call.wav", dtype="float32", always_2d=True) |
| x16 = torchaudio.functional.resample(torch.from_numpy(x[:, 0])[None], sr, 16000)[0].numpy() # one channel |
| x16 = x16 / (np.abs(x16).max() + 1e-9) * 0.95 |
| y = ic.restore_channel(x16, fe, dec, proc, dev, chunk_s=4.0, bf16=True, out_sr=out_sr) |
| y = y / (np.abs(y).max() + 1e-9) * 0.97 # peak level, as the command line writes it |
| sf.write("restored48k.wav", y, out_sr) |
| ``` |
|
|
| ## Files |
|
|
| | path | what it is | |
| |---|---| |
| | `fe_callcentre/fe_adapter_full.pt` | encoder adapter for inference: 96 LoRA tensors and 48 `output_dense` biases on w2v-BERT 2.0 | |
| | `decoder_callcentre/decoder_only.pt` | decoder for inference: the 188M DAC decoder, 48 kHz | |
| | `decoder_callcentre/last.pt` | the full training state at step 150,000 (decoder, adapter, discriminator, optimisers), to continue training | |
| | `infer_callcentre.py`, `expand_decoder.py` | the inference script and the decoder builder it imports | |
| | `examples/` | five public CallHome calls (TalkBank), each one channel with both parties: the input, and the v2 and v1 restorations | |
| | `assets/` | the figures on this card | |
|
|
| ## Limitations |
|
|
| - The benchmarks are one operator's calls in English, Malay and Mandarin. Other lines and languages may differ. |
| - Sidon-style restoration resynthesises speech. Above 8 kHz the output is generated from the encoder's features, not recovered. |
| - SOTA-MOS mostly judges the band below 8 kHz. Use HBD and the spectra for the band above it. |
| - The test audio is customer data. We publish aggregates only, so the numbers cannot be reproduced outside Scicom. |
|
|
| ## License |
|
|
| CC BY-NC 4.0. Some training noise comes from non-commercial sources. |
|
|
| Built by Scicom (MSC) Berhad. |
|
|
