---
license: cc-by-nc-4.0
tags:
  - speech-restoration
  - speech-enhancement
  - bandwidth-extension
  - telephony
  - call-center
  - w2v-bert
  - lora
  - dac
base_model: facebook/w2v-bert-2.0
pipeline_tag: audio-to-audio
---

# CallEnhancer

CallEnhancer restores call-centre speech. It takes narrowband, codec'd, noisy phone audio, such as 8 kHz G.711 or GSM calls,
and returns clean speech at 48 kHz.

This is **version 2**. Version 1 stays available at the [`v1` tag](https://huggingface.co/Scicom-intl/CallEnhancer/tree/v1).

## What changed in v2

- **The full band.** v2's high-band distance to clean speech is 5.5 dB on our 1,289 call-centre clips, against 6.9 dB for v1, and 3.9 dB against 5.3 dB on 58 real calls. Clean studio recordings score 2.0 to 2.3 dB.
- **Intelligibility.** On the 58 real calls v2 reads 30.8 % nCER and v1 30.1 %. The mean per-call difference is +1.1 points (95 % CI [-0.9, +3.5]), within noise. v2 is better on 27 calls and worse on 30.
- **Deletions.** v2 deletes less speech. It mutes 10.9 % of speech against 13.2 % for v1, at 40.5 % against 41.9 % silence filled: more than a change in loudness balance explains.
- **Two parties in one channel.** v2 keeps a quieter second voice. With the two sides of our 58 real calls mixed into one channel, v2 dropped 4.4 % of the quieter party's speech and v1 4.1 %. v2 still drops more than a quarter of it in 4 of 56 calls, v1 in 1.
- **Naturalness.** About the same. SOTA-MOS is 2.77 for v2 and 2.70 for v1 (300 clips). Two runs of one recipe differ by 0.07 to 0.08.
- **Same size, same files.** A w2v-BERT 2.0 encoder with a LoRA adapter and a 188M DAC decoder. The weights keep v1's paths, so
  existing download code keeps working.

## Comparison

We ran CallEnhancer v2, CallEnhancer v1 and three open restoration models on the same call-centre audio, through the same scripts.

![Median spectrum on 1,289 call-centre clips](assets/spectrum_clips.png)

Each curve is a system's median spectrum, relative to its own 1 to 4 kHz band, on speech-active frames. Black is clean speech.
The number in the legend is the high-band distance (HBD): the RMS gap to the black curve from 4 to 20 kHz. Lower is closer to
clean speech.

| model | output | HBD, 1,289 clips (dB) ↓ | HBD, 58 real calls (dB) ↓ | nCER, 58 real calls (%) ↓ | speech muted (%) ↓ | silence filled (%) | SOTA-MOS ↑ | DNSMOS overall ↑ |
|---|---|---:|---:|---:|---:|---:|---:|---:|
| **CallEnhancer v2 (this release)** | 48 kHz | 5.5 | 3.9 | 30.8 | **10.9** | 40.5 | **2.77** | 2.97 |
| CallEnhancer v1 | 48 kHz | 6.9 | 5.3 | **30.1** | 13.2 | 41.9 | 2.70 | 2.76 |
| sidon-v0.1 | 48 kHz | 4.4 | 5.2 | 30.8 | 28.2 | 20.9 | 2.64 | **3.17** |
| VoiceFixer | 44.1 kHz | **3.0** | **2.1** | 50.0 | 27.1 | 52.9 | 2.19 | 2.78 |
| Resemble Enhance | 44.1 kHz | 4.8 | 8.6 | 48.8 | 19.5 | 15.7 | 2.28 | 3.05 |
| input (telephone audio) | 8 to 16 kHz | 51.3 | 49.2 | 48.8 | | | | |

Bold marks the best system in each column. Silence filled has no best (see How we measured).

![Median spectrum on 58 real calls](assets/spectrum_calls.png)

![Character error rate on 58 real calls](assets/ncer.png)

![Speech muted against silence filled](assets/deletions.png)

![SOTA-MOS on 300 clips](assets/sotamos.png)

### How we measured

- **Test audio.** 1,289 call-centre clips (English, Malay and Mandarin, 9.4 hours) and 58 real calls with human transcripts, one
  party per channel. Both are customer data; we publish aggregates only.
- **Systems.** Each restores the same files with its default settings. 44.1 kHz outputs are resampled to 48 kHz.
- **nCER.** [Malaysian Whisper](https://huggingface.co/mesolitica/Malaysian-whisper-large-v3-turbo-v3) against the human transcripts,
  after LLM normalisation, mean of 3 runs. Each party is restored on its own channel.
- **Speech muted.** The share of 20 ms speech frames that the output drops by more than 6 dB. A band-pass that deletes nothing
  scores 5.9 %.
- **Silence filled.** The share of silent input frames that the output raises by more than 6 dB. A counter-metric, not a score: a
  change in loudness balance trades one against the other, so compare muting at similar silence filled. The shaded band in the
  deletions figure is v1 with only its balance changed.
- **One channel.** Both sides of each real call mixed into one file at their recorded levels (11.4 dB apart at the median). Speech muted is read on the quieter party, at 8 kHz.
- **SOTA-MOS** ([HighRateMOS](https://huggingface.co/Scicom-intl/HighRateMOS-VoiceMOS2025)) on 300 clips at 48 kHz.
- **DNSMOS** P.835 overall, the median over the clips it can score (1,243 to 1,283 of 1,289).
- **Silent outputs.** Resemble Enhance returns silence (under -60 dBFS) on 41 clips; the other systems never do. Speech muted counts these as deletions.

## Quick start

```bash
pip install torch torchaudio "transformers>=4.56" "descript-audio-codec>=1.0.0" soundfile "huggingface_hub>=0.34"

hf download Scicom-intl/CallEnhancer infer_callcentre.py expand_decoder.py \
    fe_callcentre/fe_adapter_full.pt decoder_callcentre/decoder_only.pt --local-dir CallEnhancer
cd CallEnhancer

python infer_callcentre.py --input your_call.wav --out-dir out \
    --fe-adapter fe_callcentre/fe_adapter_full.pt \
    --decoder decoder_callcentre/decoder_only.pt --device cuda
```

- `out/your_call_restored48k.wav` is the restored speech at 48 kHz.
- `out/your_call_orig48k.wav` is the input, upsampled with no model, for an A/B listen.
- `--input` takes a file or a directory (`.wav`, `.flac`, `.mp3`, `.ogg`, `.opus`, `.m4a`). Stereo calls are restored per channel.
  One channel holding both parties works too: v2 keeps the quieter voice.
- The default restores in 4-second windows with a 2-second crossfade. Keep it: the encoder's feature normalisation depends on
  clip length, and a single pass over a long file drops speech. Use `--device cpu` without a GPU.
- v1 stays at the tag `v1`. To run it, keep this script and fetch only v1's weights:
  `hf download Scicom-intl/CallEnhancer fe_callcentre/fe_adapter_full.pt decoder_callcentre/decoder_only.pt --revision v1 --local-dir v1`,
  then pass them to `--fe-adapter` and `--decoder`. v1's own script restores in a single pass by default.

### Python

```python
import numpy as np, soundfile as sf, torch, torchaudio
from huggingface_hub import hf_hub_download
from transformers import AutoFeatureExtractor
import infer_callcentre as ic          # from the download above, next to expand_decoder.py

torch.set_float32_matmul_precision("medium")                 # as the command line sets it: same output, bit for bit
dev = torch.device("cuda" if torch.cuda.is_available() else "cpu")
fe = ic.load_fe(hf_hub_download("Scicom-intl/CallEnhancer", "fe_callcentre/fe_adapter_full.pt"), dev)
dec, out_sr = ic.load_decoder(hf_hub_download("Scicom-intl/CallEnhancer", "decoder_callcentre/decoder_only.pt"), dev)
proc = AutoFeatureExtractor.from_pretrained("facebook/w2v-bert-2.0")

x, sr = sf.read("your_call.wav", dtype="float32", always_2d=True)
x16 = torchaudio.functional.resample(torch.from_numpy(x[:, 0])[None], sr, 16000)[0].numpy()   # one channel
x16 = x16 / (np.abs(x16).max() + 1e-9) * 0.95
y = ic.restore_channel(x16, fe, dec, proc, dev, chunk_s=4.0, bf16=True, out_sr=out_sr)
y = y / (np.abs(y).max() + 1e-9) * 0.97                       # peak level, as the command line writes it
sf.write("restored48k.wav", y, out_sr)
```

## Files

| path | what it is |
|---|---|
| `fe_callcentre/fe_adapter_full.pt` | encoder adapter for inference: 96 LoRA tensors and 48 `output_dense` biases on w2v-BERT 2.0 |
| `decoder_callcentre/decoder_only.pt` | decoder for inference: the 188M DAC decoder, 48 kHz |
| `decoder_callcentre/last.pt` | the full training state at step 150,000 (decoder, adapter, discriminator, optimisers), to continue training |
| `infer_callcentre.py`, `expand_decoder.py` | the inference script and the decoder builder it imports |
| `examples/` | five public CallHome calls (TalkBank), each one channel with both parties: the input, and the v2 and v1 restorations |
| `assets/` | the figures on this card |

## Limitations

- The benchmarks are one operator's calls in English, Malay and Mandarin. Other lines and languages may differ.
- Sidon-style restoration resynthesises speech. Above 8 kHz the output is generated from the encoder's features, not recovered.
- SOTA-MOS mostly judges the band below 8 kHz. Use HBD and the spectra for the band above it.
- The test audio is customer data. We publish aggregates only, so the numbers cannot be reproduced outside Scicom.

## License

CC BY-NC 4.0. Some training noise comes from non-commercial sources.

Built by Scicom (MSC) Berhad.