| Model | Access | Word error rate % | Background words leaked / min |
|---|---|---|---|
| **VoxLock** | **open** | **4.95** | **0.11** |
| Krisp VI 2.5 Default | paid | 5.30 | 1.52 |
| Krisp VI 2.5 Balanced | paid | 5.32 | 2.02 |
| Krisp VI 2.5 HD | paid | 5.79 | 2.74 |
| Krisp VI 2.5 Lite | paid | 6.21 | 4.52 |
| ai-coustics Voice Focus 2.2 L | paid | 6.84 | 0.66 |
| ai-coustics Voice Focus 2.2 S | paid | 6.91 | 1.00 |
| Sanas Voice Isolation SE 1.0 | paid | 9.38 | 13.76 |
| REAL-TSE causal (VoxLock's starting checkpoint) | open | 9.81 | 13.73 |
| Hush | open | 12.09 | 19.58 |
| DeepFilterNet3 | open | 13.30 | 22.77 |
| TIGER | open | 14.72 | 37.24 |
| No processing | | 23.21 | 66.48 |
| SAM-Audio small (Meta) | open | 33.77 | 21.15 |
| ElevenLabs Voice Isolator\*\* | paid | 43.79\*\* | 144.6\*\* |
265 real recordings from the [Krisp Voice Isolation Benchmark](https://huggingface.co/datasets/Krisp-AI/VoiceIsolation-Benchmark-Dataset), transcribed with Parakeet TDT 0.6B v2 and scored with the benchmark's published method. **Background words leaked** counts words transcribed while only the background was talking, per minute of that audio.
\*\* ElevenLabs Voice Isolator, run on the 7 of these recordings with the most background speech, not all 265. On the same 7, no processing scores 45.82% and 154.3 leaked words per minute, and VoxLock 2.98% and 0.0.
On real AMI meetings not used in training, with the headset wearer as the caller:
| Model | Access | Word error rate % | Background words leaked / min |
|---|---|---|---|
| **VoxLock** | **open** | **10.12** | 2.12 |
| ai-coustics Voice Focus 2.2 S | paid | 11.85 | **1.06** |
| ai-coustics Voice Focus 2.2 L | paid | 12.13 | 1.72 |
| Sanas Voice Isolation SE 1.0 | paid | 13.30 | 5.30 |
| TIGER | open | 22.84 | 36.85 |
| DeepFilterNet3 | open | 26.78 | 38.84 |
| Hush | open | 33.00 | 68.93 |
| No processing | | 33.03 | 78.34 |
Caller words missed, deletions, insertions and 95% intervals: [bench/results](https://github.com/arbazsiddiqui/voxlock/tree/main/bench/results).
## Use
```bash
pip install voxlock
voxlock call.wav -o caller.wav
```
Any sample rate, mono or stereo. The model downloads from Hugging Face on first use: 75 MB, the 8-bit version (see [Quantized models](#quantized-models)).
```python
import voxlock
vl = voxlock.VoxLock() # precision="fp32" for the benchmarked model
call = voxlock.load_audio("call.wav") # 16 kHz mono float32
caller = vl.isolate(call) # same length as the input
```
VoxLock keeps the voice in a 3-second sample of the caller. `isolate` takes the loudest 3 seconds in the first 15 seconds of the call, which is how it was benchmarked. Pass `sample=` to choose it yourself.
### Streaming
Push the call in as it arrives, as 16 kHz float32 chunks of any size. Each `push` returns the isolated audio that is ready, about 24 ms behind the input.
```python
vl = voxlock.VoxLock(threads=4)
stream = vl.live() # finds the caller by itself
for chunk in call_audio: # e.g. 40 ms = 640 samples
send(stream.push(chunk))
send(stream.flush()) # at the end of the call
```
`live()` passes the audio through unchanged until it has heard 3 seconds of speech, takes that speech as the voice sample and isolates from then on. On our test calls it switched over a median of 3.6 seconds in.
If you already have a sample, from earlier in the call or from a previous call, isolation starts with the first chunk:
```python
stream = vl.stream(sample) # about 3 s of the caller, 16 kHz mono
```
Streaming gives the same output as `isolate` with the same sample. To keep up with a live call, push chunks of 40 ms or more: 40 ms chunks on 4 threads run at 0.69x real time on an Apple M2 Pro and 0.89x on an 8-vCPU Xeon server, and 80 ms chunks on 2 threads at 0.67x on the Xeon. Larger chunks use less CPU and add their length to the delay.
### Pipecat
```bash
pip install "voxlock[pipecat]"
```
```python
from pipecat.transports.base_transport import TransportParams
from voxlock.pipecat import VoxLockFilter
params = TransportParams(audio_in_enabled=True, audio_in_filter=VoxLockFilter())
```
The filter cleans the caller's audio before VAD and speech-to-text, at any transport sample rate (Twilio's 8 kHz included). Like `live()`, it passes audio through until it has heard 3 seconds of the caller; pass `voice_sample=` to isolate from the start. It runs the `int8` model in 80 ms chunks on 2 threads by default (`chunk_ms=`, `threads=`, `precision=`). With a single vCPU, such as Pipecat Cloud's `agent-2x`, use `VoxLockFilter(chunk_ms=160, threads=1)`: 0.73x real time on the Xeon server. Half a vCPU (`agent-1x`) is too little. A full voice agent: [examples/pipecat_bot.py](https://github.com/arbazsiddiqui/voxlock/blob/main/examples/pipecat_bot.py). Tested with Pipecat 1.12.0.
### PyTorch
```bash
git clone https://github.com/arbazsiddiqui/voxlock && cd voxlock
pip install -r requirements.txt
huggingface-cli download arbazsiddiqui/voxlock model.safetensors config.yaml --local-dir ckpt
python isolate.py --ckpt ckpt --in call.wav --out caller.wav
```
## Quantized models
The ONNX model comes in four precisions, all in [`onnx/`](https://huggingface.co/arbazsiddiqui/voxlock/tree/main/onnx). Choose with `VoxLock(precision=...)`, `voxlock --precision ...` or `VoxLockFilter(precision=...)`.
| Precision | Quality | Files | Download | Word error rate % | Background words leaked / min | Real-time factor, 40 ms chunks, 2 / 4 threads |
|---|---|---|---|---|---|---|
| `int8` (default) | 99.8% | `enroll.onnx`, `chunk_int8.onnx` | 75 MB | 5.11 | 0.12 | 0.89 / 0.69 |
| `int8-full` | 99.1% | `enroll.onnx`, `chunk_int8_full.onnx` | 67 MB | 5.85 | 0.04 | 0.67 / 0.54 |
| `fp16` | 100%\* | `enroll_fp16.onnx`, `chunk_fp16.onnx` | 48 MB | 4.95\* | 0.11\* | 1.31 / 1.00 |
| `fp32` | 100% | `enroll.onnx`, `chunk.onnx` | 95 MB | 4.95 | 0.11 | 1.33 / 0.97 |
Quality: the share of `fp32`'s correctly transcribed words each precision keeps, on the 265 Krisp recordings. Real-time factor: streaming on an Apple M2 Pro; below 1 keeps up with a live call.
\* `fp16` is the float32 model with its weights rounded to float16. Its output matches float32 to 45 dB, so it carries the float32 scores. The browser demo uses it.
`enroll` runs once on the voice sample, `chunk` runs on each chunk of the call and carries the LSTM state.
PyTorch: `model.safetensors` and `config.yaml`, 104 MB.
## Training
A 27M causal band-split RNN target-speaker extractor, fine-tuned from the [REAL-TSE challenge baseline](https://github.com/REAL-TSE/wesep-real-tse) on a single NVIDIA L4:
```
REAL-TSE tfmap_context_causal_100
-> simulated calls 4,000 steps LibriSpeech callers + background talkers in simulated rooms
-> real AMI crosstalk 4,000 steps headset recordings from 40 meetings
-> long calls (30 s) 800 steps callers who talk throughout
-> rooms and distances 400 steps callers near or far from the mic, background voices dry or in the room, room noise
```
## Acknowledgements
Built on the REAL-TSE challenge baseline and [WeSep](https://github.com/wenet-e2e/wesep) by Shuai Wang, Ke Zhang and co-authors, with [WeSpeaker](https://github.com/wenet-e2e/wespeaker)'s ECAPA model. Trained on [LibriSpeech](https://www.openslr.org/12) and the [AMI Meeting Corpus](https://groups.inf.ed.ac.uk/ami/corpus/).
## License
Apache-2.0.