VoxLock

The best performing voice isolator there is. VoxLock is an open-source voice isolation model: it keeps the caller's voice and removes the other voices in the room before speech-to-text hears them. Made for voice agents and phone calls: one input, the call audio, and it picks out the caller by itself. It streams on a CPU with 24 ms of algorithmic latency.

Live demo · Model on Hugging Face

Results

265 real recordings from the Krisp Voice Isolation Benchmark, transcribed with Parakeet TDT 0.6B v2 and scored with the benchmark's published method. Background words leaked counts words transcribed while only the background was talking, per minute of that audio.

On real AMI meetings not used in training, with the headset wearer as the caller:

Caller words missed, deletions, insertions and 95% intervals: bench/results.

Use

pip install voxlock
voxlock call.wav -o caller.wav

Any sample rate, mono or stereo. The model, 50 MB, downloads from Hugging Face on first use.

import voxlock

vl = voxlock.VoxLock()
call = voxlock.load_audio("call.wav")      # 16 kHz mono float32
caller = vl.isolate(call)                  # same length as the input

VoxLock keeps the voice in a 3-second sample of the caller. isolate takes the loudest 3 seconds in the first 15 seconds of the call, which is how it was benchmarked. Pass sample= to choose it yourself.

Live calls

stream = vl.stream(sample)                 # about 3 s of the caller, 16 kHz mono
for chunk in call_audio:                   # 16 kHz float32 chunks of any size
    send(stream.push(chunk))               # the isolated audio, about 24 ms behind
send(stream.flush())                       # at the end of the call

The stream needs the sample before it starts. In a voice agent, take the first 3 seconds the caller speaks, or a sample saved from an earlier call, and pass the audio through until then. Streaming gives the same output as isolate. With 4 CPU threads on an Apple M2 Pro it runs in real time from 40 ms chunks (VoxLock(threads=4)).

PyTorch

git clone https://github.com/arbazsiddiqui/voxlock && cd voxlock
pip install -r requirements.txt
huggingface-cli download arbazsiddiqui/voxlock model.safetensors config.yaml --local-dir ckpt
python isolate.py --ckpt ckpt --in call.wav --out caller.wav

Training

A 27M causal band-split RNN target-speaker extractor, fine-tuned from the REAL-TSE challenge baseline on a single NVIDIA L4:

REAL-TSE tfmap_context_causal_100
  -> simulated calls        4,000 steps   LibriSpeech callers + background talkers in simulated rooms
  -> real AMI crosstalk     4,000 steps   headset recordings from 40 meetings
  -> long calls (30 s)        800 steps   callers who talk throughout
  -> rooms and distances     400 steps   callers near or far from the mic, background voices dry or in the room, room noise

Repository

Acknowledgements

Built on the REAL-TSE challenge baseline and WeSep by Shuai Wang, Ke Zhang and co-authors, with WeSpeaker's ECAPA model. Trained on LibriSpeech and the AMI Meeting Corpus.

License

Apache-2.0.