pipecat-resampler

A small C extension for Python that converts 16-bit PCM audio, mono or with interleaved channels, from one sample rate to another, chunk by chunk. It's made for real-time audio, such as voice pipelines built with Pipecat. Equal chunks in give equal chunks out: 40 ms of audio in always gives 40 ms out, the first chunk included.

from pipecat_resampler import Resampler

resampler = Resampler(24000, 16000)
chunk_16k = resampler.process(chunk_24k)  # bytes in, bytes out
tail = resampler.flush()  # the audio the filter is delaying

stereo = Resampler(48000, 16000, channels=2)
frames_16k = stereo.process(frames_48k)  # interleaved frames in and out

Platforms

The extension uses Python's stable ABI, so one wheel per platform covers Python 3.11 and later.

Why it exists

A real-time voice pipeline has to resample whenever its parts run at different rates. WebRTC audio arrives at 48 kHz while a speech recognizer might want 16 kHz, or a TTS voice that only comes at 24 kHz has to go out on an 8 kHz phone call. When that happens, the conversion runs chunk by chunk, every 10 to 40 ms, for every stream the process is handling.

We wanted a resampler built for that:

• Sinc quality. A band-limited polyphase sinc filter, flat to 90% of the band, with about 90 dB of aliasing rejection.
• Fast. C with SIMD, so one process can carry many streams at once.
• Small. About 0.1 MB installed, with no runtime dependencies.
• Steady output. The same input size gives the same output size on every call, so nothing downstream has to absorb bursts of audio or wait for audio that's held back.
• Low latency. A fixed delay of a few milliseconds.
• Easy to ship. BSD 2-Clause licensed, like Pipecat, with wheels for Linux (glibc 2.17 or later, and Alpine), macOS and Windows.

Each library we tried covers part of this list, and none covered all of it.

Comparison

Each resampler converted 24 kHz noise to 16 kHz in 40 ms chunks, called from Python, on an Intel i9-10920X. Aliasing is how loud a 10 kHz tone comes out after it folds down to 6 kHz; lower is better.

How each one fits this use:

• soxr reaches its quality cheaply by filtering with FFTs over large blocks. Fed small chunks, that means its output arrives in bursts. Its quick setting, QQ, is steady and the fastest option here; it uses cubic interpolation instead of a long filter, so content above the new rate folds back into the speech. soxr is licensed under the LGPL 2.1, and its Linux wheels need glibc 2.27 or later (2.26 on ARM), with none for Alpine.
• samplerate (libsamplerate) is steady after its first chunk and very clean. Its sinc converters compute the filter for each output sample, which handles any ratio and costs about 15 times soxr's CPU here.
• PyAV wraps FFmpeg's resampler, which is steady after its first chunk and fast. It brings FFmpeg along, 127 MB installed, and each call from Python costs about 30 µs whatever the chunk size.
• Speex has a steady, clean resampler. It has no Python binding on PyPI (the speexdsp package covers its echo canceller), so we measured the system library through ctypes. It uses about twice soxr's CPU here, and more at 44.1 kHz, where it computes the filter on the fly.

Performance

Called from Python, in microseconds per chunk, from the same run as the comparison; lower is better.

At 24 → 16 kHz, by chunk size:

At 40 ms chunks, the size Pipecat's output transport sends, by rate pair:

At 24 → 16 kHz, pipecat-resampler is the fastest of the filtered options at every chunk size, and its lead grows as chunks get smaller, because each call costs less. With 40 ms chunks, soxr VHQ is a little faster on 8 → 16 and 48 → 16 kHz, where its multistage FFT design is very cheap. Even pipecat-resampler's slowest case here, 17 µs per 40 ms chunk, is under 0.05% of one core per stream.

Run uv run python scripts/bench.py to measure on your machine. It times soxr, and also samplerate, PyAV and the Speex library when they're installed.

How it works

It's a polyphase Kaiser-windowed sinc filter, the design WebRTC and Speex use. The filter keeps the lower rate's band flat up to 90% of its Nyquist frequency and attenuates aliasing and images by about 90 dB.

Each call works out how many output samples its input is worth and carries any fraction of a sample into the next call. That's what keeps the output steady, for any pair of rates and any chunk size. Instead of holding output back, the filter delays the audio by a few milliseconds; flush() returns that delayed end, and reset() discards it.

With several channels, the audio is interleaved frames, and each channel is filtered on its own with the same filter. The cost grows with the channel count, so stereo costs twice what mono does.

The inner loop picks the fastest kernel the CPU supports when a resampler is created: AVX2 with FMA or SSE2 on x86-64, NEON on ARM64, and plain C elsewhere. Filter phases with symmetric taps add mirrored samples before multiplying, which halves the multiplications. Set PIPECAT_RESAMPLER_KERNEL to generic, sse2, avx2 or neon to force a kernel.

Development

uv sync
uv run pytest
uv run python scripts/bench.py

Wheels are built and tested with cibuildwheel on every pull request and every push to main.

License

pipecat-resampler is released under the BSD 2-Clause License.