A community integration for Pipecat

pipecat-effects

pipecat-effects runs an audio effects chain on the output of a pipecat bot. You build the chain from 8 effects, and it adds no latency.

Install

pip install pipecat-effects

Use

EffectsFilter is a pipecat audio filter. Give it to audio_out_filter, the output filter field of pipecat 1.13 and later. Tested with pipecat 1.14.0.

from pipecat.transports.base_transport import TransportParams
from pipecat_effects import (
    AGC,
    Biquad,
    Compressor,
    DeEsser,
    Effects,
    EffectsFilter,
    Limiter,
    Saturation,
)

CHAIN: Effects = (
    AGC(target_lufs=-20.0),
    Biquad(kind="highpass", hz=90.0),
    Biquad(kind="lowshelf", hz=200.0, gain_db=2.5),
    Biquad(kind="peak", hz=3200.0, q=1.2, gain_db=-2.0),
    DeEsser(hz=6500.0, threshold_db=-32.0, ratio=4.0),
    Compressor(threshold_db=-20.0, ratio=3.0, attack_ms=8.0, release_ms=120.0, makeup_db=2.0),
    Saturation(drive=1.2, mix=0.15),
    Limiter(ceiling_db=-1.0, knee_db=3.0),
)


def transport_params() -> TransportParams:
    return TransportParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
        audio_out_filter=EffectsFilter(CHAIN),
    )

examples/bot.py runs one voice bot with that chain. It needs 4 pipecat extras and OPENAI_API_KEY. The wheel has no examples/, so run it from a clone of this repository.

uv run --with "pipecat-ai[runner,webrtc,openai,silero]" examples/bot.py --chain telephone

--chain picks one of the 4 chains in Listen. --bypass runs the session with the chain off. A chain with Reverb runs in FilterMixer, so its tail rings out. --meter runs any chain in FilterMixer, and the log gives the loudness and the true peak of each bot turn.

Filter or mixer

FilterMixer runs the same filter as the output mixer. It adds the meter, and the chain runs through silence.

from pipecat_effects import FilterMixer

TransportParams(
    audio_in_enabled=True,
    audio_out_enabled=True,
    audio_out_channels=1,
    audio_out_mixer=FilterMixer(EffectsFilter(CHAIN), channels=1),
)

Pick audio_out_filter unless you need the meter, a Reverb tail or a chain on another destination.

Listen

One sentence of Cartesia sonic-3.5 speech, 24 kHz mono, run in 40 ms chunks, the stock output chunk of pipecat. examples/chains.py holds each chain. examples/clips.py renders the clips with ffmpeg. GitHub mutes each player at load. Unmute it to listen. On PyPI each player is a link. Loudness is integrated, measured by ffmpeg ebur128.

Unprocessed

https://github.com/user-attachments/assets/9b4a4b80-8168-4380-a12d-b98f9fdcaaa7

No filter. -18.3 LUFS, -1.3 dBTP.

EQ and compression

https://github.com/user-attachments/assets/33bbbafc-3627-416c-84f0-baf398f4922f

The chain in Use: 2.5 dB low shelf at 200 Hz, 2 dB cut at 3.2 kHz, de-esser, 3:1 compressor. -23.2 LUFS, -5.5 dBTP.

Telephone

https://github.com/user-attachments/assets/6fff6b7a-ef53-44a1-b275-5e29ca4ab1bb

300 Hz to 3400 Hz band, 4:1 compressor, drive 3.0 at 0.3 mix. -22.0 LUFS, -5.1 dBTP.

Broadcast

https://github.com/user-attachments/assets/0e336600-fde6-4ddb-8210-bd74be7fe0b4

100 Hz high-pass, de-esser, 4:1 compressor with 0.5 ms attack and 18 dB makeup, +4 dB at 3.5 kHz. -16.3 LUFS, -1.4 dBTP. Against the unprocessed clip: +2.0 LU, and +3.7 dB in the share of power from 2 to 4 kHz.

Room

https://github.com/user-attachments/assets/6cbc794c-5a33-4cb4-a2be-ccfebffe1f79

Schroeder reverb, 400 ms decay, 0.3 mix. -21.2 LUFS, -3.5 dBTP.

Build a chain

• A chain is any sequence of effects. The filter runs them in order and adds no stage.
• Build one filter per session. Each effect holds its state across chunks.
• Put AGC first. It reads the level before any stage changes it.
• If a chain ends without Limiter, the int16 cast hard clips each sample over full scale. End each chain with Limiter.
• Keep audio_out_channels at 1, the default. Give FilterMixer that value as channels.
• Each effect validates its values at build time. A value outside the range raises ValueError that names the field and the range.

Effects

mix sets the dry/wet ratio. 0 gives only the input, the dry signal. 1 gives only the processed signal, the wet signal.

Method of each effect:

Control

Change the chain at runtime with 2 stock frames.

from pipecat.frames.frames import FilterEnableFrame, FilterUpdateSettingsFrame
from pipecat_effects import Gain, Limiter

await worker.queue_frame(FilterEnableFrame(enable=False))
await worker.queue_frame(
    FilterUpdateSettingsFrame(settings={"effects": (Gain(db=-3.0), Limiter())})
)

• FilterEnableFrame(enable=False) bypasses the chain. The filter gives the input, and the chain keeps running on it, so envelopes and delay lines stay current.
• FilterUpdateSettingsFrame with an effects key builds a new chain and swaps it in.
• Each change fades over one chunk, from the output last heard to the new output. If 2 updates arrive before one chunk, the chain last heard fades to the last chain. The chain between them is not heard.
• The output transport passes the 2 frames to audio_out_filter, in order with the audio.
• With FilterMixer, send MixerEnableFrame and MixerUpdateSettingsFrame with the same fields. FilterMixer maps them to the 2 filter frames. The transport gives filter frames to audio_out_filter only.
• Call EffectsFilter.process_frame with the 2 filter frames when you hold the filter directly.
• An effects value outside a sequence of effects raises TypeError that names the field. An item whose start gives no callable raises TypeError that names the field and the index.
• A chain that fails to build raises ValueError that names the effects field. The old chain keeps running.
• An update before start builds at start. A failed build then raises at start and names the field of the effect, not effects.

Meter

reading = mixer.read()  # Reading(lufs=-19.92, dbtp=-1.04), or None on silence

• The meter runs in FilterMixer. On audio_out_filter there is no reading.
• FilterMixer meters each chunk it gives to the transport, after the int16 cast.
• FilterMixer.read() gives a Reading with lufs and dbtp since the last read, then clears both. Silence gives None.
• lufs is the K-weighted loudness of ITU-R BS.1770, the loudness standard. The published 48 kHz filter moves to the session rate by the bilinear transform. A 997 Hz sine at 0 dBFS reads -3.01 LUFS.
• dbtp is the true peak, on 4 times oversampling with 48 taps.
• Meter reads float chunks for any other caller. Its read gives the floor, -120.0 dB, in place of None.

Cost

One 40 ms chunk at 24 kHz, the stock output chunk of pipecat. The 8 effect chain above, mean of 1000 chunks, median of 4 runs. Python 3.13.9, macOS arm64.

python examples/bench.py --out rows.jsonl --sha <commit> writes one row.

Limits

Chain

• The filter runs on mono. FilterMixer.start raises on more than 1 channel. A filter gets no channel count, so on audio_out_filter stereo output runs through the chain as one interleaved signal, without an error.
• In FilterMixer the chain runs on each chunk, silence included, so attack and release stay continuous through silence. At the stock audio_out_10ms_chunks of 4, one idle session costs 25 silent chunks of 40 ms a second, 7.9 ms of compute a second on the Cost machine.
• On audio_out_filter the chain runs on bot audio only. Each effect holds its state over the gap between turns, and a Reverb tail stops at the last bot chunk.
• On audio_out_filter pipecat applies audio_out_volume and VolumeFrame after the chain. A volume over 1 drives the output past the Limiter ceiling. At a volume of 4 a chain that ends in a -1 dB Limiter writes full scale. In FilterMixer the volume comes before the chain.
• audio_out_filter runs on the default destination only. Audio for a destination in audio_out_destinations skips the chain. audio_out_mixer takes a mapping of destination to mixer, so each destination can get its own FilterMixer.
• A bypassed chain keeps running, so it costs the same as an active chain.
• Outside a bypass, the chunk after an update runs the old chain and the new chain.
• A chunk of 0 samples passes through. An odd byte count raises ValueError that names audio.
• A Biquad or DeEsser centre over 0.45 of the sample rate raises at start, with the highest allowed centre and the rate. The error names hz, not the effect.
• At 8 kHz the highest centre is 3600 Hz. The chain in Use raises there, since its DeEsser sits at 6500 Hz.
• Not included: lookahead, convolution, pitch shift, formant shift. The broadcast clip holds a peak-to-loudness ratio, true peak minus loudness, of 14.9 dB. The unprocessed clip holds 17.0 dB. A hall reverb needs convolution.

Effects

• Compressor and DeEsser set the gain of each 1 ms block from the peak of that block. AGC sets one target per chunk from the loudness at the chunk end. Each gain reads ahead inside its chunk, by up to 1 ms or 1 chunk.
• AGC reads momentary loudness without the gate of BS.1770. Under -50 LUFS the gain holds.
• AGC sets the level at its place in the chain. Later stages move the output level. The EQ and compression clip holds AGC(target_lufs=-20.0) and measures -23.2 LUFS.
• Limiter is a memoryless soft clipper. It has no attack or release, so a signal driven far over the ceiling distorts.
• The -1 dB ceiling leaves the margin for inter-sample peaks at a codec resampler. The true peak meter only reports.
• One section falls 12 dB per octave. A lowpass at 6000 Hz drops an 8 kHz tone by 10.0 dB at a 24 kHz rate.
• Reverb takes decay_ms to 500.

Meter

• The true peak meter cuts at the input Nyquist. At a 24 kHz rate it reads a 10 kHz sine 0.5 dB under its peak.
• The K-weighting at 8 kHz differs from the standard by 0.43 dB at most from 100 Hz to 3 kHz. At 24 kHz it differs by 0.040 dB at most.

Pipecat

• With an output mixer, a flush that fails to drain does not time out. Each mixer chunk reaches the pipeline sink and counts as progress. A flush that drains returns. The defect holds in pipecat 1.14.0 for each output mixer. audio_out_filter needs no mixer.

Develop

• The package ships a py.typed marker. It exports Effect, Apply and Samples for a caller who writes an effect.
• make lint checks the lock, the format, the lint rules, and the types with pyright.
• make test runs the tests. make test-lowest runs them on the lowest allowed pipecat-ai, numpy and scipy. CI runs both.
• make audit checks uv.lock for known vulnerabilities.
• make build builds the wheel and the sdist into dist/. CI runs it on each push to main.
• A v tag that matches the project version publishes the package to PyPI.

License

BSD 2-Clause. Softcery builds and maintains the package.