Skip to main content

pipecat-behavioralsignals

Behavioral Signal Technologies

Discord readme.io CI Ruff uv PyPI Python Pipecat License Contributor Covenant

Lets a Pipecat voice agent hear how the user speaks, not just what they say. It streams the user's voice to the Behavioral Signals API and adds what it reads, such as emotion, energy and hesitation, to each user turn. The LLM reads it with the user's words.

It uses Pipecat's Real-Time Context Standard (RCS): it is a perception producer. Pipecat's own PipecatToneReader reads the words; this reads the voice.

Maintained by Behavioral Signal Technologies, the company behind the Behavioral Signals API.

Installation

uv add pipecat-behavioralsignals

Or with pip:

pip install pipecat-behavioralsignals

Requires Python 3.12 or later and Pipecat 1.15.

Prerequisites

A Behavioral Signals account. Create one and an API key in the Behavioral Signals portal. Set your client ID (CID) and API key:

export BEHAVIORALSIGNALS_CID=...
export BEHAVIORALSIGNALS_API_KEY=...

You can also pass them to the service as cid and api_key.

The service streams all of the user's input audio for the whole call, silence included, and streaming is billed by the minute. See the portal for prices.

Usage

Put BehavioralSignalsPerceptionService right after your STT service, and turn on expressive conversation:

from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineParams, PipelineWorker

from pipecat_behavioralsignals import BehavioralSignalsPerceptionService

perception = BehavioralSignalsPerceptionService()

pipeline = Pipeline([
    transport.input(),
    stt,
    perception,
    user_aggregator,
    llm,
    tts,
    transport.output(),
    assistant_aggregator,
])

worker = PipelineWorker(pipeline, params=PipelineParams(enable_expressive_conversation=True))

Things to know:

  • It reads the audio that the STT service passes on. STT services do that by default; do not set audio_passthrough=False.
  • It expects 16-bit mono audio, which is what Pipecat transports give by default. On audio with more channels it reports an error and stops reading.
  • A pipeline has one perception producer. Use this instead of PipecatToneReader, not with it.
  • Without enable_expressive_conversation=True, the LLM does not see the readings.

What the LLM sees

At start, the service declares its signals in a PerceptionMetadataFrame. Pipecat uses it to tell the LLM how to read them. Then each user message starts with the reading, like this:

([emotion angry=0.71, neutral=0.20, sad=0.06, happy=0.03],[strength strong=0.64, neutral=0.30, weak=0.06],[positivity negative=0.72, neutral=0.20, positive=0.08],[speaking_rate fast=0.50, normal=0.40, slow=0.10],[hesitation no=0.80, yes=0.20],[engagement engaged=0.55, neutral=0.35, withdrawn=0.10],[intensity value=0.61]) I already told you, I was charged twice.

The signals have the names of the Behavioral Signals tasks:

Signal What it reads Labels
emotion The emotion in the voice happy, angry, sad, neutral
strength Arousal: the energy in the voice strong, weak, neutral
positivity Valence: how positive the tone of voice is positive, negative, neutral
speaking_rate How fast the user speaks, compared to speakers in general fast, slow, normal
hesitation Whether the user sounds hesitant yes, no
engagement Whether the user sounds involved or detached engaged, withdrawn, neutral
intensity How intense the emotion is, whichever emotion it is a value from 0 to 1

Each label has its probability, likeliest first. The API's speaker signals (age and gender) are not sent.

How it works

Behavioral Signals reads the voice in short windows of about 2 seconds, and each reading arrives about 1 to 2.5 seconds after its audio. On each final transcription, the service takes the latest reading of each signal that overlaps the transcription's audio: the audio since the previous final transcription. So a reading usually covers the start or middle of a turn rather than its last words, and a very short turn may have no reading. Then the message has no reading at the start.

Your users' audio

The service sends the user's audio to the Behavioral Signals API. The SDK README says a stream's results are saved there and can be fetched later. Tell your users, and check that this fits your privacy terms.

Known limits

  • The service logs in when the pipeline starts. With wrong or missing credentials it reports one error and turns itself off for the call. The pipeline keeps running, with no readings.
  • If the stream keeps failing later, for example with no credits left or no network, the service reports one error upstream and keeps retrying, in case the cause passes. Turns get no readings meanwhile.
  • If the network drops without closing the connection, the stream can hang for minutes before it fails and a new one opens. Turns get no readings meanwhile.
  • If the API stops reading audio, the service keeps up to about 20 seconds of it. Beyond that it drops the audio and starts a new stream.
  • When the pipeline ends, the service closes its stream. But it cannot stop a call to the API that hangs: the SDK has no way to cancel it. Then the stream's thread lives on until the call returns. It is a daemon thread, so it does not keep the process from exiting.

Running the example

examples/bot.py is a voice agent you talk to in the browser. It uses Deepgram for speech to text and speech, and OpenAI for the LLM.

git clone https://github.com/BehavioralSignalTechnologies/pipecat-behavioralsignals.git
cd pipecat-behavioralsignals
uv sync
cp examples/env.example .env   # then fill in the keys
uv run examples/bot.py

Open http://localhost:7860/client and talk. The bot logs each turn's reading, as Voice: ....

Compatibility

Tested with Pipecat v1.15.0 and behavioralsignals 1.0.0. RCS is an alpha in Pipecat, so its types may change between releases.

Development

uv sync
uv run pytest
uv run ruff check
uv run ruff format --check

The tests need no credentials: the Behavioral Signals client is faked. See CONTRIBUTING.md for more, and SECURITY.md to report a security problem.

License

BSD 2-Clause. See LICENSE.

Metadata

Release files for pipecat-behavioralsignals 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pipecat-behavioralsignals 0.1.0
File Size Uploaded Provenance Actions
pipecat_behavioralsignals-0.1.0.tar.gz 11.0 kB With provenance Details for pipecat_behavioralsignals-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pipecat-behavioralsignals 0.1.0
File Interpreter ABI Platform Actions
pipecat_behavioralsignals-0.1.0-py3-none-any.whl Python 3 none any Details for pipecat_behavioralsignals-0.1.0-py3-none-any.whl

Total release size: 23.8 kB

Release files / pipecat_behavioralsignals-0.1.0.tar.gz

Download URL pipecat_behavioralsignals-0.1.0.tar.gz
Size 11.0 kB
Tags Source
SHA-256 checksum
How to use checksums
42cd53cb9b499b1aa981e138d070ad64fd5be3e58be58a59edcbe0b505cd0bd4
BLAKE2b-256 checksum
How to use checksums
671a537da034509a46bda37151f13706a56d2206cb87925d81fad9dfb55393ec
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Release files / pipecat_behavioralsignals-0.1.0-py3-none-any.whl

Download URL pipecat_behavioralsignals-0.1.0-py3-none-any.whl
Size 12.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b85e9531879f5bdddfbd7720b92e8ecbff7c38d625318b5bbcd8de7645ee5245
BLAKE2b-256 checksum
How to use checksums
3ca194aa9a9aca21bb78951656048fa9f0a6171c29658103355ea7d5c2bff532
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Security for pipecat-behavioralsignals 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for pipecat_behavioralsignals-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page