pipecat-behavioralsignals
Lets a Pipecat voice agent hear how the user speaks, not just what they say. It streams the user's voice to the Behavioral Signals API and adds what it reads, such as emotion, energy and hesitation, to each user turn. The LLM reads it with the user's words.
It uses Pipecat's Real-Time Context Standard (RCS): it is a perception producer. Pipecat's own
PipecatToneReader reads the words; this reads the voice.
Maintained by Behavioral Signal Technologies, the company behind the Behavioral Signals API.
Installation
uv add pipecat-behavioralsignals
Or with pip:
pip install pipecat-behavioralsignals
Requires Python 3.12 or later and Pipecat 1.15.
Prerequisites
A Behavioral Signals account. Create one and an API key in the Behavioral Signals portal. Set your client ID (CID) and API key:
export BEHAVIORALSIGNALS_CID=...
export BEHAVIORALSIGNALS_API_KEY=...
You can also pass them to the service as cid and api_key.
The service streams all of the user's input audio for the whole call, silence included, and streaming is billed by the minute. See the portal for prices.
Usage
Put BehavioralSignalsPerceptionService right after your STT service, and turn on expressive
conversation:
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
from pipecat_behavioralsignals import BehavioralSignalsPerceptionService
perception = BehavioralSignalsPerceptionService()
pipeline = Pipeline([
transport.input(),
stt,
perception,
user_aggregator,
llm,
tts,
transport.output(),
assistant_aggregator,
])
worker = PipelineWorker(pipeline, params=PipelineParams(enable_expressive_conversation=True))
Things to know:
- It reads the audio that the STT service passes on. STT services do that by default; do not set
audio_passthrough=False. - It expects 16-bit mono audio, which is what Pipecat transports give by default. On audio with more channels it reports an error and stops reading.
- A pipeline has one perception producer. Use this instead of
PipecatToneReader, not with it. - Without
enable_expressive_conversation=True, the LLM does not see the readings.
What the LLM sees
At start, the service declares its signals in a PerceptionMetadataFrame. Pipecat uses it to tell
the LLM how to read them. Then each user message starts with the reading, like this:
([emotion angry=0.71, neutral=0.20, sad=0.06, happy=0.03],[strength strong=0.64, neutral=0.30, weak=0.06],[positivity negative=0.72, neutral=0.20, positive=0.08],[speaking_rate fast=0.50, normal=0.40, slow=0.10],[hesitation no=0.80, yes=0.20],[engagement engaged=0.55, neutral=0.35, withdrawn=0.10],[intensity value=0.61]) I already told you, I was charged twice.
The signals have the names of the Behavioral Signals tasks:
| Signal | What it reads | Labels |
|---|---|---|
emotion |
The emotion in the voice | happy, angry, sad, neutral |
strength |
Arousal: the energy in the voice | strong, weak, neutral |
positivity |
Valence: how positive the tone of voice is | positive, negative, neutral |
speaking_rate |
How fast the user speaks, compared to speakers in general | fast, slow, normal |
hesitation |
Whether the user sounds hesitant | yes, no |
engagement |
Whether the user sounds involved or detached | engaged, withdrawn, neutral |
intensity |
How intense the emotion is, whichever emotion it is | a value from 0 to 1 |
Each label has its probability, likeliest first. The API's speaker signals (age and gender) are not sent.
How it works
Behavioral Signals reads the voice in short windows of about 2 seconds, and each reading arrives about 1 to 2.5 seconds after its audio. On each final transcription, the service takes the latest reading of each signal that overlaps the transcription's audio: the audio since the previous final transcription. So a reading usually covers the start or middle of a turn rather than its last words, and a very short turn may have no reading. Then the message has no reading at the start.
Your users' audio
The service sends the user's audio to the Behavioral Signals API. The SDK README says a stream's results are saved there and can be fetched later. Tell your users, and check that this fits your privacy terms.
Known limits
- The service logs in when the pipeline starts. With wrong or missing credentials it reports one error and turns itself off for the call. The pipeline keeps running, with no readings.
- If the stream keeps failing later, for example with no credits left or no network, the service reports one error upstream and keeps retrying, in case the cause passes. Turns get no readings meanwhile.
- If the network drops without closing the connection, the stream can hang for minutes before it fails and a new one opens. Turns get no readings meanwhile.
- If the API stops reading audio, the service keeps up to about 20 seconds of it. Beyond that it drops the audio and starts a new stream.
- When the pipeline ends, the service closes its stream. But it cannot stop a call to the API that hangs: the SDK has no way to cancel it. Then the stream's thread lives on until the call returns. It is a daemon thread, so it does not keep the process from exiting.
Running the example
examples/bot.py is a voice agent you talk to in the browser. It uses Deepgram for speech to text and speech, and OpenAI for the LLM.
git clone https://github.com/BehavioralSignalTechnologies/pipecat-behavioralsignals.git
cd pipecat-behavioralsignals
uv sync
cp examples/env.example .env # then fill in the keys
uv run examples/bot.py
Open http://localhost:7860/client and talk. The bot logs each turn's reading, as Voice: ....
Compatibility
Tested with Pipecat v1.15.0 and behavioralsignals 1.0.0. RCS is an alpha in Pipecat, so its types may change between releases.
Development
uv sync
uv run pytest
uv run ruff check
uv run ruff format --check
The tests need no credentials: the Behavioral Signals client is faked. See CONTRIBUTING.md for more, and SECURITY.md to report a security problem.
License
BSD 2-Clause. See LICENSE.
Metadata
Release files for pipecat-behavioralsignals 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| pipecat_behavioralsignals-0.1.0.tar.gz | 11.0 kB | With provenance | Details for pipecat_behavioralsignals-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pipecat_behavioralsignals-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for pipecat_behavioralsignals-0.1.0-py3-none-any.whl |
Total release size: 23.8 kB
Release files / pipecat_behavioralsignals-0.1.0.tar.gz
| Download URL | pipecat_behavioralsignals-0.1.0.tar.gz |
|---|---|
| Size | 11.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
42cd53cb9b499b1aa981e138d070ad64fd5be3e58be58a59edcbe0b505cd0bd4
|
|
BLAKE2b-256 checksum How to use checksums |
671a537da034509a46bda37151f13706a56d2206cb87925d81fad9dfb55393ec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.
Transparency logRelease files / pipecat_behavioralsignals-0.1.0-py3-none-any.whl
| Download URL | pipecat_behavioralsignals-0.1.0-py3-none-any.whl |
|---|---|
| Size | 12.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b85e9531879f5bdddfbd7720b92e8ecbff7c38d625318b5bbcd8de7645ee5245
|
|
BLAKE2b-256 checksum How to use checksums |
3ca194aa9a9aca21bb78951656048fa9f0a6171c29658103355ea7d5c2bff532
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.
Transparency logSecurity for pipecat-behavioralsignals 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for pipecat_behavioralsignals-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.