PocketStation for Python

Capture one desktop application and an optional microphone as separate live audio stems. Use the same native Session for Python model code, browser delivery, and one recording file per stem.

desktop application ─┐
system audio ────────┼─ native Session ─┬─ Python model code
microphone ──────────┤                  ├─ Relay and a browser
generated PCM ───────┘                  └─ separate recording stems

PocketStation runs capture, frame timing, routing, recording, and Relay publication in Rust. Python owns the application and provider code. A slow Python integration cannot run on the operating-system capture callback.

Capture a desktop application

You need Python 3.11 or newer, a supported desktop operating system, and one running application that is producing audio.

python -m pip install pocketstation

import pocketstation

with pocketstation.capture(application="Spotify") as live:
    for frame in live.audio:
        print(frame.source_id, frame.stem_id)

Replace Spotify with the display name or application identifier shown by the operating system. You may also pass a positive process ID. Selection must match one running application; PocketStation does not guess when several processes match.

The context manager starts one native Session and joins it when the block exits. No microphone opens and no file is written unless you request them.

Capture the complete output mix

Use a Session directly when the workflow intentionally needs every sound playing through the computer:

import pocketstation as pks

session = pks.Session()
system_audio = session.capture(pks.Source.system_audio())
system_audio.send(session.polled_audio())

with session.start() as running:
    for frame in running.audio:
        print(frame.source_id, frame.stem_id)

Use Source.application(...) instead when unrelated desktop audio must remain outside the Session.

Add a microphone or recording

import pocketstation

with pocketstation.capture(
    application="Zoom",
    microphone=True,
    record_to="recordings",
) as live:
    for frame in live.audio:
        print(frame.source_id, frame.stem_id)

The application and microphone keep different source and stem identities. Recording writes a separate stem for each source and reports its final result when the Session stops.

Use a microphone device ID instead of True when the application must select a specific input. See capture one application for source discovery, permissions, and asyncio.

Use each source independently

One Session can send a stem to more than one destination:

• iterate over frames in Python;
• transcribe or process audio with an Operator;
• publish audio with a Connector;
• send a named AudioBus through Relay;
• record the original source;
• add generated PCM without mixing it into captured audio.

Every destination receives source, stream, stem, sequence, timestamp, clock, and discontinuity information. Application and microphone audio do not have to be mixed before model processing or remote delivery.

The default Python audio queue holds 32 frames. If Python stops reading and the queue fills, new frames are dropped and the Session reports the queue depth, dropped-frame count, and discontinuity. Other destinations continue according to their own delivery settings.

Build a voice workflow

pocketstation.voice defines the interfaces for streaming transcription, response generation, speech synthesis, speech detection, and duplex voice models. Provider packages implement those interfaces; PocketStation does not embed a model catalog or API key.

Use separate providers when the application chooses each stage:

conversation = session.conversation(
    input=microphone,
    output=assistant,
    stt=transcriber,
    llm=response_model,
    tts=synthesizer,
    vad=speech_detector,
)

Use one duplex provider when it accepts audio and returns audio through one stateful connection:

conversation = session.conversation(
    input=microphone,
    output=assistant,
    voice_model=voice_model,
)

The two forms cannot be combined. Provider capabilities are checked before the Session starts. Generated speech enters the Session through audio_input(), so it can be recorded, published, observed, or removed from pending local output without stopping microphone capture.

This API does not provide model intelligence, acoustic echo cancellation, or a hosted inference service. Provider-side cancellation and receiver playout remain separate observations; PocketStation does not report that a person stopped hearing audio unless the receiver can prove it.

Read compose a voice workflow for transcript revisions, provider capabilities, interruption, retained history, and current limits.

Debug a voice interruption

examples/debug_voice_ai.py shows the complete Session declaration. It connects a physical microphone to OpenAI Realtime, returns assistant PCM to PocketStation, publishes named buses through Relay, captures the browser application playing the assistant, and records the three sources separately.

python -m pip install 'pocketstation[voice-agent-debug]'
export OPENAI_API_KEY='...'
python examples/debug_voice_ai.py

Speak while the assistant is replying. The final report separates provider events from what PocketStation observed:

• microphone capture and continuity;
• transcript and response events reported by the provider;
• assistant PCM accepted by the Session;
• pending output removed after cancellation;
• Relay and browser WebRTC observations;
• the browser application's captured output;
• finalized recording stems.

The example does not prove the exact loudspeaker sample a person heard, AEC, or provider-history truncation. Those fields remain unavailable when the provider or receiver cannot report them. Read the example instructions before running it.

Transcribe both sides of a voice application

The transcription example sends application and microphone audio through one faster-whisper model while keeping the resulting text attributed to its source:

python -m pip install 'pocketstation[transcription]'
python examples/transcribe_voice_app.py

The program asks which running application to inspect. It opens the default microphone because that example explicitly requests both sides. It does not start Relay or record audio.

faster-whisper processes finite audio windows and emits final transcripts. This is batch transcription, not a claim of streaming interim text or voice-agent turn handling. The adapter is example code and imports faster_whisper.WhisperModel only when transcription starts.

Stream any application to a browser

python examples/stream_any_app_audio.py

Choose a running application. The example publishes it as one named AudioBus, waits for Relay readiness, and prints a single-use word code and browser URL. It does not open a microphone or record audio.

The example uses PocketStation's small, rate-limited demonstration services unless you set POCKETSTATION_CONTROL_URL and POCKETSTATION_RELAY_URL to services you operate. Shared service URLs live in pocketstation_demo; they are not repeated in application code. The demonstration is not a hosted service or SLA and can return HTTP 429 when its configured capacity is in use.

Send audio to your own provider

A Connector sends Session audio to an API, socket, file, or provider. Pass one async function when the connection is already open:

import pocketstation as pks
import pocketstation.aio as pks_aio


async def send_audio(frame: pks.AudioFrame) -> None:
    await socket.send(frame.samples)


destination = pks_aio.Connector(send=send_audio)
application.send_to(destination)

Use a class when the Connector opens and closes provider resources:

class WebSocketConnector(pks_aio.Connector):
    def __init__(self, url: str, token: str) -> None:
        self.url = url
        self.token = token

    async def start(self) -> None:
        self.socket = await connect(self.url, token=self.token)

    async def send(self, frame: pks.AudioFrame) -> None:
        await self.socket.send(frame.samples)

    async def stop(self) -> None:
        await self.socket.close()

One Connector object represents one configured provider connection:

destination = WebSocketConnector(url, token)
application.send_to(destination)
microphone.send_to(destination)

PocketStation calls start() once, sends both source-aware stems, and calls stop() once. Each stem retains its own delivery queue and observations. Create another Connector object when credentials, failure handling, or shutdown must be independent.

The integration guide covers deadlines, failures, sync Connectors, Sources, Operators, Endpoints, native extensions, and managed processes. The manifest and driver APIs are for distributable integrations that need typed configuration or custom service status; they are not required for a normal Python Connector.

Write PCM into a Session

Use audio_input() for generated speech, call audio, or decoded network media:

session = pocketstation.Session(recording_root="recordings")
assistant = session.audio_input("assistant")
assistant.output.record("assistant")

with session.start():
    assistant.write(samples)

Core preallocates the input buffers. A write reports whether it was accepted, full, closed, cancelled, or invalid. Read application-owned audio before selecting a queue capacity or cancelling pending output.

Handle permissions and source changes

PocketStation does not prompt during import or discovery. Check microphone permission without prompting with pocketstation.sources.microphone_permission_observation(), then let source opening report the operating system's final result. Application and microphone capture use separate permissions.

If an application or microphone disappears, PocketStation reports the change and does not choose a replacement. Stop the Session, discover sources again, confirm the new selection, and start another Session. Store a discovered source only for its reported persistence scope.

See platform operations for permission states, persistence, recovery, and native dependencies.

Sync and asyncio

pocketstation and pocketstation.aio control the same Rust Session. The asyncio API adds awaitable lifecycle, streams, Relay calls, provider callbacks, and audio writes; it does not create another capture or routing engine.

Python callbacks enter the interpreter and pay Python scheduling and conversion costs. Capture, routing, recording, and the shared Relay Connector remain native. Python-authored model and provider code does not have identical cost to Rust code.

Platform support

Version 0.1.4 uses PocketStation Core 1.1.9 and the shared Relay Connector 0.1.5.

Reading native audio into Python copies samples into Python-owned bytes before exposing a memoryview. The view avoids another Python-side copy; the Rust-to-Python call is not zero-copy.

Continue from the task you have

Develop the SDK

uv sync --extra transcription
uv run pytest -q
uv run ruff check python tests examples
uv run ruff format --check python tests examples
uv run mypy python tests/qualification/typing_contract.py examples

License

PocketStation for Python is available under the MIT or Apache-2.0 license.