OpenAdapt Capture

Lifecycle: Experimental. OpenAdapt Capture records native mouse, keyboard, and screen activity into a time-aligned local capture session. Its current product role is the optional desktop recorder used by openadapt-flow, OpenAdapt's workflow compiler and governed runtime.

Start with the OpenAdapt documentation if you want to record, compile, verify, and replay a workflow.

Where it fits

The browser path stays inside openadapt-flow because the compiler needs ordered before/after frames, page state, secret-field redaction, and events in its own recording format. The extension captures useful DOM context, but it does not provide that end-to-end contract.

Use it with OpenAdapt

Install the compiler with the optional native recorder:

pip install "openadapt-flow[capture]"

Record a desktop demonstration, then compile it:

openadapt-flow record --backend windows --out recording --task "Describe the workflow"
# Perform the workflow, then press Ctrl-C.

openadapt-flow compile recording --out bundle --name my-workflow

Use --backend rdp when recording inside the RDP client pixel space. Replay setup and substrate-specific requirements are documented in the openadapt-flow desktop recording guide.

Use it as a library

Install the capture package directly:

pip install openadapt-capture

Record from the command line:

capture record ./my-capture --description "Describe the workflow"
# Press Ctrl-C to stop.

capture info ./my-capture

Or inspect processed actions in Python:

from openadapt_capture import CaptureSession

with CaptureSession.load("./my-capture") as capture:
    for action in capture.actions():
        print(action.timestamp, action.type, action.x, action.y)
        frame = action.screenshot

A capture normally contains:

my-capture/
├── recording.db
├── oa_recording-*.mp4
└── profiling.json

Audio and individual images are optional.

Window-scoped recording

Status: implemented and unit-proven on all CI platforms; live-validated end to end on macOS (frames, translated coordinates, bounds timeline, and video verified against a real window on a real display). Windows uses a Win32 + mss region grab and is exercised by the same unit suite; its live smoke test awaits an interactive Windows desktop. Not yet validated against a Parallels/Citrix client window specifically.

By default the recorder captures the full screen. Window-scoped mode records ONE window in that window's own pixel space — the mode built for remote-display demonstrations (Parallels, Citrix Workspace, Microsoft Remote Desktop), where openadapt-flow's rdp_window replay drives the client window's pixels directly. Recording scoped to the same window removes the full-screen-vs-window coordinate mismatch at the source:

from openadapt_capture import Recorder

with Recorder(
    "./my-capture",
    task_description="Demonstrate the workflow",
    window={"owner": "Parallels", "title": None},  # substring match
) as recorder:
    input("Perform the task, then press Enter...")

owner matches the application (macOS: window owner name; Windows: process executable name) and title optionally disambiguates among its windows; both are case-insensitive substrings, mirroring how openadapt-flow's remote-display backend identifies the same window at replay time. The selectors can also be set via config/environment (RECORD_WINDOW_OWNER / RECORD_WINDOW_TITLE).

In this mode:

• Frames are the target window's pixels. macOS captures the window's own buffer (CGWindowListCreateImage, the identical call flow's replay uses); Windows grabs the window's screen region, so keep the window unoccluded.
• Input coordinates are translated at capture time into the captured frame's pixel space (pixel = (global_point - window_origin) * scale, the exact inverse of the replay mapping). Input outside the window records out-of-range coordinates rather than being silently clamped.
• The window scoping is persisted: the recording's config JSON carries the target, resolved window, initial bounds, scale, and viewport (CaptureSession.window_capture), and the window is re-resolved every frame with bounds changes recorded as window events — a bounds timeline converters can use to be exact even when the window moves.
• Fail-loud guarantees: recording refuses to start if the window cannot be resolved and captured; input arriving before the first frame is discarded with a warning instead of being recorded in the wrong coordinate space; a mid-recording window resize skips unencodable video frames loudly (screenshots and the bounds timeline stay exact) — avoid resizing the target during a demonstration.

Note for converters: window-mode coordinates are already in captured-frame pixels (coordinate_space == "window_pixels"); do not rescale them by pixel_ratio.

Data and privacy boundary

A raw capture can contain everything visible on screen and everything typed, including credentials, personal data, or protected health information. Keep the entire capture directory inside its approved local boundary and apply an appropriate retention policy.

Capture does not upload a session by default. The sharing command, remote transcription, and profiling transfer are explicit opt-in operations. Installing the privacy extra alone does not automatically scrub a recording.

The current desktop-to-Flow conversion has no field geometry for reliable secret redaction and no live UIA locator. openadapt-flow therefore refuses its desktop --secret option, and converted desktop workflows rely on retained visual evidence unless a separate structural recording path arms them. Review the desktop guide before recording sensitive workflows.

The experimental Chrome extension can observe pages across its configured host permissions and can emit DOM text and keyboard events to a local WebSocket. Treat it as development code; do not deploy it in a sensitive browser profile.

Current limitations

• Native recording requires a visible user session plus the operating system's screen-recording and input-monitoring permissions.
• Desktop capture records pixels and coordinates, not a structural accessibility locator for each demonstrated target.
• The Flow adapter rejects unsupported input such as drag, non-left-click, and modifier-chord actions instead of silently compiling an incomplete workflow.
• Browser-extension installation, security hardening, and compiler integration are not part of the current product path.

See the organization-wide repository lifecycle registry and openadapt-flow product status for the evidence behind current maturity labels.

Optional extras

Development

uv sync --dev
uv run pytest -m "not slow"

Slow native-capture tests require a visible session and operating-system permissions:

uv run pytest -m slow

License

MIT