openadapt-capture

Record what a person does on a Windows or macOS desktop, then read it back in Python. Capture saves mouse clicks, drags, scrolls, keystrokes, and screen video to one local folder, and pairs every action with the screen frame from the moment it happened.

Use it when you need real demonstrations of GUI work: to build an automation or replay tool, to collect training or test data for a computer-use model, or to study how a task gets done. OpenAdapt uses it to record the demonstrations that it compiles into replayable programs.

Status: maintained. OpenAdapt uses it to record the demonstrations it compiles. Everything stays in a local folder.

Quick start

You need Python 3.10 or later, and before the first recording:

• FFmpeg. Capture writes video with the ffmpeg executable on your PATH. If you don't have one, install FFmpeg with your package manager. On Capture 1.3.0 or later, capture install-ffmpeg can instead install a pinned, checksum-verified build for your user account.
• macOS. In System Settings > Privacy & Security, allow your terminal app under Screen Recording, Input Monitoring, and Accessibility. Then quit and reopen the terminal.
• Windows. Run from a signed-in desktop session.

pip install openadapt-capture
capture record ./my-capture --description "Rename a file"

If FFmpeg or a permission is missing, capture record stops before it records and says what to fix. Otherwise, do the task. To stop, press Ctrl+C in the terminal, or press Ctrl three times in any app. Then look at what you recorded:

capture info ./my-capture

from openadapt_capture import CaptureSession

with CaptureSession.load("./my-capture") as capture:
    for action in capture.actions():
        print(action.timestamp, action.type, action.x, action.y, action.text)
        frame = action.screenshot  # PIL image of the screen at that moment

What a recording contains

my-capture/
├── recording.db                     # actions, raw input events, window metadata (SQLite)
├── oa_recording-*.mp4               # screen video
├── capture-state.json               # whether the session finished and passed its integrity check
├── capture-artifact-manifest.json   # size and SHA-256 of every other file
├── capture-terminal.json            # completion record that covers the manifest
└── *.json                           # profiling data, and sign-in markers if you used them

Each action has a type (click, double-click, drag, scroll, typed text, or keyboard shortcut), screen coordinates, any typed text, and a timestamp. action.screenshot and action.after_screenshot give the frames before and after it.

Where the operating system exposes it, Capture also stores the accessibility element under each action: its role, name, bounds, available actions, and window, from Windows UI Automation or macOS Accessibility. Read it with action.structural_observation. Capture never guesses these fields from pixels. Turn this off with Recorder(..., capture_structural_observations=False).

Record from Python

from openadapt_capture import Recorder

with Recorder("./my-capture", task_description="Rename a file") as recorder:
    recorder.wait_for_ready()
    input("Do the task, then press Enter...")

To record one window instead of the whole desktop, pass window={"owner": "Parallels", "title": None} (substring match), or use capture record --window-owner Parallels. Coordinates are then in that window's own pixels, which is what remote-desktop clients such as RDP and Citrix need. Details: docs/WINDOW_CAPTURE.md.

To control a running recording from another process:

capture status --session-id SESSION_ID
capture stop --session-id SESSION_ID

capture record prints the session ID when it starts. From Python, use status_recording(session_id) and stop_recording(session_id, timeout=60). stop waits until the video and database are written and checked, and returns an error if anything is incomplete.

If the task includes signing in, wrap the sign-in in an authentication handoff. Capture then stops recording the screen, input, and accessibility data until the sign-in ends, and stores only a marker:

handoff = recorder.begin_authentication(
    methods="password_manager",
    requires_user_presence=True,
)
# Sign in.
recorder.end_authentication(handoff, outcome="completed")

See docs/AUTHENTICATION_HANDOFF.md.

Use it with OpenAdapt

To compile a recording into a program that replays the task, install OpenAdapt with this package as an extra. OpenAdapt needs Python 3.10, 3.11, or 3.12.

pip install "openadapt[capture]"
openadapt flow record --backend windows --out recording --task "Rename a file"
openadapt flow compile recording --out bundle --name rename-file

Other platforms, replay, and remote desktops are covered in the desktop recording guide. The compiler accepts left and right clicks, left-button drags, typed text, named keys, key combinations, and scrolling. It rejects a recording that contains anything else, such as a middle click.

FFmpeg

capture install-ffmpeg downloads one pinned FFmpeg build for your platform, checks its SHA-256 against a digest stored in this package, and installs it under your user data folder. Builds exist for macOS (Apple silicon and Intel), Linux x86-64, and Windows x86-64. --dry-run prints the URL, digest, and destination without downloading anything, and capture uninstall-ffmpeg removes the build. The package itself contains no FFmpeg, and Capture downloads nothing unless you run that command.

An FFmpeg you already set up takes priority. Capture looks in this order: Recorder(ffmpeg_path=...) or OPENADAPT_FFMPEG_PATH, then OPENADAPT_DESKTOP_FFMPEG_PATH, then OpenAdapt Desktop's runtime, then your PATH, and the installed build last. Before recording, Capture runs a short encode-and-decode test and refuses to start if it fails.

Microphone narration

capture record --audio also records your voice and transcribes it on this computer. It's off by default. It needs a local transcription backend:

pip install "openadapt-capture[transcribe-fast]"

Capture refuses --audio before it opens the microphone if no local backend is installed, and it deletes the audio after transcription unless you set RECORD_AUDIO_RETAIN_WAVEFORM.

Privacy

A recording can contain anything that was on screen or typed, including passwords and personal data. It stays on your computer. Capture uploads nothing unless you run capture share or set send_profile=True. The privacy extra installs openadapt-privacy but doesn't remove anything from a recording by itself.

Shadow-mode pattern mining, which notices repeated tasks and offers a draft, is off by default, needs the person's acknowledgement to turn on, and keeps no typed values. Its encrypted buffer lives in the unsynced app-data folder and is cleared, with its key destroyed, at least every 24 hours and when shadow mode stops. OpenAdapt never sends shadow mode data off the computer, but OS backup, sync, and endpoint security tools can still copy local files. See docs/SHADOW_MINING.md.

How Capture handles sensitive data, its licensing boundaries, and how releases are qualified are covered in docs/COMPLIANCE.md.

Limits

• Recording needs a signed-in desktop session and the permissions listed in the quick start.
• On Linux, recording needs an X11 session. Capture doesn't record under Wayland. It saves the text of a key only when an application receives the key. After a click or key that no application receives (for example, typing with no window focused), the rest of that recording saves each key's code but not its text. scripts/linux_xvfb_quickstart.sh runs the quick start on Linux in a container.
• Accessibility data appears only where the application exposes it. Across a remote-desktop connection, Capture sees pixels only.
• Changing a display (connecting a monitor, rotating, or changing resolution or scale) ends the recording. Start a new one afterward.
• These need Capture 1.3.0 or later: capture install-ffmpeg, capture status and capture stop (and status_recording and stop_recording), authentication handoffs, action.after_screenshot, and the capture-*.json files. If pip install --upgrade openadapt-capture gives you an earlier version, install from GitHub: pip install "git+https://github.com/OpenAdaptAI/openadapt-capture".
• For web apps, use OpenAdapt's browser recorder. The Chrome extension in experimental/ is an unfinished prototype and isn't part of the package.

Extras

Development

uv sync --extra dev
uv run pytest -m "not slow"

The slow tests record for real, so they need a desktop session and the operating system permissions above:

uv run pytest -m slow

docs/DESIGN.md describes the recording pipeline.

License

MIT