alphawindow 0.1.6

Windows-first window automation abstractions with pluggable input and output modes.

AlphaWindow

简体中文

AlphaWindow is a Windows-first automation abstraction layer. It exposes one session API for window operations while allowing callers to choose different input and output modes.

AlphaWindow is meant to model target-window automation, not global "computer use" control. The same high-level operations can be backed by no-op inspection, window messages, UI Automation, foreground input, guarded foreground input, or optional hook-assisted virtual input.

Scope

AlphaWindow separates four concerns:

• Window discovery and state: WindowResolver, WindowSelector, and WindowSnapshot.
• Output: state inspection, direct window capture, explicit monitor capture, desktop-region capture, hook telemetry, or no output.
• Input: dry-run, window messages, UIA, global foreground input, guarded foreground input, isolated hook input, or virtualized hook input.
• Safety contracts: profile compatibility, explicit fallback modes, hook requirements, prefetch TTL, and fake backends for deterministic tests.

The project still keeps OS-specific pieces explicit and replaceable, but it now includes native Windows adapters for common test workflows: Win32 window resolution/state, Win32/GDI capture, PostMessageW input, UIA common-control input, pynput foreground input, and SendInput foreground input. Production hook injection, hardware bridges, and app-specific transports remain behind protocols so callers can plug in their own adapters without changing automation code. WGC can capture a native Direct3D surface, read it back to CPU memory, keep a persistent frame cache for low-latency repeated screenshots, and recreate the frame pool when content size changes. Win32/GDI capture can use PrintWindow, window BitBlt, or explicit desktop-region BitBlt.

Supported Today

• Built-in I/O profiles for every declared input and output mode, including inspect, observe, desktop_observe, monitor_observe, dry_run, win32_message, message_only, uia_control, uia_visual, global_input, foreground_visual, guarded_input, guarded_desktop, isolated_observe, isolated_telemetry, virtual_window, virtual_telemetry, and virtual_input_only.
• A WindowSession API for prefetch, capture, observe, click, key_down, key_up, and type_text.
• Generic delegation backends for UIA, state, hook-telemetry, and no-output modes.
• Built-in Win32 resolver and state refresh backends: Win32WindowResolver resolves common WindowSelector fields through native top-level window enumeration, and Win32StateOutputBackend refreshes stale HWND snapshots.
• Built-in input backends for PostMessageW window-message input, UIA common-control input, pynput foreground input, SendInput foreground input, and guarded foreground input with focus/cursor restoration.
• Built-in native desktop-region capture through explicit Win32/GDI GetDC(0) + BitBlt, without using PIL ImageGrab or falling back from failed window capture.
• Compatibility checks so a backend cannot silently fall back to a broader side-effect mode.
• Hook-aware isolated and virtualized input backends that accept pluggable hook managers and virtual input transports.
• Hook/virtual-input helpers (RecordingHookManager, UserModeHookManager, CallbackVirtualInputTransport, JsonSocketVirtualInputTransport, JsonVirtualInputAgent) for tests, examples, user-mode hook installer wrappers, JSON socket IPC, and generic target-side request handling.
• Pluggy-based plugin registry for installed packages to register additional screenshot capture methods, audio capture methods, mouse/keyboard input methods, and optional sandbox providers without adding hardware SDKs or sandbox SDKs to core.
• Deterministic prefetch behavior: snapshots are reused within prefetch_ttl_seconds, rejected when stale, and revalidated through a resolver is_valid hook when provided.
• DPI-aware snapshots and capture requests: WindowSnapshot.dpi is carried into operation normalization and DirectX capture requests.
• Relative point normalization: session.click(0.5, 0.5) resolves against the current target window rectangle.
• HDR-aware DirectX/Windows Graphics Capture model: DirectXCaptureOutputBackend builds capture requests for an injected DirectXCapturePipeline, selects FP16/scRGB for HDR targets, selects BGRA8/sRGB for SDR, and preserves frame DPI and SDR white-level metadata.
• CPU frame conversion: CpuFrame supports FP16/scRGB to SDR BGRA8 tone mapping with configurable ToneMappingConfig / ToneMapper choices, pure-Python PNG export, HDR-preserving AlphaWindow archive export through save_hdr(), richer ColorMetadata, and optional Pillow / numpy output formats selected by CaptureOutputFormat.
• Vision and OCR helpers: find_template() provides deterministic BGRA template matching, and recognize_text() accepts a custom OcrEngine or optional pytesseract integration.
• Wait and accessibility helpers: wait_until(), wait_for_window(), WaitPolicy, and WaitTimeoutError cover polling contracts, while UiaAccessibilityTree and AccessibilityNode expose a small Windows UIA tree-query wrapper for standard controls.
• Experimental native WGC pipeline: WinrtWgcCapturePipeline uses PyWinRT, creates a Direct3D11 device, creates a window or monitor GraphicsCaptureItem, starts a free-threaded WGC frame pool, returns a CaptureFrame containing the native IDirect3DSurface, and can read it back with capture_cpu().
• Explicit monitor capture requests: MonitorSnapshot, CaptureTargetKind.MONITOR, and DirectXCaptureConfig.create_monitor_request() model WGC display capture through create_for_monitor, including HDR/scRGB and DPI metadata for the selected monitor.
• Persistent native WGC frame cache: WinrtWgcFrameCache keeps the WGC session and D3D11 device alive, tracks the newest frame, reports content-size resize events, can recreate the frame pool on resize, can wait for a bounded fresh frame, and can run a before_readback hook while holding that frame so external labels can be sampled against the same image moment.
• Classic native Win32 window capture: Win32WindowCaptureBackend supports explicit win32_print_window and win32_bitblt_window methods, returning CpuFrame without falling back to global screenshots.
• A Windows-focused alphawindow Web UI that starts from the current working directory, opens in pywebview by default, can also run in browser-only mode for Codex or normal browser debugging, attaches a target HWND, previews WGC or classic Win32 captures, accepts custom target capture Hz, exposes audio and video-size controls, and provides a Web menu bar for common target/capture/recording/playback settings.
• A reusable recording layer: RecordingSession, RecordingConfig, RecordingInputCaptureMethod, RecordingAudioCaptureMethod, RecordingMouseMode, RecordingLabelWriter, PynputRecordingControls, read_recording_bundle(), and write_recording_bundle() can write and read session.mp4, session.labels.parquet, optional system_audio.wav, and metadata.json for an attached window.
• PyPI packaging for pip install alphawindow and typed package data via py.typed.
• Examples for sharing one automation script across dry-run, message, foreground, isolated, and virtualized modes, plus a JSON virtual-input agent skeleton built on JsonVirtualInputAgent.
• Tests that cover profile coverage, compatibility failures, prefetch reuse/refresh behavior, hook backend contracts, Win32 resolver/input/capture behavior, UIA/SendInput contracts, DirectX/HDR capture contracts, native WGC dependency and resize/recreate behavior, generic backends, wait helpers, vision/OCR wrappers, accessibility tree querying, CLI preview controller behavior, packaging import behavior, optional Windows integration scaffolding, and the publish workflow.

Screenshot Support Matrix

AlphaWindow treats screenshot mode as an explicit choice. It does not silently fall back from one capture path to another.

Window Targets

Mode	Target	Entry points	Current status	Best fit	Limits
WGC cached window capture	HWND / WindowSnapshot	wgc_window_cached, WinrtWgcFrameCache.start_for(window, request)	Supported as experimental native PyWinRT capture	Repeated screenshots of a normal window, borderless window, or compositor-visible app	Depends on Windows Graphics Capture allowing that window; fails closed on WGC/readback/tone-map errors.
WGC single-shot window capture	HWND / WindowSnapshot	wgc_window_single, WinrtWgcCapturePipeline.capture_cpu(window, request)	Supported as experimental native PyWinRT capture	One-off validation screenshots or simple tooling	Recreates the WGC session per capture, so it is not the preferred high-frequency path.
DirectX capture model	WindowSnapshot	DirectXCaptureConfig.create_request(), DirectXCaptureOutputBackend with an injected pipeline	Supported as an abstraction and contract	External adapters that want AlphaWindow's HDR/DPI/request normalization	The injected pipeline must provide the real capture implementation.
Win32 desktop-region BitBlt capture	Window rectangle in desktop coordinates	desktop_observe, OutputMode.DESKTOP_REGION, Win32DesktopRegionCaptureBackend	Supported through native Win32/GDI	Foreground/compositor-visible targets where the caller explicitly wants the screen rectangle	Captures the visible desktop pixels for that rectangle, so occlusion and unrelated overlays can affect the result. It is an explicit mode, not a fallback from failed window capture.
Win32 PrintWindow capture	HWND / WindowSnapshot	win32_print_window, Win32WindowCaptureBackend(method="win32_print_window")	Supported through pywin32/GDI	Legacy Win32 apps, dialogs, or apps where WGC is not desired	Uses PrintWindow(hwnd, hdc, 2); many GPU/game surfaces may return black, stale, or incomplete pixels.
Win32 BitBlt window capture	HWND / WindowSnapshot	win32_bitblt_window, Win32WindowCaptureBackend(method="win32_bitblt_window")	Supported through pywin32/GDI	Visible windows and direct window DC copying	Uses GetWindowDC + BitBlt(..., SRCCOPY); occlusion/compositor/GPU behavior can limit results.

Fullscreen And Exclusive Targets

Scenario	Recommended mode	Current status	What it can capture	What it cannot guarantee
Borderless fullscreen / fullscreen windowed game visible in the compositor	wgc_monitor_cached / WinrtWgcFrameCache.start_for(monitor, request)	Supported as experimental native monitor WGC capture	The whole monitor containing the attached window, including compositor-visible fullscreen output	It captures unrelated pixels on that monitor and is not target-window isolated.
Fullscreen app where window-target WGC is the wrong abstraction	wgc_monitor_cached	Supported as explicit monitor capture, not fallback	Display output if Windows Graphics Capture exposes the monitor frame	Protected, blocked, or non-composited content can still fail or return unusable frames.
True exclusive fullscreen swap chain	None	Not supported as a guaranteed mode	No guaranteed capture path in AlphaWindow today	AlphaWindow does not bypass exclusive fullscreen restrictions, driver behavior, anti-cheat policy, or protected content.
Secure desktop, protected video, anti-cheat blocked content, or protected process	None	Not supported	Nothing guaranteed	No kernel driver, privilege escalation, protected-process bypass, or anti-cheat bypass is provided.

Input Support Matrix

AlphaWindow uses the same high-level operations for mouse and keyboard input: click, key_down, key_up, and type_text. The selected input backend decides whether those operations become window messages, UI Automation actions, global foreground input, or hook-assisted virtual input.

Mouse And Keyboard Modes

Mode	Entry points / profiles	Mouse behavior	Keyboard behavior	Main device impact	Current status and limits
Dry run	dry_run, inspect, observe, desktop_observe, monitor_observe	Records the requested mouse operation without executing it.	Records the requested keyboard operation without executing it.	None.	Built in through DryRunInputBackend; useful for tests and planning.
Window message input	win32_message, message_only, InputMode.MESSAGE, Win32MessageInputBackend	Built-in backend posts mouse messages to a target HWND in client coordinates.	Built-in backend posts key down/up and WM_CHAR text messages.	Does not need to move the primary cursor when the target accepts messages.	Uses PostMessageW; many games/custom renderers ignore these messages, and send success does not prove the target consumed the input.
UI Automation input	uia_control, uia_visual, InputMode.UIA, UiaControlInputBackend	Built-in UIA backend invokes common controls through invoke/click patterns.	Built-in UIA backend supports value/text operations and simple key events through uiautomation.	Usually avoids primary mouse/keyboard takeover.	Requires optional uiautomation; this is for standard controls, not a general game input path.
Foreground global input	global_input, foreground_visual, InputMode.FOREGROUND, PynputForegroundInputBackend, SendInputForegroundInputBackend	Built-in pynput and SendInput backends move/click the global mouse against the foreground desktop.	Built-in pynput and SendInput backends emit global key events and text.	Affects the active desktop and primary mouse/keyboard stream.	Requires focus/foreground correctness; these are not isolated input modes.
Guarded foreground input	guarded_input, guarded_desktop, InputMode.GUARDED_FOREGROUND, GuardedForegroundInputBackend	Wraps a foreground backend and restores cursor/focus state after the operation.	Wraps a foreground keyboard backend with the same cleanup guard.	Still affects the primary input stream while the operation is running.	Reduces cleanup risk but is not input isolation. Restore failures are still possible under normal Windows foreground restrictions.
Isolated hook input	isolated_observe, isolated_telemetry, InputMode.ISOLATED, IsolatedInputBackend	Hook manager can windowize or suppress real mouse input for the target before optionally delegating to another backend.	Hook manager can windowize or suppress target-local keyboard input before optional delegation.	Intended to reduce or prevent target impact from the user's primary devices, depending on hook implementation.	Hook-aware backend contract is built in; production hook manager/DLL injection is not shipped. Fails closed without hook capability.
Virtualized hook input	virtual_window, virtual_telemetry, virtual_input_only, InputMode.VIRTUALIZED, VirtualizedInputBackend, JsonSocketVirtualInputTransport, NamedPipeVirtualInputTransport, JsonVirtualInputAgent	Sends target-local virtual mouse operations through a VirtualInputTransport after hook injection.	Sends target-local virtual keyboard operations through a VirtualInputTransport after hook injection.	Designed for multi-window automation without taking over the primary mouse/keyboard, if the hook and transport support it.	Backend contract, JSON socket IPC client, named-pipe client, and generic JSON request handler are built in; app-specific delivery is supplied by your backend/agent or injected hook.
Custom delegated input	DelegatingInputBackend with message, uia, foreground, or guarded_foreground mode	Caller supplies the concrete mouse implementation.	Caller supplies the concrete keyboard implementation.	Depends on the chosen custom backend.	Supported extension point for pywin32, pynput, SendInput, UIA, app-specific IPC, or test doubles. Hook modes must use the dedicated hook-aware backends.

Hook Modes

Hook path	Required pieces	What it is for	What it is not
Isolation hook	HookManager plus optional delegated input backend	Blocking, rewriting, or observing target-local mouse/keyboard input so real user input can be isolated from the target.	It is not a complete implementation by itself; the package only defines the contract.
Virtual input hook	HookManager plus VirtualInputTransport	Sending per-window or per-process virtual mouse/keyboard operations without relying on the primary cursor or global keyboard stream.	It is not a kernel driver, anti-cheat bypass, or universal game-input solution.
Hook telemetry	Hook-capable input profile plus OutputMode.HOOK_TELEMETRY	Reading hook-side labels, events, or state that can be paired with capture frames.	It is not OCR/image matching and does not replace screenshot capture.

RecordingHookManager, ExternalHookManager, UserModeHookManager, CallbackVirtualInputTransport, JsonSocketVirtualInputTransport, NamedPipeVirtualInputTransport, create_named_pipe_virtualized_backend, and JsonVirtualInputAgent are included as protocol helpers. They are not kernel drivers, anti-cheat bypasses, or app-specific target-side delivery implementations.

Input Language Utilities

The package also exposes small Windows utilities for keyboard layout and CapsLock state:

Utility	Purpose	Notes
switch_input_language(language="next", hwnd=None)	Requests a target or foreground window to switch input language through WM_INPUTLANGCHANGEREQUEST.	language accepts next, previous, common aliases such as en-US / zh-CN, or a Windows KLID such as 00000409.
set_caps_lock(enabled)	Sets global CapsLock state if it differs from the requested value.	CapsLock is a desktop keyboard toggle, not a target-window-local state.
switch_input_state(language=None, caps_lock=None, toggle_caps_lock=False, hwnd=None)	Convenience wrapper that can switch language and set/toggle CapsLock in one call.	Exposes user32 injection for tests or custom wrappers.

Plugin API

AlphaWindow uses pluggy for optional extension packages. This is the right place for hardware device bridges, proprietary SDK adapters, app-specific capture engines, or experimental input transports that should not become core dependencies. See the formal plugin protocol in docs/plugin-protocol.zh-CN.md.

alphawindow-xxx is only a recommended package naming convention. AlphaWindow does not auto-import packages by name prefix. Automatic discovery only loads the alphawindow.plugins entry point group.

A plugin package exposes methods through the alphawindow.plugins entry point group:

[project.entry-points."alphawindow.plugins"]
my_hardware = "my_hardware_plugin"

The plugin module returns method descriptors from hook implementations:

from alphawindow import Capability, InputMode, OutputMode
from alphawindow.plugins import (
    PluginAudioCaptureMethod,
    PluginCaptureMethod,
    PluginInputMethod,
    PluginSandboxProvider,
    hookimpl,
)

class HardwareCaptureBackend:
    mode = OutputMode.WINDOW_CAPTURE
    capabilities = frozenset({Capability.WINDOW_CAPTURE})

    def __init__(self, *, device_id: str):
        self.device_id = device_id

    def capture(self, target):
        ...

    def observe(self, target):
        return target

class HardwareInputBackend:
    mode = InputMode.FOREGROUND
    capabilities = frozenset({Capability.GLOBAL_INPUT})

    def __init__(self, *, device_id: str):
        self.device_id = device_id

    def perform(self, target, operation):
        ...

class HardwareAudioRecorder:
    metadata = {}

    def __init__(self, *, device_id: str):
        self.device_id = device_id

    def start(self, path, *, sample_rate: int, chunk_frames: int):
        ...

    def pause(self):
        ...

    def resume(self):
        ...

    def close(self):
        ...

@hookimpl
def alphawindow_capture_methods():
    return [
        PluginCaptureMethod(
            name="hardware_capture",
            factory=HardwareCaptureBackend,
            mode=OutputMode.WINDOW_CAPTURE,
            capabilities=frozenset({Capability.WINDOW_CAPTURE}),
        )
    ]

@hookimpl
def alphawindow_audio_capture_methods():
    return [
        PluginAudioCaptureMethod(
            name="hardware_audio",
            factory=HardwareAudioRecorder,
            description="capture audio through a hardware bridge",
        )
    ]

@hookimpl
def alphawindow_input_methods():
    return [
        PluginInputMethod(
            name="hardware_input",
            factory=HardwareInputBackend,
            mode=InputMode.FOREGROUND,
            capabilities=frozenset({Capability.GLOBAL_INPUT}),
        )
    ]

@hookimpl
def alphawindow_sandbox_providers():
    return [
        PluginSandboxProvider(
            name="cua",
            factory=MySandboxProvider,
            description="create a Cua-backed sandbox provider",
        )
    ]

Callers discover installed plugins and instantiate backends by method name:

from alphawindow import discover_plugins

registry = discover_plugins()
output_backend = registry.create_capture_backend("hardware_capture", device_id="cap0")
audio_recorder = registry.create_audio_recorder("hardware_audio", device_id="aud0")
input_backend = registry.create_input_backend("hardware_input", device_id="kbd0")
sandbox_provider = registry.create_sandbox_provider("cua")

Plugin method names must be unique within their method kind. Screenshot/input backends are checked against advertised mode and capabilities before use; audio recorders must expose start(path, sample_rate=..., chunk_frames=...), pause(), resume(), and close(). The desktop tool lists built-in none / system plus installed plugin audio method names in the System audio selector, and --record-audio-method accepts either a built-in name or a plugin method name.

Optional Cua Sandbox Adapter

AlphaWindow core does not depend on Cua and does not start full desktop sandboxes by default. Core only defines the small SandboxConfig, SandboxInfo, SandboxSession, and SandboxProvider protocol surface. Cua support lives in the optional alphawindow_cua adapter package and can also be exposed through PluginSandboxProvider.

Unit tests use injected fake Cua clients. Real Cua integration smoke tests run only when ALPHAWINDOW_RUN_CUA_INTEGRATION=1 is set. Until a concrete Cua client factory is supplied, the default adapter fails closed instead of guessing how to create or control a sandbox.

Built-in Profiles

Profile	Input mode	Output mode	Current behavior
inspect	dry_run	state	Inspect resolver state only.
observe	dry_run	window_capture	Observe/capture without input.
desktop_observe	dry_run	desktop_region	Observe through explicit desktop-region output.
monitor_observe	dry_run	monitor_capture	Observe through explicit monitor capture.
dry_run	dry_run	none	Record requested operations without side effects.
win32_message	message	window_capture	Built-in PostMessageW input plus window capture.
message_only	message	none	Built-in PostMessageW input without output.
uia_control	uia	state	Adapter slot for UIA control operations.
uia_visual	uia	window_capture	Adapter slot for UIA input plus visual capture.
global_input	foreground	desktop_region	Built-in pynput foreground input plus desktop-region output.
foreground_visual	foreground	window_capture	Foreground input with target-window output.
guarded_input	guarded_foreground	window_capture	Built-in foreground input guard with focus/cursor cleanup.
guarded_desktop	guarded_foreground	desktop_region	Built-in guarded foreground input with desktop-region capture.
isolated_observe	isolated	window_capture	Requires a hook-capable input backend; can isolate real mouse input before optional delegated input.
isolated_telemetry	isolated	hook_telemetry	Requires hook support for isolation and telemetry.
virtual_window	virtualized	window_capture	Requires hook support and a virtual input transport.
virtual_telemetry	virtualized	hook_telemetry	Requires hook support for virtual input and telemetry.
virtual_input_only	virtualized	none	Requires hook support; routes virtual input without output capture.

Considered Design Surface

These capabilities are represented in the model and can be supplied by external adapters today:

• Native window resolution by hwnd, title, class name, PID, visibility, DPI, and process/thread metadata.
• Direct window output through built-in PrintWindow and BitBlt, plus extension slots for DWM thumbnails or application-specific capture.
• HDR-aware DirectX/Windows Graphics Capture output with explicit pixel format, color space, buffer count, free-threaded frame-pool preference, DPI, target kind, and SDR white-level metadata.
• Native WGC capture through PyWinRT for callers that want a window or monitor IDirect3DSurface, a tone-mapped CPU image, persistent latest-frame cache for high-frequency capture loops, content-size resize metadata, or automatic frame-pool recreate on resize.
• Monitor-targeted WGC output for cases where the desired image is the visible display containing a target window, including borderless-fullscreen style scenarios where window-target capture is the wrong abstraction.
• Desktop-region output for modes where the target must be foreground or compositor-visible.
• Window-message input for controls that accept PostMessage/SendMessage style input.
• Windows UI Automation input for standard controls.
• Foreground input through built-in pynput and SendInput backends, or equivalent custom global input libraries.
• Guarded foreground input that restores focus/cursor state after an operation.
• Optional hook-assisted mouse and keyboard windowization, including target-local telemetry.
• Per-window or per-process virtual input transports for multi-window automation without taking over the primary mouse.
• Configurable HDR-to-SDR conversion through ToneMappingConfig / ToneMapper, while keeping true color-managed HDR preview outside the core package.
• High-level wait/retry helpers, template matching, OCR engine integration, and UIA accessibility tree querying for standard control workflows.

Not Supported Yet

The package still keeps most native adapters out of core. In particular, it does not currently include:

• Color-managed HDR preview or HDR-preserving video encoders.
• Built-in DLL injection.
• App-specific target-side virtual input delivery beyond the generic JSON request handler.
• Non-Windows native backends.
• Guaranteed capture of true exclusive fullscreen swap chains, protected content, anti-cheat blocked content, secure desktops, or other content Windows/WGC refuses to expose.
• Kernel drivers, privilege escalation, protected-process bypasses, anti-cheat bypasses, or game-specific automation logic.

The legacy Tkinter desktop tool described throughout this section (alphawindow.cli.AlphaWindowApp, not the default alphawindow UI — see React Web UI) is intended for manual adapter verification. Its attach action first opens a searchable window selector; the optional by-click path observes mouse movement/clicks through pynput, minimizes the AlphaWindow app while selecting, and shows a best-effort hover border around the window under the cursor. The final click may still focus or interact with the target application. The preview is SDR/tone-mapped for Tk display and is not a color-managed HDR viewer.

Window screenshot failures are fail-closed. If WGC frame creation, Win32/GDI capture, readback, or tone mapping fails, AlphaWindow surfaces that exception to the caller/UI instead of falling back to desktop screenshots, PIL ImageGrab, or another global capture path.

The wgc_monitor_cached mode is an explicit whole-monitor capture mode, not an automatic fallback. It can help test borderless-fullscreen or compositor-visible game output because it captures the monitor that contains the attached window. It can also expose unrelated pixels on that display, and it still cannot bypass exclusive fullscreen, protected content, or anti-cheat restrictions.

Hook-capable profiles fail closed unless the provided backend advertises hook capability. VirtualizedInputBackend also requires a VirtualInputTransport.

Input And Video Recording

The desktop tool can record the current preview stream and optionally record global mouse/keyboard events plus global system audio for the attached window. The screen capture, input capture, and system audio capture settings are separate: choose the preview/capture backend from Capture method, choose input label capture from Input capture (none or pynput), and choose audio from System audio (none or system).

Hotkey	Action
Ctrl+Alt+F1	Start a new recording or resume the current recording.
Ctrl+Alt+F2	Stop/finalize the current recording without stopping preview capture.

Each session is written under <workspace>/alphawindow-recordings/<timestamp>-hwnd-<handle>/:

File	Contents
session.mp4	SDR MP4 encoded with OpenCV from the same CpuFrame stream shown in the preview.
system_audio.wav	Optional 16-bit PCM WAV captured from the default output device through system loopback when System audio is system.
session.labels.parquet	Standardized mouse/keyboard labels in typed Parquet columns. It shares the session stem with session.mp4. New live recordings use frame-left timestamps: labels observed in [t_i, t_{i+1}) carry t_i.
metadata.json	Target window, recording FPS, mouse mode, input/audio capture methods, label alignment, actual frame timestamps, audio parameters, and output filenames.

Mouse labels support two modes. normalized_window maps absolute screen positions into the attached window as x/y values in 0.0..1.0 and includes inside_window. relative_delta records mouse_delta labels with dx/dy, which is a better shape for FPS-like camera movement datasets. With input capture set to pynput, the built-in recorder observes global mouse/keyboard events only while recording is active. With input capture set to none, it records video only, writes an empty session.labels.parquet, and does not start mouse/keyboard event listeners; only the start/stop hotkeys remain active. The reserved Ctrl+Alt+F1/F2 control chords are filtered from labels. Explicitly declared labels.jsonl bundles remain readable. True raw-input or hook-side FPS deltas should be supplied by a future hook/raw-input event source using the same label model.

For new live recordings, record_frame(t_i) establishes the timestamp used by every mouse, keyboard, and button label until the next accepted frame. Input before the first frame is omitted. Pause clears the anchor and relative-mouse baseline, so resume waits for a new frame before accepting labels. metadata.json declares this contract as "label_alignment": "frame_left"; older bundles without the field retain their existing event-time interpretation. Video writer operations use a separate lock and do not hold the state lock used by input listeners.

Frame-left action semantics

frame_left treats the frame at t_i as the observation before the actions that lead to the next frame. A transition is therefore (frame(t_i), a_i, frame(t_{i+1})), and a_i is stored with timestamp t_i, not t_{i+1}. For a per-frame relative mouse displacement, this means delta_i = pos(t_{i+1}) - pos(t_i) belongs to t_i. When several raw relative-delta events occur during the interval, their sum has the same meaning.

Input observation time	Stored label timestamp	Meaning
Before the first accepted frame t_0	Omitted	There is no recorded visual state to pair with the action.
[t_0, t_1)	t_0	a_0 transforms frame(t_0) toward frame(t_1).
[t_1, t_2)	t_1	a_1 transforms frame(t_1) toward frame(t_2).

Multiple labels may share the same frame timestamp; their Parquet row order preserves their observed order within that frame interval. record_frame() only publishes the new anchor after accepting the frame, while input callbacks remain independent of video encoding. Use event_time only when each label must retain its raw event clock instead of participating in frame-aligned transitions. Writers and readers validate the declared alignment against frame_timestamps so a bundle cannot silently mix these two meanings.

System audio capture is disabled by default. When enabled, AlphaWindow records the default playback device through a loopback backend into system_audio.wav; it is global output-device audio, not per-window or per-HWND audio isolation.

Recording and playback show a topmost click-through input overlay. In normalized_window mode it draws a mouse pad with the current 0.0..1.0 position. In relative_delta mode it draws a joystick-like vector for the latest dx/dy. Keyboard labels are shown on a compact QWERTY layout with pressed keys highlighted.

Recorded sessions can also be loaded back into the desktop tool. Click Load in the Playback section, choose a recording directory, then use Play, Pause, or Stop. Playback uses the same preview canvas and displays labels whose timestamps fall into the current recorded video-frame interval; when metadata.json contains frame_timestamps, playback uses those real capture timestamps instead of assuming perfectly even N / fps frame timing. Label replay output is explicit and defaults to none, so loading a recording does not send mouse or keyboard input unless the caller selects a backend.

Replay output	Behavior	Notes
none	Preview video, labels, and overlay only.	Default; no mouse or keyboard output.
foreground_real	Replays label operations through the guarded foreground input backend.	Uses real desktop input and can move/click/type during the operation; focus/cursor restore is best-effort under normal Windows rules.
win32_message	Replays supported labels as PostMessageW mouse/key messages to the selected HWND.	Does not move the primary cursor, but targets may ignore messages; mouse_delta labels are skipped.
uia	Maps mouse_down to a UIA click and replays key labels through the UIA input backend.	For standard controls; movement, mouse-up, and delta labels are skipped. Requires optional UIA dependency and is not a game-input path.
virtualized_hook	Replays labels through a VirtualizedInputBackend and a named-pipe virtual input transport.	The Web playback service and legacy desktop playback create a fail-closed, session-scoped named-pipe backend for an external hook/agent. Direct LabelReplayController use can pass an explicit backend. The transport writes target-local mouse_move, mouse_delta, click, key, and text commands to a hook/agent pipe instead of using the primary cursor or global keyboard stream.

Example wiring for a target that was already launched with a compatible hook agent. The default pipe template is \\.\pipe\alphawindow_virtual_input_{pid}_{session}; pass a stable session_id when the agent is launched separately:

from alphawindow import (
    LabelReplayController,
    ReplayOutputMode,
    create_named_pipe_virtualized_backend,
)

controller = LabelReplayController(
    target,
    mode=ReplayOutputMode.VIRTUALIZED_HOOK,
    input_backend=create_named_pipe_virtualized_backend(session_id="agent-session"),
)

The Web UI provides a built-in managed virtual input agent for virtualized_hook playback. Start Hook launches it in a separate console window and requests Administrator permission through UAC for that agent process, not for a second AlphaWindow frontend. The built-in agent is tied to the Web UI process: Stop Hook and normal app shutdown send it a stop command, closing the agent console also ends the hook, and it also watches the parent process so it exits if the UI process goes away. To replace the built-in agent with a product-specific injector, pass a hook installer command explicitly. The command may receive the resolved pipe path through {pipe} and the random session token through {session}. If the external hook agent supports explicit teardown, pass --virtual-hook-stop-command before --virtual-hook-command; the Web UI Start Hook / Stop Hook controls and app shutdown use that stop command through the same placeholders. Use --elevate instead when the AlphaWindow Web UI process itself must run as Administrator from startup.

alphawindow --replay-output virtualized_hook `
  --virtual-hook-elevated `
  --virtual-hook-stop-command D:\hooks\uninjector.exe {pid} {pipe} `
  --virtual-hook-command D:\hooks\injector.exe {pid} {pipe} D:\hooks\hook.dll

Programmatic playback uses the same bundle format:

from alphawindow import RecordingBundle, RecordingPlayback

bundle = RecordingBundle.open("alphawindow-recordings/20260625-120000-hwnd-123")
playback = RecordingPlayback(bundle)
try:
    for item in playback.frames():
        print(item.t, item.frame.size, [label.type for label in item.labels])
finally:
    playback.close()

Recording Bundle API

Third-party tools can write AlphaWindow-compatible single-track recordings with write_recording_bundle() and read any supported bundle with read_recording_bundle().

from alphawindow import (
    RecordingLabel,
    RecordingLabelAlignment,
    RecordingMouseMode,
    read_recording_bundle,
    write_recording_bundle,
)

bundle = write_recording_bundle(
    "dataset/my-run-001",
    video="captures/my-run-001.mp4",
    labels=[
        RecordingLabel(t=0.0, type="key_down", fields={"key": "w"}),
        RecordingLabel(
            t=0.0625,
            type="mouse_delta",
            fields={"mode": "relative_delta", "dx": 12, "dy": -3},
        ),
        {"t": 0.125, "type": "key_up", "key": "w"},
    ],
    fps=16,
    label_alignment=RecordingLabelAlignment.FRAME_LEFT,
    frame_timestamps=[0.0, 0.0625, 0.125],
    mouse_mode=RecordingMouseMode.RELATIVE_DELTA,
    metadata={"target": {"title": "external source"}},
)

same_bundle = read_recording_bundle("dataset/my-run-001")
print(same_bundle.label_alignment, same_bundle.frame_timestamps)

The public writer creates the stable v1 bundle shape:

my-run-001/
  metadata.json
  session.mp4
  session.labels.parquet
  system_audio.wav      # optional

metadata.json records the relative video, labels, optional audio, fps, mouse_mode, input_method, audio_method, frame_timestamps, label_alignment, and target fields. Public writes default to RecordingLabelAlignment.FRAME_LEFT; every non-empty frame-left label stream must provide strictly increasing finite frame timestamps, and each label t must match one of them. Use RecordingLabelAlignment.EVENT_TIME explicitly for raw event-time streams. Readers normalize legacy bundles without the field to event_time and reject unknown or contradictory contracts. The normalized values are available as bundle.label_alignment / bundle.frame_timestamps and, for manifests, on each TrackView. video and audio inputs are copied into the bundle by default. Pass copy_media=False only when the media file is already inside the bundle directory; external absolute media paths are rejected so bundles stay portable. Existing output files are protected unless overwrite=True is passed.

Labels may be RecordingLabel instances or JSON-like dict rows with t, type, and type-specific fields. The standard label types are:

Label type	Main fields	Notes
key_down, key_up	key	Keyboard press/release labels.
type_text	text	Text input label for replay backends that support text.
mouse_move	mode, x, y, screen_x, screen_y, inside_window	normalized_window absolute pointer labels.
mouse_down, mouse_up	mode, button, x, y, screen_x, screen_y, inside_window	Button labels usually include the current pointer position.
mouse_delta	mode, dx, dy	relative_delta labels for FPS/camera-style input streams.

session.labels.parquet is the default and preferred label file. labels.jsonl remains readable and can be selected by passing labels_filename="labels.jsonl". RecordingBundle.open() and read_recording_bundle() also read forward manifest bundles and AlphaCSGO dem replay video datasets, but write_recording_bundle() intentionally writes only the stable single-track v1 format. Multi-track/resumable manifest writing remains a separate roadmap item.

Install

From PyPI:

pip install alphawindow

From this checkout:

pip install .

Native Windows adapters are optional:

pip install "alphawindow[native]"

Pillow and numpy screenshot conversions are optional:

pip install "alphawindow[image]"

OCR through pytesseract is optional:

pip install "alphawindow[ocr]"

The pywebview desktop shell is optional:

pip install "alphawindow[web]"

Command-line Web UI

alphawindow

The command starts the local AlphaWindow API and opens the React/Vite UI in pywebview using the current working directory as its workspace label. Click Attach to open a task-manager-style window selector. From there you can:

• Refresh the window list.
• Search/filter by title, handle, PID, thread ID, or class name.
• Select a listed HWND directly.

The by-click/hover-to-attach picker described under Plugin API and Not Supported Yet below is a feature of the legacy Tkinter tool only; this React attach modal is list/search based and has no by-click mode. Picking a past recording is handled by the recordings browser (see React Web UI) rather than a blocking native folder dialog — the native folder picker is now offered only inside the pywebview shell.

After a target is attached, choose the capture mode and press Start when you want the preview loop to begin:

• wgc_window_cached for a persistent window-target WGC readback loop.
• wgc_window_single for one WGC window session per capture.
• wgc_monitor_cached for a persistent WGC readback loop against the monitor containing the attached window.
• win32_print_window for classic PrintWindow(hwnd, hdc, 2) capture.
• win32_bitblt_window for classic GetWindowDC + BitBlt(..., SRCCOPY) capture.
• HDR mode: auto, force, or off.
• Target maximum refresh rate: the default is 16 Hz, and the Hz field accepts custom positive integer values.
• Audio capture intent: none or system.
• Video size: window follows the attached target size; custom accepts explicit output width and height.

The preview refreshes in the app and shows the latest capture time plus the average of the most recent 30 captures. Use alphawindow --workspace D:\path\to\workspace to show a different workspace path in the tool.

The Web menu bar exposes common actions without relying on toolbar controls: File can reload state or close the window, Target can attach windows and control preview capture, Recording exposes input/audio/mouse capture settings, and Playback exposes replay output settings.

React Web UI

The React/Vite Web UI is the default AlphaWindow UI. Packaged mode serves the built assets from the Python package and opens them in pywebview:

alphawindow

To request Administrator rights for the Web UI from a normal launch, use --elevate. Windows opens the standard UAC prompt, starts a new elevated AlphaWindow process, and the original unelevated process waits until that elevated UI exits. Windows cannot elevate the already-running process in place.

alphawindow --elevate

For Codex or browser debugging, run the API without opening pywebview:

alphawindow --browser-only

For frontend development, run Vite in one terminal and the AlphaWindow API in another:

cd webui
npm install
npm run dev

alphawindow --dev --browser-only

The printed Vite URL includes the local API URL and session token. Open that URL in the Codex in-app browser to inspect the same UI that pywebview loads. The old alphawindow web ... prefix is accepted as a compatibility alias, but new usage should call alphawindow directly. To refresh packaged assets after frontend edits, run:

cd webui
npm run build

The UI ships with a dark theme by default and a light theme toggle in Settings. A recordings browser lists past sessions from <workspace>/alphawindow-recordings/ so you can load or delete a recording without a folder dialog; the native folder picker remains available as an alternate entry point only when running inside the pywebview shell. Once a recording is loaded, "Replay onto target" drives the replay_output setting against the currently attached target (see Input And Video Recording for the replay-output modes and their caveats). Playback uses a native <video> scrubber, with frame-by-frame PNG fallback. The preview and target panels also surface DPI physical_rect/physical_size/work_rect and preview.info when the backend provides them.

Headless CLI

A bare alphawindow (no subcommand) still launches the pywebview/React Web UI exactly as before. The subcommands below are additive and opt-in: none of them opens a GUI, so they are suitable for scripting and automation.

• alphawindow windows [--format table|json] [--include-invisible] — list attachable windows and monitors. Default output is a table; --format json prints {"windows": [...]} using the same window/monitor snapshot shape as the Web API.

  alphawindow windows --format json
  

• alphawindow capture (--hwnd <INT> | --desktop | --monitor <HMONITOR>) [--method <METHOD>] [--hdr auto|force|off] --output <PATH.png> — capture a single frame from a window (--hwnd, from the windows listing), the primary desktop (--desktop), or a specific monitor (--monitor, HMONITOR from the windows listing) and save it as a PNG. --method accepts the same capture methods as the app (e.g. wgc_window_cached for windows, wgc_monitor_cached for desktop/monitor; classic win32_* methods only work on windows, not monitors).

  alphawindow capture --desktop --method wgc_monitor_cached --output shot.png
  

• alphawindow export <RECORDING_DIR> (--frame-at <SECONDS> | --clip <START> <END> | --gif <START> <END>) [--fps <FLOAT>] --output <PATH> — export from a saved recording directory (containing metadata.json + session.mp4): a single PNG frame at a timestamp (--frame-at), a trimmed MP4 clip over a time range (--clip START END), or an animated GIF over a range (--gif START END, requires the [image]/Pillow extra). --fps optionally overrides the output frame rate.

  alphawindow export alphawindow-recordings/20260625-120000-hwnd-123 --frame-at 4.5 --output frame.png
  

Import

import alphawindow

from alphawindow import (
    AccessibilityNode,
    AutomationConfig,
    CaptureOutputFormat,
    CaptureTargetKind,
    CallbackVirtualInputTransport,
    ColorMetadata,
    DirectXCaptureConfig,
    GuardedForegroundInputBackend,
    HdrImageFormat,
    ImageMatch,
    JsonSocketVirtualInputTransport,
    JsonVirtualInputAgent,
    MonitorSnapshot,
    OcrEngine,
    OcrResult,
    PynputForegroundInputBackend,
    RecordingHookManager,
    SendInputForegroundInputBackend,
    ToneMapper,
    ToneMappingConfig,
    UiaAccessibilityTree,
    UiaControlInputBackend,
    UserModeHookManager,
    WaitPolicy,
    WaitTimeoutError,
    WgcFrameReadbackResult,
    Win32DesktopRegionCaptureBackend,
    Win32MessageInputBackend,
    Win32StateOutputBackend,
    Win32WindowCaptureBackend,
    Win32WindowCaptureMethod,
    Win32WindowResolver,
    WinrtWgcCapturePipeline,
    WinrtWgcFrameCache,
    InputMode,
    OutputMode,
    WindowSelector,
    WindowSession,
    find_template,
    get_profile,
    recognize_text,
    switch_input_state,
    wait_for_window,
    wait_until,
)

print(alphawindow.__version__)
print(AccessibilityNode.__name__)
print(get_profile("observe").input_mode is InputMode.DRY_RUN)
print(CaptureOutputFormat.NUMPY_BGR.value)
print(HdrImageFormat.ALPHAWINDOW_HDR.value)
print(ColorMetadata.__name__)
print(ImageMatch.__name__)
print(Win32WindowResolver.__name__)
print(Win32StateOutputBackend.__name__)
print(Win32MessageInputBackend.__name__)
print(UiaControlInputBackend.__name__)
print(PynputForegroundInputBackend.__name__)
print(SendInputForegroundInputBackend.__name__)
print(GuardedForegroundInputBackend.__name__)
print(Win32DesktopRegionCaptureBackend.__name__)
print(RecordingHookManager.__name__)
print(CallbackVirtualInputTransport.__name__)
print(UserModeHookManager.__name__)
print(JsonSocketVirtualInputTransport.__name__)
print(JsonVirtualInputAgent.__name__)
print(DirectXCaptureConfig().hdr_mode.value)
print(CaptureTargetKind.MONITOR.value)
print(MonitorSnapshot.__name__)
print(OcrEngine.__name__)
print(OcrResult.__name__)
print(ToneMapper.ACES_FITTED.value)
print(ToneMappingConfig.__name__)
print(UiaAccessibilityTree.__name__)
print(WaitPolicy.__name__)
print(WaitTimeoutError.__name__)
print(WgcFrameReadbackResult.__name__)
print(Win32WindowCaptureBackend.__name__)
print(Win32WindowCaptureMethod.PRINT_WINDOW.value)
print(WinrtWgcCapturePipeline.__name__)
print(WinrtWgcFrameCache.__name__)
print(find_template.__name__)
print(recognize_text.__name__)
print(switch_input_state.__name__)
print(wait_for_window.__name__)
print(wait_until.__name__)

Native WGC Screenshot

from alphawindow import DirectXCaptureConfig, HdrCaptureMode, WindowSnapshot
from alphawindow.native_wgc import WinrtWgcCapturePipeline

target = WindowSnapshot(
    hwnd=123456,
    title="Target",
    rect=(0, 0, 1920, 1080),
    dpi=144,
    metadata={"hdr_enabled": True, "sdr_white_level_nits": 203.0},
)
request = DirectXCaptureConfig(hdr_mode=HdrCaptureMode.AUTO).create_request(target)

pipeline = WinrtWgcCapturePipeline(timeout_seconds=2.0)
frame = pipeline.capture_cpu(target, request, tone_map=True)
frame.save_png("target.png")
pipeline.close()

capture_cpu() and the Win32 capture backends return CpuFrame by default. Callers can choose a higher-level image output format explicitly:

from alphawindow import CaptureOutputFormat

pil_image = frame.to_output(CaptureOutputFormat.PILLOW)      # PIL.Image.Image, RGB
opencv_bgr = frame.to_output(CaptureOutputFormat.NUMPY_BGR)  # numpy uint8, H x W x 3, BGR
numpy_rgb = frame.to_output(CaptureOutputFormat.NUMPY_RGB)   # numpy uint8, H x W x 3, RGB
numpy_bgra = frame.to_output("numpy_bgra")                   # numpy uint8, H x W x 4, BGRA

Numpy output is zero-copy by default when the frame is already SDR BGRA8. numpy_bgra, numpy_bgr, and numpy_rgb are views over CpuFrame.data; numpy_bgr drops alpha through strides, and numpy_rgb reverses channels through a negative stride. If a downstream library requires a C-contiguous array, request a copy explicitly:

opencv_bgr = frame.to_output(CaptureOutputFormat.NUMPY_BGR, copy=True)

The preview controller exposes the same choice through CaptureSettings while still preserving the underlying CpuFrame:

from alphawindow import CaptureOutputFormat
from alphawindow.cli_app import CaptureSettings

settings = CaptureSettings(
    output_format=CaptureOutputFormat.NUMPY_BGR,
    output_copy=False,
)
result = controller.capture(target, settings)
frame = result.frame      # CpuFrame
image = result.output     # zero-copy numpy view in the requested format

To capture the full monitor containing a target, pass a MonitorSnapshot and create a monitor request:

from alphawindow import DirectXCaptureConfig, HdrCaptureMode, MonitorSnapshot
from alphawindow.native_wgc import WinrtWgcCapturePipeline

target = MonitorSnapshot(
    hmonitor=123456,
    device_name="DISPLAY1",
    rect=(0, 0, 3840, 2160),
    work_rect=(0, 0, 3840, 2080),
    dpi=144,
    primary=True,
    metadata={"hdr_enabled": True, "sdr_white_level_nits": 203.0},
)
request = DirectXCaptureConfig(hdr_mode=HdrCaptureMode.AUTO).create_monitor_request(target)

pipeline = WinrtWgcCapturePipeline(timeout_seconds=2.0)
frame = pipeline.capture_cpu(target, request, tone_map=True)
frame.save_png("monitor.png")
pipeline.close()

For repeated capture, keep the WGC session alive and read back the latest frame:

from alphawindow.native_wgc import WinrtWgcFrameCache

def sample_labels(frame_info):
    return {"sequence": frame_info["sequence"], "labels": []}

cache = WinrtWgcFrameCache.start_for(target, request, timeout_seconds=2.0)
try:
    result = cache.readback_latest_cpu(
        tone_map=True,
        fresh_max_age_ms=8.0,
        fresh_wait_ms=16.0,
        before_readback=sample_labels,
    )
    result.cpu.save_png("target.png")
    print(result.info["size"], result.info["content_size"], result.hook_result)
finally:
    cache.close()