macos-computer-use

LLM-free macOS desktop automation primitives for agent applications.

Monorepo Direction

This repository is being migrated into an app-control tools monorepo. The first new package is packages/app-control-protocol, which owns the shared ToolCommand, ToolObservation, ToolEvent, error, and configuration contracts.

The existing root macos_computer_use package remains in place for compatibility, and packages/computer-use-macos now owns the planned computer_use_macos import path and computer-use-macos CLI with a package-local backend implementation. The goal is a developer-facing software suite that can serve Plato as the first consumer while remaining independent from any product-specific runtime.

The monorepo now also contains packages/wechat-desktop-tool, a semantic WeChat Desktop package built on top of the protocol and a compatible app-control client.

This package is deliberately small. It provides local macOS capability only:

• readiness and permission checks;
• bounded structure-first observation;
• allowlisted app opening and activation;
• conservative text input;
• conservative click attempts;
• high-risk action classification metadata.

It does not include an LLM, planner, task queue, UI, confirmation store, network worker, or business workflow. Applications such as Plato should consume this package through a normal package dependency and map package results into their own task, confirmation, evidence, and audit systems.

Install

The package is intended for PyPI publication. During local development:

python -m pip install -e .

For a new developer-preview setup, follow docs/quickstart.md. It covers the 30 minute TextEdit smoke, local service mode, helper path, and 60 minute WeChat focus/draft smoke.

Quick Start

from macos_computer_use import MacOSComputerUseClient

client = MacOSComputerUseClient(
    allowed_apps=("TextEdit",),
)

print(client.readiness().to_dict())
print(client.open_app("TextEdit").to_dict())
print(client.observe(target_app="TextEdit").to_dict())

With shared app-control configuration:

client = MacOSComputerUseClient.from_config("app-control.toml")

Start from the editable template:

cp examples/app-control.toml app-control.toml

The migration distribution also exposes the planned short name:

from computer_use_macos import ComputerUseClient

client = ComputerUseClient.from_helper_manifest("helper_config.json")
observation = client.open_app("TextEdit")

Helper-backed clients expose the same protocol-first convenience methods as the direct backend (readiness, open_app, type_text, press_key, hotkey, and the other computer-use operations). These helper methods return ToolObservation objects because the helper boundary is command/observation based.

Safety Defaults

The default policy is intentionally conservative:

• non-allowlisted apps are blocked;
• raw coordinate click is disabled;
• high-risk targets such as send, pay, delete, submit, install, and permission controls are blocked with confirmation_required metadata;
• type_text does not press Enter and rejects newline text;
• password/security/system-dialog targets are blocked;
• screenshots are not captured.

The caller owns user confirmation. This package only returns risk metadata.

macOS Permissions

Most operations require macOS Accessibility permission for the Python process or the app launching it.

The package reports missing permissions through readiness() instead of trying to bypass them.

Production callers should use a signed helper app as the stable macOS permission subject. The permission model is documented in docs/permissions.md. The helper manifest, JSON-lines unix socket transport, and doctor workflow are documented in docs/helper-packaging.md.

computer-use-macos helper init ./computer-use-helper \
  --name "Example Computer Use Helper" \
  --bundle-id com.example.computer-use-helper
computer-use-macos helper build ./computer-use-helper
computer-use-macos helper doctor ./computer-use-helper

Non-Python callers can use the local Unix socket service mode documented in docs/local-service.md:

computer-use-macos serve \
  --config ./app-control.toml \
  --socket-path /tmp/app-control.sock \
  --token-file ./app-control.token
computer-use-macos request \
  --socket-path /tmp/app-control.sock \
  --token-file ./app-control.token \
  --operation readiness

When [helper] endpoint and optional token are set in app-control.toml, both serve and request can read the local service connection settings from --config; explicit CLI socket/token values still take precedence.

API

See docs/api.md for the API contract.

client.readiness()
client.run_command(command)
client.run_stream(command)
client.observe(target_app=None)
client.open_app("TextEdit")
client.focus_app("TextEdit")
client.type_text("hello", target_app="TextEdit")
client.press_key("Return", target_app="TextEdit")
client.hotkey(("Command", "K"), target_app="TextEdit")
client.click("OK", target_app="TextEdit")
client.click_accessibility({"role": "button", "name": "OK"}, target_app="TextEdit")
client.click_coordinate(120, 240)  # requires allow_coordinate_click=True
client.wait(seconds=1.0)

All methods return dataclass models with .to_dict() for JSON-friendly transport.

Development

python scripts/dev_check.py

For targeted debugging, the unified check runs these underlying suites:

python -m unittest discover -s tests
PYTHONPATH=packages/app-control-protocol/src \
  python -m unittest discover -s packages/app-control-protocol/tests
PYTHONPATH=packages/app-control-protocol/src:packages/wechat-desktop-tool/src \
  python -m unittest discover -s packages/wechat-desktop-tool/tests
PYTHONPATH=src:packages/app-control-protocol/src:packages/computer-use-macos/src \
  python -m unittest discover -s packages/computer-use-macos/tests
python scripts/release_preflight.py
python -c "import macos_computer_use; print(macos_computer_use.__version__)"

To see optional checks, including the strict external-proof release gate:

python scripts/dev_check.py --list
python scripts/dev_check.py --check wheel-preflight
python scripts/dev_check.py --check release-proof-preflight

wheel-preflight builds all local wheels in a temporary directory, verifies their packaged contents, installs them from the local wheelhouse into a clean virtual environment, and runs the API smoke snippets. release-proof-preflight expects the JSON assets in ./release-proof/ as prepared by scripts/release_proof_bundle.py.

Build And Publish

See docs/release-checklist.md for the release checklist and docs/publishing.md for the PyPI release flow. Migration guidance is in docs/migration-notes.md.

Local wheel check:

python -m pip wheel --no-build-isolation --no-deps . -w dist
python scripts/release_preflight.py --wheel-dir dist

Manual macOS validation is documented in docs/manual-smoke.md. Manual WeChat validation is documented in docs/wechat-smoke.md.

Recommended public release flow:

1. run tests;
2. build wheel and sdist in a clean environment;
3. publish to TestPyPI first;
4. install from TestPyPI in a clean macOS virtual environment;
5. publish the tagged release to PyPI after the smoke check passes.

Prefer PyPI trusted publishing from GitHub Actions for public release. Do not commit PyPI tokens to this repository.

Package Boundary

This package must not depend on:

• Plato / Taskweavn;
• LLM SDKs;
• Agent frameworks;
• UI frameworks;
• network task systems.

Consumers should wrap this package with their own adapter.