auto-any

goal + URL -> signature -> exact receipt? -> deterministic replay
                         \-> local/remote engine -> browser -> distiller -> receipt

Run with uvicorn auto_any.main:app. For local development install RTR first with uv pip install -e <path-to-replayable-test-runner>. Configuration uses the AANY_ prefix; see .env.example.

Install / share

One command gets a teammate from zero to "auto-any MCP Connected":

pwsh install/install.ps1

macOS, Linux, or Git Bash:

bash install/install.sh

See install/README.md for options (custom venv path, local wheel) and the two .env knobs you must set.

Reuse my Chrome logins

Chrome >= 127 (app-bound cookie encryption): use persistent mode. A clone of your Chrome profile copies the cookie files intact, but Chrome cannot decrypt them from a different tree and silently drops every session cookie — the browser is logged out on every run. A persistent profile is a tree auto_any owns; sign in once in a headed window, then every headless run reuses that session:

auto-any profile login --name workspace --url https://accounts.google.com/   # sign in, close the window
auto-any profile login --name personal  --url https://accounts.google.com/   # one tree per identity
auto-any run --url https://mail.google.com/ --goal "..." --profile-mode persistent --profile-name workspace
auto-any profile list

Per-run selection: MCP auto_any_run and HTTP POST /automations take profile: {"mode": "persistent", "name": "workspace"}; AANY_PROFILE_MODE=persistent plus AANY_PROFILE_NAME=<identity> sets the default. Trees live under AANY_PROFILES_DIR/persistent/<name>/User Data and hold live sessions — never commit or share them.

For an older Chrome without the app-bound key, cloning still works. To run against a cloned copy of an already-authenticated Chrome profile:

$env:AANY_PROFILE_MODE = "clone"
$env:AANY_CHROME_SOURCE_PROFILE = "Profile 2"  # or Default
uvicorn auto_any.main:app

The clone is created under AANY_PROFILES_DIR, and the source User Data tree is never written. Chrome may be left open; a cookie written very recently may not have been flushed yet. AANY_PROFILE_KEEP=false removes the clone after the run. Cloning copies live authentication cookies, so treat the profiles directory as sensitive and do not commit or share it. Set AANY_BLOCK_PRIVATE_NETWORKS=true for hosted deployments; the default is false so operator-owned localhost and LAN targets remain usable. For a persistent named profile, use AANY_PROFILE_MODE=named and select it with profile: {"name":"work"} (or use a pack that declares profile_name). Named profiles are managed with GET /profiles and DELETE /profiles/{name}.

Capabilities

Capability	Action kinds	Runner
browser	playwright	RTR Playwright meta-runner
shell	shell	RTR shell runner (allow/deny policy)
http	http_request	RTR HTTP runner
extract	extract	typed Pydantic model
verify	verify	explicit RTR assertion

Add a capability by implementing the Capability contract, registering its name in CAPABILITIES, and registering an RTR StepRunner for each action kind. CapabilityGate then dispatches it without changing the engine loop.

Rubricon / EFA scoring

Install the optional evaluator integration with pip install 'auto-any[efa]' and select it with score="efa" in an automation request. The adapter uses Rubricon's dynamic criteria generation and per-criterion evaluator to attach RAS, APR, and criterion scores to the run result. It evaluates the completed response only; replay remains deterministic and does not invoke an LLM.

Paused OTP/CAPTCHA/confirmation turns emit needs_input; resume with POST /automations/{run_id}/resume. Secret values are stored in receipts only as {"$var":"field"}. Trace artifacts are per-run and video is opt-in.

Storage layout

The storage root is AANY_HOME, defaulting to Path.home() / ".auto_any". Relative AANY_RECEIPTS_DIR and AANY_ARTIFACTS_DIR values resolve under this root; absolute values are honored as supplied.

~/.auto_any/
├── receipts/                 # replayable receipts
├── artifacts/<run_id>/       # per-run screenshots + trace.zip
│   ├── 001_open.png
│   ├── 002_click.png
│   └── trace.zip
├── ledger.jsonl              # append-only run history
└── site_profiles.json        # site → browser-profile mapping

Browser screenshots are captured automatically after browser actions, with zero-padded ordered names such as 001_open.png and 002_click.png. Capture is non-fatal and is enabled by the auto_screenshot=True driver constructor option. A run without an explicit ID uses a timestamped artifact directory. The run ledger is enabled by default as an append-only JSONL file; pass AANY_LEDGER=null to opt out, or AANY_LEDGER=<path>.db for SQLite.

Economics: replayed runs cost 0 LLM tokens; planned runs cost N LLM tokens.

MCP server

auto_any can run as a STDIO Model Context Protocol (MCP) server so Claude and other MCP clients can invoke it in-process.

# install the MCP extra
uv pip install -e ".[mcp,browser]"

# register with Claude Desktop (do not edit ~/.claude manually)
claude mcp add-json auto-any '{"command": "auto-any-mcp"}'

The server boots one shared headless runtime (AANY_PROFILE_MODE=fresh) and exposes six tools:

Tool	Purpose
auto_any_list_packs	List installed packs + parameter schemas
auto_any_run_pack	Run a pack by name with validated params
auto_any_run	Run a free-text goal + URL
auto_any_screenshot_page	Capture a screenshot with optional waiting (no LLM, no receipt)
auto_any_list_receipts	List saved receipts (signature key + title)
auto_any_replay	Replay a receipt with zero LLM tokens

Start it manually with auto-any-mcp.

Screenshot tool

Capture a deterministic screenshot of any URL (no LLM, no receipt):

# CLI: capture screenshot of a local preview server
python -m auto_any.cli screenshot --url "http://127.0.0.1:5199/?preview=kragent" --wait-text "Create agent"
{"ok":true,"path":"C:\\Users\\user\\.auto_any\\artifacts\\cli-abc123\\screenshot.png","width":1280,"height":800,"title":"Create Agent","waited_for":"text:Create agent","console_errors":[]}

# MCP: same operation via tool
auto_any_screenshot_page(url="http://127.0.0.1:5199/?preview=kragent", wait_for_text="Create agent")

The screenshot tool supports:

• Waiting for a CSS selector or text content before capturing (for lazy-loading pages)
• Custom viewport dimensions and full-page capture
• Dark mode emulation
• Path validation (must be inside artifacts, temp, or home directory)

Three Lanes

auto_any exposes the same automation engine through three interfaces. All three delegate to the shared automation_events generator in api/automations.py.

Lane	Command	Description
CLI	auto-any run --goal "..." --url "..."	Direct command-line execution
HTTP	auto-any serve (starts uvicorn)	REST API at POST /automations
MCP	python -m auto_any.mcp_server	STDIO tools for Claude Desktop

All lanes accept the same request shape: goal, url, mode (auto/always_plan/replay_only), and return streaming events containing run_started, progress, run_finished, and optionally receipt_saved.

Additional CLI commands:

• auto-any receipts — list stored receipts (signature key + title)
• auto-any replay <signature_key> — replay a receipt with zero LLM tokens
• auto-any screenshot --url <URL> — capture a deterministic screenshot (no LLM)
• auto-any version — print the running build's identity

Receipt Contract

A receipt is a deterministic, replayable record of an automation run. It contains:

• signature_key: SHA256 hash of goal + url + context — the identity for exact-match replay
• title: Human-readable description
• steps: Ordered list of actions (goto, click, type, extract, verify...)
• url: Target URL at time of execution

What replay guarantees:

• Same goal + same URL → same actions in same order
• Zero LLM tokens consumed (deterministic playback)
• Pass/fail per step — if a page element changed, replay reports failure

What replay does NOT guarantee:

• External state (logged-in sessions, database data, third-party API responses)
• Timing/performance — replay is deterministic but not real-time
• Cross-browser parity — receipts are driver-specific (Playwright vs Selenium)
• Future site changes — replay fails if UI changed; this is the intended behavior

The signature key is the lookup key for both auto-any receipts and the MCP auto_any_replay tool.

How to Tell Which Build is Running

On 2026-07-26, the MCP server ran for hours importing stale code from a different git worktree while reporting healthy. Nothing revealed which build was actually executing.

Two ways to verify the running build:

# CLI — prints version, git SHA, dirty flag, and import path
$ auto-any version
auto-any 0.1.2 git:abcdef1

$ auto-any version --json
{"version": "0.1.2", "git_sha": "abcdef1", "dirty": false, "import_path": "/path/to/site-packages/auto_any"}

# MCP — check the startup log line
$ python -m auto_any.mcp_server
selected browser driver=playwright tier=config_explicit task_hint=
MCP server starting: auto-any 0.1.2 git:abcdef1; import_path=/path/to/site-packages/auto_any

The import_path field shows the exact directory from which auto_any was loaded. If this differs from the expected worktree, the running code is stale.

Quickstart — adopt it for any UI automation

uv venv && uv pip install -e ".[dev,browser]"
uv run playwright install chromium          # real browser lane

# list built-in automation packs (smoke_test, form_fill, example_form)
uv run auto-any packs

# run the API
uv run auto-any serve --port 8080
#   POST /automations            free-text goal (the raw engine)
#   GET  /packs                  installed packs + their param schemas
#   POST /packs/{name}/run       typed params -> same pipeline -> receipt

# reuse your real Chrome logins (no interactive auth):
#   AANY_PROFILE_MODE=clone AANY_CHROME_SOURCE_PROFILE="Profile 2"
# real LLM to drive arbitrary UIs:
#   AANY_LLM_PROVIDER=anthropic AANY_LLM_API_KEY=...
# MiniMax through the OpenAI-compatible adapter (install the openai extra):
#   OPENAI_BASE_URL=https://api.minimax.io/v1 AANY_LLM_PROVIDER=openai
#   AANY_LLM_MODEL=MiniMax-M2.5 AANY_LLM_REASONING_SPLIT=true
#   Provide AANY_LLM_API_KEY through your protected environment.

AANY_LLM_REASONING_SPLIT defaults to false. Enable it only for an OpenAI-compatible provider that supports reasoning_split; it keeps reasoning separate from the JSON action content. Other providers receive no extra parameter.

Packs quickstart

The built-in packs are smoke_test, form_fill, and example_form. For a local unauthenticated development server, set AANY_AUTH_DISABLED=true, then inspect schemas and run smoke_test as an SSE stream:

export AANY_AUTH_DISABLED=true
uv run auto-any serve --port 8080

curl http://127.0.0.1:8080/packs
curl -N -X POST http://127.0.0.1:8080/packs/smoke_test/run \
  -H 'Content-Type: application/json' \
  -d '{"base_url":"http://127.0.0.1:8080","checks":[{"path":"/health","expect_text":"ok"}]}'

With authentication enabled, add -H "Authorization: Bearer <token>" to both requests.

Add a new use-case as a pack (no core edits) - see ARCHITECTURE.md.

Multi-vendor execution

AANY_BROWSER_DRIVER=playwright|selenium|appium|auto selects the per-action driver. Install optional integrations with uv pip install -e ".[selenium]" or .[appium]; absent SDKs and servers produce an actionable unavailable event. A request can override this with {"driver":"selenium"}. Export receipts with POST /receipts/{key}/export and {"format":"cypress"} (playwright and robot are also supported). $var values become vendor environment lookups and are never written as secrets. Set score to passmark for a tolerant score report.

Tests

uv run pytest -q            # hermetic unit suite (fakes only, no browser)
AANY_E2E=1 uv run pytest tests/test_e2e_real_browser.py   # real headless Chromium

Isolated public research

auto_any.public_research.PublicResearchReader reads public HTTP(S) documents through AutoAny's browser and extraction capabilities, without a logged-in profile. It disables JavaScript, child-frame requests and WebSockets, validates public destinations, limits requests and text, and writes an atomic evidence receipt with a SHA-256 content hash. read(url) has a bounded deadline; close() must succeed before callers acknowledge shutdown. Failed browser cleanup or unfinished receipt writes retain the busy state.

Deploy this reader inside an unprivileged, resource-limited container with enforced egress restrictions. Application URL checks do not independently prevent DNS rebinding. Advertised document sizes are checked; chunked response size relies on container memory and execution limits. Never mount browser profiles or unrelated credentials.

For reproducible UI verification, set both AANY_VIEWPORT_WIDTH (320–3840) and AANY_VIEWPORT_HEIGHT (320–2160). They apply to fresh and cloned contexts; explicit viewport overrides are rejected when attaching to an existing CDP browser.