auto-any

Release notes

2026-10-21 · 0.5.5

Change	Does
test usernames and the README preview example use generic names	no personal initials in the shipped files

Source-reference: tests/test_e2e_login_remember.py, tests/test_mcp_sessions.py, tests/test_mcp_surface.py, tests/test_step_effects.py, README.md MCP section; full suite green.

2026-10-21 · 0.5.4

Change	Does
.gitignore local-tool entries are generic globs	hatch always ships .gitignore in the sdist, so it names no local tools; the same files stay ignored

Source-reference: .gitignore; git check-ignore matches the same files; full suite green.

2026-10-21 · 0.5.3

Change	Does
code comments, tests and notes use generic examples	no named sites or internal projects in the shipped files
docs/known-failure-modes.md repaired	escape characters lost in an old write are back (browsergate/, replay/, needs_input)
sdist built from an allowlist	only src, tests, tools, install, one doc, README, tox.ini, .env.example ship; local untracked files and the repo's working tracker stay out

Source-reference: pyproject.toml [tool.hatch.build.targets.sdist]; src/auto_any/browsergate/playwright_driver.py; full suite green.

2026-10-17 · 0.5.2

Change	Does
the PHP package manager dropped from the default shell allowlist	a PHP project that needs it adds the name via RTR_EXTRA_ALLOWED_BINARIES

Source-reference: src/replayable_test_runner/runners/shell.py _DEFAULT_ALLOWED_BINARIES; full suite green.

2026-10-17 · 0.5.1

Console errors raised while a page LOADS are now recorded. The driver attached its console listener after page.goto, so a boot-time error (a CSP block, a failing inline script) was missed and no_console_errors passed on a page whose console was not clean. Found in a test run: auto_any_evaluate saw a CSP error on the live site that the session's check reported as zero. The listener now attaches before the first navigation; an adopted page still attaches on connect. Test: tests/test_console_capture_at_load.py (fails before the fix, passes after).

2026-10-17 · 0.5.1

Two pieces for driving apps the DOM verbs could not reach, plus one guard test.

Change	Does
insert_text verb	{"insert_text": css, "value": txt}: focus, then one CDP Input.insertText (no key events). For rich editors that ignore both fill and type
RTR_EXTRA_ALLOWED_BINARIES	opt-in extra names for the shell step allowlist (e.g. peekaboo for macOS app automation). Shells, script hosts (osascript), sudo, network and destructive tools are dropped whatever is written, .exe spellings included
T1.5 guard test	every verb that calls page.goto must be a key the gate's URL check reads

receipt step --shell--> ExecutionPolicy = default allowlist + extra (minus never-list) --> exec
receipt step --browser--> insert_text --> focus --> Input.insertText --> one "insertText" input event

Source-reference: src/replayable_test_runner/runners/shell.py (ExecutionPolicy.default, _extra_allowed_binaries); runners/browser_actions.py (insert_text); tests/test_shell_extra_allowed_binaries.py (20), tests/test_insert_text_verb.py (4, one in real Chromium: text set, 0 keydowns, inputs == ["insertText"]).

2026-10-17 · 0.5.0

Live browser sessions over MCP: a page now stays open between tool calls, so a sign-in, a click and the check of what the click did can be separate, deterministic calls instead of one LLM run.

Tool	Does
auto_any_session_open	open a page, return a session_id; cdp_url (localhost only) attaches to your own Chrome
auto_any_session_act	run receipt-vocabulary actions in order; get_text/get_attr values come back as extracted
auto_any_session_check	verify_page checks against the page as it is now
auto_any_session_extract	extract_page fields off the page as it is now
auto_any_session_screenshot	PNG of the page as it is now
auto_any_session_close	close it (an attached Chrome is left running)

session_open --> act --> check / extract / screenshot --> close
      |           |
      |           +-- every goto checked by the same UrlPolicy as a run ("refused: ...")
      +-- own driver + gate; never touches the runtime's run driver

Where the page LANDS is checked too, not only the goto: a redirect at open, or a click / redirect during act, onto a blocked target (metadata, private, file:) is refused and the session closed. upload takes files only from the artifacts directory, so a page cannot talk an agent into uploading ~/.ssh. UrlPolicy refuses URLs with a backslash, whitespace or control character: Python and the browser parse those differently, so the host checked would not be the host hit.

Limits are config: AANY_SESSION_MAX (default 4 open) and AANY_SESSION_IDLE_S (default 900 s, then the session closes itself). Errors are labelled session:, cdp:, page:, refused: or invalid:. Tests: tests/test_mcp_sessions.py (35, including real-Chromium flows: state across calls, refused redirect), tests/test_mcp_surface.py.

2026-10-14 · 0.4.2

auto_any_screenshot_page (and cli screenshot) can scroll before the capture: scroll_y (pixels) or scroll_to (a CSS selector, top-aligned, wins over scroll_y); wait_ms counts from after the scroll, and the result reports the real scroll_y. A scroll_to selector that is not on the page is an error, never a silent top-of-page shot. Needed because a page whose chrome changes on scroll (sticky bars that minimise, cards that reveal on enter, scroll-driven animation) could not be captured in its scrolled state through AutoAny; a visual check had to fall back to a hand-written Playwright script for exactly that. Tests: tests/test_screenshot.py::TestCaptureScreenshot::test_scroll_y_before_capture, ::test_scroll_to_selector.

2026-09-28 · 0.4.1

The isolated public researcher now includes literal mailto: addresses, data-cfemail values, and JSON-LD email fields from the same document in its hashed evidence receipt. It executes no scripts and makes no extra requests. Ownership remains unverified; downstream qualification is unchanged. Parsing is bounded to the first 1 MiB of HTML characters, eight 64 KiB JSON-LD blocks, 64 observations, and 8 KiB of metadata within the existing 40 KiB text cap. The original visible body stays first and is never shortened for metadata; the suffix is omitted if the remaining text budget cannot hold it. Consumers should check visible-body quality separately from the delimited metadata.

goal + URL -> signature -> exact receipt? -> deterministic replay
                         \-> local/remote engine -> browser -> distiller -> receipt

Run with uvicorn auto_any.main:app. For local development install RTR first with uv pip install -e <path-to-replayable-test-runner>. Configuration uses the AANY_ prefix; see .env.example.

Install / share

One command gets a teammate from zero to "auto-any MCP Connected":

pwsh install/install.ps1

macOS, Linux, or Git Bash:

bash install/install.sh

See install/README.md for options (custom venv path, local wheel) and the two .env knobs you must set.

Reuse my Chrome logins

Chrome >= 127 (app-bound cookie encryption): use persistent mode. A clone of your Chrome profile copies the cookie files intact, but Chrome cannot decrypt them from a different tree and silently drops every session cookie — the browser is logged out on every run. A persistent profile is a tree auto_any owns; sign in once in a headed window, then every headless run reuses that session:

auto-any profile login --name workspace --url https://accounts.google.com/   # sign in, close the window
auto-any profile login --name personal  --url https://accounts.google.com/   # one tree per identity
auto-any run --url https://mail.google.com/ --goal "..." --profile-mode persistent --profile-name workspace
auto-any profile list

Per-run selection: MCP auto_any_run and HTTP POST /automations take profile: {"mode": "persistent", "name": "workspace"}; AANY_PROFILE_MODE=persistent plus AANY_PROFILE_NAME=<identity> sets the default. Trees live under AANY_PROFILES_DIR/persistent/<name>/User Data and hold live sessions — never commit or share them.

Adopt an existing Electron or Chromium target

Existing-target adoption is explicit and identity-bound. Set the CDP endpoint, pass the page's exact current URL, and opt in. If several pages share that URL, also pass the exact CDP target ID; AutoAny neither creates nor navigates a page:

AANY_CDP_URL=http://127.0.0.1:9222 auto-any run \
  --url http://127.0.0.1:5173/index.html \
  --goal "Inspect the existing application window" \
  --profile-mode attach --attach-existing-target \
  --attach-target-id TARGET_ID

MCP and HTTP callers use profile: {"mode":"attach", "cdp_url":"...", "adopt_existing":true, "target_id":"..."}. The ID is resolved from each page's live CDP session and must match alongside the byte-exact URL. Missing or ambiguous matches fail closed. replay_only is unsupported for adopted targets; auto always verifies the live target instead of using a cached replay. Explicit navigation actions are disabled, teardown disconnects without closing the externally owned window, and attach without adopt_existing retains its existing Chrome behavior.

For an older Chrome without the app-bound key, cloning still works. To run against a cloned copy of an already-authenticated Chrome profile:

$env:AANY_PROFILE_MODE = "clone"
$env:AANY_CHROME_SOURCE_PROFILE = "Profile 2"  # or Default
uvicorn auto_any.main:app

The clone is created under AANY_PROFILES_DIR, and the source User Data tree is never written. Chrome may be left open; a cookie written very recently may not have been flushed yet. AANY_PROFILE_KEEP=false removes the clone after the run. Cloning copies live authentication cookies, so treat the profiles directory as sensitive and do not commit or share it. Set AANY_BLOCK_PRIVATE_NETWORKS=true for hosted deployments; the default is false so operator-owned localhost and LAN targets remain usable. For a persistent named profile, use AANY_PROFILE_MODE=named and select it with profile: {"name":"work"} (or use a pack that declares profile_name). Named profiles are managed with GET /profiles and DELETE /profiles/{name}.

Capabilities

Capability	Action kinds	Runner
browser	playwright	RTR Playwright meta-runner
shell	shell	RTR shell runner (allow/deny policy)
http	http_request	RTR HTTP runner
extract	extract	typed Pydantic model
verify	verify	explicit RTR assertion

Add a capability by implementing the Capability contract, registering its name in CAPABILITIES, and registering an RTR StepRunner for each action kind. CapabilityGate then dispatches it without changing the engine loop.

Rubricon / EFA scoring

Install the optional evaluator integration with pip install 'auto-any[efa]' and select it with score="efa" in an automation request. The adapter uses Rubricon's dynamic criteria generation and per-criterion evaluator to attach RAS, APR, and criterion scores to the run result. It evaluates the completed response only; replay remains deterministic and does not invoke an LLM.

Paused OTP/CAPTCHA/confirmation turns emit needs_input; resume with POST /automations/{run_id}/resume. Secret values are stored in receipts only as {"$var":"field"}. Trace artifacts are per-run and video is opt-in.

Storage layout

The storage root is AANY_HOME, defaulting to Path.home() / ".auto_any". Relative AANY_RECEIPTS_DIR and AANY_ARTIFACTS_DIR values resolve under this root; absolute values are honored as supplied.

~/.auto_any/
├── receipts/                 # replayable receipts
├── artifacts/<run_id>/       # per-run screenshots + trace.zip
│   ├── 001_open.png
│   ├── 002_click.png
│   └── trace.zip
├── ledger.jsonl              # append-only run history
└── site_profiles.json        # site → browser-profile mapping

Browser screenshots are captured automatically after browser actions, with zero-padded ordered names such as 001_open.png and 002_click.png. Capture is non-fatal and is enabled by the auto_screenshot=True driver constructor option. A run without an explicit ID uses a timestamped artifact directory. The run ledger is enabled by default as an append-only JSONL file; pass AANY_LEDGER=null to opt out, or AANY_LEDGER=<path>.db for SQLite.

Economics: replayed runs cost 0 LLM tokens; planned runs cost N LLM tokens.

MCP server

auto_any can run as a STDIO Model Context Protocol (MCP) server so Claude and other MCP clients can invoke it in-process.

# install the MCP extra
uv pip install -e ".[mcp,browser]"

# register with Claude Desktop (do not edit ~/.claude manually)
claude mcp add-json auto-any '{"command": "auto-any-mcp"}'

The server boots one shared headless runtime (AANY_PROFILE_MODE=fresh) and exposes six tools:

Tool	Purpose
auto_any_list_packs	List installed packs + parameter schemas
auto_any_run_pack	Run a pack by name with validated params
auto_any_run	Run a free-text goal + URL
auto_any_screenshot_page	Capture a screenshot with optional waiting (no LLM, no receipt)
auto_any_evaluate	Evaluate JavaScript in the page, or on one element (no LLM, no receipt)
auto_any_capture_network	Record every request the page makes; per-endpoint counts (no LLM, no receipt)
auto_any_list_receipts	List saved receipts (signature key + title)
auto_any_replay	Replay a receipt with zero LLM tokens
auto_any_session_open … _close	Keep one page open across calls: act, check, extract, screenshot (no LLM, no receipt)

Start it manually with auto-any-mcp.

Network capture tool

Records every request the page makes and returns the counts a retry storm is diagnosed with. Query strings are collapsed, so one storm cannot scatter across hundreds of cache-busted keys. settle_ms keeps recording after the waits — that is what catches a storm which only starts once the app boots.

auto_any_capture_network(url="https://app/", wait_for_text="Dashboard", settle_ms=5000)

{"ok": true,
 "summary": {"total": 348, "busiest_endpoint": "https://app/api/me", "busiest_count": 347,
             "by_status": {"401": 347, "200": 1}, "failure_count": 347},
 "console_errors": [], "truncated": false}

Evaluate tool

Arbitrary in-page JavaScript — DOM assertions, computed-style and theme checks, storage reads, in-page fetch. function is () => { ... }, or (element) => { ... } when target names an element.

auto_any_evaluate(url="http://127.0.0.1:5199/", function="() => document.querySelectorAll('li').length")
auto_any_evaluate(url="...", function="(element) => getComputedStyle(element).color", target="#themed")

The value is serialised in-page, so a DOM node, a function, undefined or a cyclic object comes back as a typed description instead of null:

{"ok": true, "result": {"__type": "Node", "nodeName": "DIV", "text": "hello"},
 "type": "Node", "title": "...", "url": "...", "console_errors": [], "truncated": false}

Parameter names mirror browser_evaluate from Microsoft's playwright-mcp (Apache-2.0), so call sites port across unchanged. A bare expression (document.title) is rejected — pass a function.

Screenshot tool

Capture a deterministic screenshot of any URL (no LLM, no receipt):

# CLI: capture screenshot of a local preview server
python -m auto_any.cli screenshot --url "http://127.0.0.1:5199/?preview=agent" --wait-text "Create agent"
{"ok":true,"path":"C:\\Users\\user\\.auto_any\\artifacts\\cli-abc123\\screenshot.png","width":1280,"height":800,"title":"Create Agent","waited_for":"text:Create agent","console_errors":[]}

# MCP: same operation via tool
auto_any_screenshot_page(url="http://127.0.0.1:5199/?preview=agent", wait_for_text="Create agent")

The screenshot tool supports:

• Waiting for a CSS selector or text content before capturing (for lazy-loading pages)
• Scrolling first: scroll_y (pixels) or scroll_to (selector, top-aligned); wait_ms then counts from after the scroll, so sticky bars, lazy sections and scroll-driven reveals are captured in their scrolled state; the result reports the real scroll_y
• Custom viewport dimensions and full-page capture
• Dark mode emulation
• Path validation (must be inside artifacts, temp, or home directory)

Three Lanes

auto_any exposes the same automation engine through three interfaces. All three delegate to the shared automation_events generator in api/automations.py.

Lane	Command	Description
CLI	auto-any run --goal "..." --url "..."	Direct command-line execution
HTTP	auto-any serve (starts uvicorn)	REST API at POST /automations
MCP	python -m auto_any.mcp_server	STDIO tools for Claude Desktop

All lanes accept the same request shape: goal, url, mode (auto/always_plan/replay_only), and return streaming events containing run_started, progress, run_finished, and optionally receipt_saved.

Additional CLI commands:

• auto-any receipts — list stored receipts (signature key + title)
• auto-any replay <signature_key> — replay a receipt with zero LLM tokens
• auto-any screenshot --url <URL> — capture a deterministic screenshot (no LLM)
• auto-any version — print the running build's identity

Receipt Contract

A receipt is a deterministic, replayable record of an automation run. It contains:

• signature_key: SHA256 hash of goal + url + context — the identity for exact-match replay
• title: Human-readable description
• steps: Ordered list of actions (goto, click, type, extract, verify...)
• url: Target URL at time of execution

What replay guarantees:

• Same goal + same URL → same actions in same order
• Zero LLM tokens consumed (deterministic playback)
• Pass/fail per step — if a page element changed, replay reports failure

What replay does NOT guarantee:

• External state (logged-in sessions, database data, third-party API responses)
• Timing/performance — replay is deterministic but not real-time
• Cross-browser parity — receipts are driver-specific (Playwright vs Selenium)
• Future site changes — replay fails if UI changed; this is the intended behavior

The signature key is the lookup key for both auto-any receipts and the MCP auto_any_replay tool.

How to Tell Which Build is Running

On 2026-07-26, the MCP server ran for hours importing stale code from a different git worktree while reporting healthy. Nothing revealed which build was actually executing.

Two ways to verify the running build:

# CLI — prints version, git SHA, dirty flag, and import path
$ auto-any version
auto-any 0.1.2 git:abcdef1

$ auto-any version --json
{"version": "0.1.2", "git_sha": "abcdef1", "dirty": false, "import_path": "/path/to/site-packages/auto_any"}

# MCP — check the startup log line
$ python -m auto_any.mcp_server
selected browser driver=playwright tier=config_explicit task_hint=
MCP server starting: auto-any 0.1.2 git:abcdef1; import_path=/path/to/site-packages/auto_any

The import_path field shows the exact directory from which auto_any was loaded. If this differs from the expected worktree, the running code is stale.

Quickstart — adopt it for any UI automation

uv venv && uv pip install -e ".[dev,browser]"
uv run playwright install chromium          # real browser lane

# list built-in automation packs (smoke_test, form_fill, example_form)
uv run auto-any packs

# run the API
uv run auto-any serve --port 8080
#   POST /automations            free-text goal (the raw engine)
#   GET  /packs                  installed packs + their param schemas
#   POST /packs/{name}/run       typed params -> same pipeline -> receipt

# reuse your real Chrome logins (no interactive auth):
#   AANY_PROFILE_MODE=clone AANY_CHROME_SOURCE_PROFILE="Profile 2"
# real LLM to drive arbitrary UIs:
#   AANY_LLM_PROVIDER=anthropic AANY_LLM_API_KEY=...
# MiniMax through the OpenAI-compatible adapter (install the openai extra):
#   OPENAI_BASE_URL=https://api.minimax.io/v1 AANY_LLM_PROVIDER=openai
#   AANY_LLM_MODEL=MiniMax-M2.5 AANY_LLM_REASONING_SPLIT=true
#   Provide AANY_LLM_API_KEY through your protected environment.

AANY_LLM_REASONING_SPLIT defaults to false. Enable it only for an OpenAI-compatible provider that supports reasoning_split; it keeps reasoning separate from the JSON action content. Other providers receive no extra parameter.

Packs quickstart

The built-in packs are smoke_test, form_fill, and example_form. For a local unauthenticated development server, set AANY_AUTH_DISABLED=true, then inspect schemas and run smoke_test as an SSE stream:

export AANY_AUTH_DISABLED=true
uv run auto-any serve --port 8080

curl http://127.0.0.1:8080/packs
curl -N -X POST http://127.0.0.1:8080/packs/smoke_test/run \
  -H 'Content-Type: application/json' \
  -d '{"base_url":"http://127.0.0.1:8080","checks":[{"path":"/health","expect_text":"ok"}]}'

With authentication enabled, add -H "Authorization: Bearer <token>" to both requests.

Add a new use-case as a pack (no core edits) - see ARCHITECTURE.md.

Multi-vendor execution

AANY_BROWSER_DRIVER=playwright|selenium|appium|auto selects the per-action driver. Install optional integrations with uv pip install -e ".[selenium]" or .[appium]; absent SDKs and servers produce an actionable unavailable event. A request can override this with {"driver":"selenium"}. Export receipts with POST /receipts/{key}/export and {"format":"cypress"} (playwright and robot are also supported). $var values become vendor environment lookups and are never written as secrets. Set score to passmark for a tolerant score report.

Tests

uv run pytest -q            # hermetic unit suite (fakes only, no browser)
AANY_E2E=1 uv run pytest tests/test_e2e_real_browser.py   # real headless Chromium

Isolated public research

auto_any.public_research.PublicResearchReader reads public HTTP(S) documents through AutoAny's browser and extraction capabilities, without a logged-in profile. It disables JavaScript, child-frame requests and WebSockets, validates public destinations, limits requests and text, and writes an atomic evidence receipt with a SHA-256 content hash. read(url) has a bounded deadline; close() must succeed before callers acknowledge shutdown. Failed browser cleanup or unfinished receipt writes retain the busy state.

Deploy this reader inside an unprivileged, resource-limited container with enforced egress restrictions. Application URL checks do not independently prevent DNS rebinding. Advertised document sizes are checked; chunked response size relies on container memory and execution limits. Never mount browser profiles or unrelated credentials.

For reproducible UI verification, set both AANY_VIEWPORT_WIDTH (320–3840) and AANY_VIEWPORT_HEIGHT (320–2160). They apply to fresh and cloned contexts; explicit viewport overrides are rejected when attaching to an existing CDP browser.