tapyr-cli 0.1.0

Playwright for Android, built for AI agents. CLI + MCP server for LLM-driven phone control.

Tapyr

Playwright for Android, built for AI agents.

A CLI and stdio MCP server that gives an LLM (Claude, Cursor, any MCP-capable agent) structured perception and action primitives for a connected Android device, plus a file-based knowledge layer so learned workflows and app notes persist across sessions.

Tapyr is not an agent. It is the hands, eyes, and memory that an agent uses. Bring your own brain.

Why

LLMs can already read text and reason about intent. They cannot, by default, open your Gojek app, tap "GoFood", type "sate", and hand off to checkout. Tapyr closes that gap with a small, opinionated set of primitives designed for tool-use from the ground up:

• JSON-first output. Every command returns {"ok": bool, ...} — no log-scraping.
• Semantic targeting. tap --text "Order" or tap --id 7 (from the last perception), not tap 540 1820.
• Set-of-Mark perception. Screenshot + UI dump + an annotated PNG with numbered boxes so vision models can pick an element by number.
• Cross-device coordinates. tap_xy and swipe accept percent strings ("50%") that resolve against the live wm size at replay time. Workflows written on 720x1604 replay on 1080x2400 unchanged.
• Workflow executor. YAML workflows with template variables, weighted choose_one branches, repeat, sequence, expect_screen, verify, if / skip_if / run_if conditional gates, per-step retries with exponential backoff, and a debug mode that snapshots fingerprints and annotated PNGs per step. kb run tiktok_engage --arg count=10 — done.
• Safety gates. checkpoint: name, requires_confirmation: true halts a replay cleanly at payment / destructive steps; resume past the gate with tapyr kb run NAME --resume-from checkpoint_name without re-executing the expensive setup.
• Observability. tapyr kb stats surfaces per-workflow success rate, halted runs, and drift-marker counts from the JSONL audit log — flaky workflows bubble to the top. Soft expect_screen mismatches emit a structured needs_llm_fallback marker so an orchestrating agent can machine-detect drift without parsing prose warnings.
• Calibration for canvas apps. Apps that render via custom canvas (TikTok, Instagram Reels, YouTube Shorts) are invisible to uiautomator. calibrate-rail scans for a vertical icon rail (right-side action column), calibrate-nav scans for a horizontal one (bottom nav), and the tap_rail workflow action consumes saved calibrations by semantic name (icon: comment). See below.
• Lock screen + device status. tapyr unlock [--pin XXXX] dismisses the keyguard idempotently (no-ops if already unlocked). tapyr status returns a one-shot snapshot — locked? foreground? screen size? wakefulness? battery? — in a single call instead of five sequential adb round trips.
• File-backed memory. Workflows are YAML. App notes are Markdown. The agent can read, write, and share them.
• One binary, many front doors. The same core runs as a CLI, an MCP stdio server, or a self-installing plugin for Claude Desktop / Claude Code / Cursor.

Architecture

┌─────────────────────────────────────────────┐
│  executor     replay workflows, verify      │
├─────────────────────────────────────────────┤
│  knowledge    workflows + app notes + FTS   │
├─────────────────────────────────────────────┤
│  calibration  whiteness scan for canvas UIs │
├─────────────────────────────────────────────┤
│  actions      semantic tap, swipe, type     │
├─────────────────────────────────────────────┤
│  perception   screenshot + ui_dump + SoM    │
├─────────────────────────────────────────────┤
│  driver       adb subprocess wrapper        │
└─────────────────────────────────────────────┘

Each layer is usable independently. Agents operate at whatever level the task needs: replay a saved workflow (fast path), drop to semantic taps (normal path), coordinate-based taps on a calibrated rail (canvas-app escape hatch), or raw pixels (last resort).

Install

pipx install tapyr-cli        # or: pip install tapyr-cli
tapyr doctor                  # tells you exactly what's missing
tapyr doctor --fix            # auto-install adb if missing, re-run checks
tapyr install-adb             # one-shot Playwright-style fetch of platform-tools
tapyr demo --dry-run          # preview the 7-step scripted walkthrough
tapyr demo                    # with a phone plugged in: the hero shot

tapyr install-adb fetches the official Android platform-tools bundle from dl.google.com into ~/.tapyr/bin/, falling back to a find_adb() chain extension so PATH-installed adb still wins if present. Cached after first run; --force re-downloads. Optional TAPYR_ADB_SHA256 env var for enterprise users who need a pinned checksum.

tapyr demo runs a zero-side-effect, 7-step walkthrough (doctor → wake → perceive → HOME → deep-link Settings → BACK → save a marker workflow) that gives the first-run experience a single command. No typing into fields, no taps inside third-party apps, no installs. --dry-run works without a phone for curious users and CI.

You will also need Android platform-tools on your PATH (for adb) and a phone with USB debugging enabled. doctor walks you through both.

The TAPYR_HOME environment variable overrides the default ~/.tapyr state directory. Useful for multi-device test rigs or CI.

Wire it into your agent

tapyr install claude-desktop   # writes ~/Library/.../claude_desktop_config.json
tapyr install claude-code      # writes ~/.claude/mcp.json
tapyr install cursor           # writes ~/.cursor/mcp.json
tapyr install print            # just prints the JSON snippet

Then restart the client. The agent now has a tapyr tool group with:

• Intent resolver: do (natural-language → action, no API knowledge needed)
• Perception: perceive, screenshot
• Actions: tap, swipe, type_text, key, scroll_until, wait_for, unlock, exec_steps
• Device: device_status, list_devices
• Apps: open_app, deep_link, foreground, list_packages, read_notifications
• Calibration: calibrate_rail, calibrate_nav, calibrations_list, calibration_show, calibration_delete
• Credential vault: fill_credential, vault_set, vault_list, vault_delete (macOS / Linux / Windows; no vault_get by design)
• Safety policy: safety_list, safety_set_rule, safety_delete_rule
• Knowledge base: kb_search_workflows, kb_suggest_workflows, kb_read_workflow, kb_run_workflow, kb_save_workflow, kb_delete_workflow, kb_rename_workflow, kb_clone_workflow, kb_update_workflow_metadata, kb_lint_workflow, kb_diff_workflows, kb_find_references, kb_check_calibration, kb_record_workflow, kb_add_tags, kb_remove_tags, kb_read_history, kb_stats, kb_read_app_note, kb_append_app_note, kb_list_app_notes, kb_list_workflows, kb_export_bundle, kb_import_bundle, kb_install_bundle
• Environment: doctor

56 MCP tools across 11 families — the full surface an agent needs to perceive, act, learn, and manage workflows on any connected Android device.

Usage (CLI)

# Sanity check
tapyr doctor --pretty
tapyr devices
tapyr status --pretty            # one-shot: locked, foreground, battery, ...

# See the screen the way an agent sees it
tapyr perceive --pretty
#   -> ~/.tapyr/cache/last_screenshot.png
#   -> ~/.tapyr/cache/last_annotated.png  (numbered overlays)
#   -> JSON element list with ids
tapyr perceive --clickable-only    # trim layout noise, keep only tap targets
tapyr perceive --compact           # minimal payload: id + text + clickable only
tapyr watch --interval 2 --max-polls 10  # poll for screen changes, JSON Lines output

# AI CLI mode — just say what you want
tapyr do "like this post"                  # resolves intent → action automatically
tapyr do "go home"                         # builtin: HOME key (no perceive needed)
tapyr do "type hello world"                # detects type intent
tapyr do "open com.whatsapp"               # opens app by package
tapyr do "scroll down"                     # swipe gesture
tapyr do "like this" --dry-run             # preview the resolved action without executing
tapyr perceive --since-last        # delta mode: only what changed since the last call
tapyr --timeout 60 perceive        # global adb-shell timeout override (slow devices)

# Drive the phone
tapyr unlock --pin 1234          # dismisses keyguard, idempotent
tapyr tap --text "GoFood"
tapyr type "sate"
tapyr key enter
tapyr tap --id 3
tapyr tap --xy 50% 90%           # percent coords work everywhere
tapyr swipe 50% 75% 50% 25%
tapyr scroll-until --text "Add to cart"
tapyr wait-for --text "Order confirmed" --timeout 15

# Skip the UI entirely when you know the intent
tapyr deep-link "gojek://gofood/home"
tapyr open com.whatsapp

# Read push notifications / SMS OTPs (needs --noredact on device)
tapyr read-notifications --package com.android.messaging --pretty

# Knowledge base
tapyr kb list
tapyr kb list --query "food order"
tapyr kb search "checkout" --tag food                  # FTS + tag filter
tapyr kb read gojek_order_food
tapyr kb run tiktok_natural_engagement --arg count=10
tapyr kb run unlock_flow gofood_order verify_order    # pipeline: run 3 in sequence
tapyr kb run my_workflow --debug                      # per-step PNGs
tapyr kb run my_workflow --dry-run                    # preview only
tapyr kb run pay_flow --confirm-checkpoints           # walk past gates
tapyr kb run pay_flow --confirm-safety                # walk past require_confirmation safety rules
tapyr kb run pay_flow --resume-from confirm_payment   # pick up from halt
tapyr kb run my_workflow --dry-run --dry-run-state '{"screen_texts":["OK"]}' # evaluate if/skip_if gates against a fixture
tapyr kb lint my_workflow                             # static validation
tapyr kb stats --days 7                               # flaky workflows
tapyr kb history --limit 20 --action run_workflow
tapyr kb delete old_workflow
tapyr kb rename old_name new_name                      # atomic rename + FTS reindex
tapyr kb tag add gojek_order food delivery             # add tags to a saved workflow
tapyr kb tag remove gojek_order wip                    # remove tags
tapyr kb clone gojek_order --as gojek_order_v2         # duplicate for safe editing
tapyr kb diff workflow_v1 workflow_v2 --pretty         # colored unified-diff
tapyr kb history --export csv > history.csv             # spreadsheet-friendly audit export
tapyr kb find-refs --calibration tiktok_rail           # who uses this?
tapyr kb check-calibration tiktok_rail                 # did re-cal break anything?
tapyr kb app com.gojek.app
tapyr kb app com.gojek.app --append "Payment confirm button is bottom-right"
tapyr kb apps                                          # list all documented packages
tapyr kb export ~/gojek-pack.tapyr.zip --all            # pack workflows+cals+notes
tapyr kb import ~/gojek-pack.tapyr.zip --merge rename   # skip/overwrite/rename
tapyr kb install https://example.com/gojek-pack.tapyr.zip --sha256 ...  # fetch + verify + import

# Credential vault (macOS / Linux / Windows)
tapyr vault set com.whatsapp password --stdin           # reads from stdin, never argv
tapyr vault list                                         # {package, field} only — no secrets
tapyr fill-credential com.whatsapp password              # paste into focused field

# Action safety policy
tapyr safety list                                        # show current deny/require_confirmation/allow rules
tapyr safety set deny key --equals POWER                 # never POWER key on this device
tapyr safety set deny shell --contains "pm uninstall"    # forbid uninstalls via shell action
tapyr safety set require_confirmation deep_link --package com.android.vending
tapyr safety delete deny key --equals POWER              # exact-match delete

# Parallel test fleets
TAPYR_MULTI_DEVICE=1 tapyr --serial emulator-5554 perceive  # scoped cache + history

Calibration for canvas-rendered apps

Some apps render their entire UI through a custom canvas engine — TikTok, Instagram Reels, YouTube Shorts. Their windows have focus but shown=false in dumpsys, so uiautomator dump falls through to whatever's drawn underneath (usually the launcher). perceive returns zero meaningful elements. Semantic tap --text and tap --content_desc do not work.

Tapyr's workaround: scan the screen for white icon clusters along a vertical rail and use the cluster centers as blind-tap targets.

# 1. One-time setup per app+device: scan the rail, label the clusters,
#    save the result. The x-column auto-detects in the right 15% of the
#    screen; pass --x-col to override.
tapyr calibrate-rail \
  --y-min 500 --y-max 1400 \
  --min-cluster-pixels 30 \
  --labels heart,comment,bookmark,share \
  --save tiktok_rail.json

# 2. In a workflow yaml, reference the saved calibration by name:
#
#    - action: tap_rail
#      calibration: tiktok_rail
#      icon: comment                       # v1: bare label
#      icon: right_rail.comment            # v2: namespaced (region.label)
#
#    Unified v2 calibrations merge multiple regions (right_rail, bottom_nav)
#    into one file per app. Bare labels auto-resolve when unambiguous;
#    namespaced labels (right_rail.comment) disambiguate across regions.
#    v1 files (single rail) keep working unchanged.

# 3. Discover what's saved
tapyr calibrations --pretty

The examples/tiktok_natural_engagement.yaml workflow is a fully worked reference: a weighted choose_one loop that randomly likes, favorites, shares, or comments (with Indonesian gen-Z phrases) 5-45% of the time, passively scrolls the rest, and posts real comments via TikTok's native input field.

See examples/README.md for four more showcase workflows: whatsapp_send_message (the hello-world tutorial), gofood_order (Indonesian use case with a payment checkpoint), spotify_play_playlist (morning-routine weighted rotation), and instagram_reels_engagement (canvas-app mirror of the TikTok example). Every file in examples/ is pinned lint-clean by a regression test.

Knowledge on disk

~/.tapyr/
├── workflows/              # agent-written task recipes (yaml)
├── apps/                   # per-app declarative notes (markdown)
├── calibration/            # rail-icon calibrations for canvas apps (json)
├── history/                # jsonl audit log of every action
├── cache/                  # last screenshot, dump, annotated PNG
├── debug/                  # per-step snapshots from `kb run --debug`
└── index.sqlite            # FTS index — rebuildable from the above

Everything is git-friendly. You can version-control your ~/.tapyr/ directory and share workflows + calibrations across machines or with teammates.

Usage (from an agent, via MCP)

Once installed into your client, the agent can call the tools directly. A typical loop for a well-known app:

1. perceive → read the element list + annotated PNG
2. tap with a selector that matches an element
3. perceive → confirm the new screen
4. kb_save_workflow once the task completes, so the next agent is faster

For a canvas-rendered app (TikTok etc.) where uiautomator dump returns nothing: vision-first fallback (v0) automatically kicks in — the screenshot is analyzed via grid-based variance detection to synthesize clickable regions even without a UI tree. The agent sees numbered boxes on the annotated PNG and can tap --id N. For higher precision on known apps, calibrate once and replay:

1. calibrate_app with --rail and --nav flags to scan both icon regions in one shot and save a unified v2 calibration file
2. kb_save_workflow with tap_rail or tap_saved steps referencing namespaced labels (right_rail.comment, bottom_nav.home)
3. kb_run_workflow to replay

Status

v0 / alpha. Validated on a real Oppo 720x1604 device through 40+ loop iterations of incremental hardening. The TikTok engagement workflow posts real comments, likes, bookmarks, and shares through the coordinate-based tap_rail path. 977 tests passing, covering the executor, actions, calibration scanners, knowledge base, MCP server, CLI dispatch, and the workflow observability surface.

What's here (as of this README revision): the full primitive set listed above, 10 MCP tool families, kb stats observability over the JSONL audit log, soft-drift needs_llm_fallback markers on expect_screen mismatches, idempotent unlock and composed device_status, and a cross-device coordinate system that works on any screen that wm size can report.

Vision-first perception fallback (v0) shipped — when uiautomator dump returns nothing, perception now synthesizes clickable regions from the screenshot via grid-based variance analysis so agents can at least tap --id N on numbered areas in the annotated PNG. Still directional (v1): replacing the heuristic grid with a segmentation model (SAM / YOLO) for element-level precision, plus OCR for text field extraction. See BACKLOG.md for the full list with priority ordering.

License

MIT.