Most frameworks assume a UI element has one true locator, and a test breaks the moment that locator changes. Optics assumes an element has several plausible identities — its XPath, its visible text, what it looks like on screen — and tries all of them before giving up.

That idea runs through the whole framework: locators fall back, drivers fall back, element values fall back. Tests are data (CSV or YAML), so non-coders can write them, and the same keywords are reachable from Python, Robot Framework, HTTP, and MCP.

Why Optics

A locator ladder, not a locator. Every element-based keyword walks a priority-ordered chain until one strategy succeeds:

Cheap strategies run first, so vision only costs you time when the tree can't help. Steps 1–4 are LocatorStrategy registrations; step 5 is a separate recovery layer, bounded to five turns and a six-keyword allowlist so it re-enters the ladder rather than tapping blind coordinates. Two more fallback axes sit alongside: multiple values per element name, and multiple enabled drivers or element sources, each tried in config order.

Beyond the ladder:

• Tests are data — elements, modules and test cases as plain CSV or YAML. No IDE, no programming.
• Targets — Android, iOS, web (Selenium/Playwright), Android TV, Samsung Tizen, LG webOS.
• Non-intrusive — the ble driver drives production devices as a Bluetooth HID mouse/keyboard where debugging and screenshots are blocked. Coordinate-only, so pair it with camera_screenshot and the vision strategies.
• Agent-ready — optics mcp exposes every keyword as a typed MCP tool and device state as MCP resources.

Install

Optics needs Python 3.12+. The core install ships no drivers, OCR, or LLM backends — you add only what you need:

python3 -m venv venv && source venv/bin/activate
pip install "optics-framework[appium,easyocr]"

Most extra names match the config.yaml source keys, so the word you install is usually the word you enable (exceptions: llm enables the gemini engine, google-vision enables google_vision, and the bundles and mcp aren't config keys):

optics setup installs the drivers, OCR and LLM engines above (and the bundles) by name — pinned to your installed Optics version — and bare optics setup opens a TUI picker. optics quickstart goes one step further and offers the right extras for the platform you pick:

optics setup --list
optics setup --install appium easyocr

Quickstart

From a fresh install to a running test with one guided command:

pip install optics-framework
optics quickstart

optics quickstart asks what you want to automate, installs the matching engine, scaffolds the project, writes a platform-correct config.yaml, and runs an optics doctor check — then prints your next steps. Every prompt has a default, so pressing Enter throughout yields a runnable project.

Prefer to drive each step yourself?

optics init my_test_project --template contact
# point my_test_project/config.yaml at your device/app, start the Appium server, then:
optics dry_run my_test_project    # validate keywords, elements and module refs — no device (but the config's engines must be installed)
optics execute my_test_project

--template scaffolds a working project from a bundled sample. Each needs the matching extras installed (see Install); the headline [appium,easyocr] above covers the first three:

• contact · calendar · youtube — Appium/Android, easyocr → [appium,easyocr]
• clock — Appium/Android, image templates + Tesseract OCR → [appium,pytesseract] (plus the Tesseract binary)
• gmail_web — Selenium → [selenium,easyocr]
• playwright — Playwright → [playwright] (then playwright install for the browsers)

Omit --template for an empty scaffold with a commented starter config.yaml. Need only a config for an existing folder? optics configure <folder> writes one from a few questions, and optics doctor [folder] [--check] verifies engines, tooling, and project config.

Write a test as data

my_test_project/
├── config.yaml
├── test_cases/test_cases.csv
├── modules/modules.csv
└── test_data/
    ├── elements.csv
    ├── error_definitions.csv      # optional
    └── input_templates/*.png      # optional, for image matching

Optics discovers these files by their content (CSV headers / YAML top-level keys), not their path — the folder names above are a convention, so elements.csv works equally under test_data/ or elements/ (the folder optics live's /save writes to).

test_data/elements.csv — names mapped to locators. Doubles as a general variable store; repeating a name builds a fallback list.

Element_Name,Element_ID
Add_Contact_Button,//android.widget.Button[@content-desc="Create contact"]
First_Name_element,//android.widget.EditText[@text="First name"]
Save_Button,Save
First_Name,John

A locator can be an XPath, text=…, css=…, a plain string, an image filename from input_templates/, or TEXT_ONLY:… to force a vision-based search.

modules/modules.csv — a reusable sequence of keywords; ${name} resolves against elements.csv.

module_name,module_step,param_1,param_2
Add Contact,Press Element,${Add_Contact_Button}
Add Contact,Enter Text,${First_Name_element},${First_Name}
Add Contact,Press Element,${Save_Button}

test_cases/test_cases.csv — modules sequenced into scenarios. A test case whose name contains suite + setup (or teardown) is hoisted to run around the whole suite.

test_case,test_step
Suite Setup,Launch Contact Application
Add Contact with Contact App,Add Contact
Add Contact with Contact App,Verify Contact is Added

Six ways to run the same keywords

/save <test_case> <module_name>

from optics_framework import Optics

optics = Optics()
optics.setup(
    driver_sources=[{"appium": {"enabled": True, "url": "http://localhost:4723"}}],
    elements_sources=[{"appium_find_element": {"enabled": True}}],
)

optics.launch_app("com.example.app")
optics.enter_text("username_field", "testuser")
optics.press_element("submit_button")
optics.validate_element("welcome_message")
optics.quit()

{ "mcpServers": { "optics": { "command": "optics", "args": ["mcp"] } } }

Keywords

Every public method on the four API classes is automatically a keyword, on every surface above. CSV/YAML uses Title Case (Press Element → press_element).

Run optics list for the live catalogue with signatures, or read the Keyword Usage guide for parameters and examples. Location keywords accept percentage-based Area-of-Interest bounds (aoi_x/y/width/height, 0–100) to scope a vision search to part of the screen.

Configure once, in config.yaml

Every section is a priority-ordered list and every entry has an enabled flag. Enable a second driver and it becomes a fallback. Prefer not to hand-write it? optics configure <folder> writes one from a few questions, or optics quickstart builds the whole project including the config.

driver_sources:
  - appium:
      enabled: true
      url: "http://localhost:4723"
      capabilities:
        platformName: Android
        automationName: UiAutomator2
        deviceName: emulator-5554
        appPackage: com.google.android.contacts
        appActivity: com.android.contacts.activities.PeopleActivity

elements_sources:
  - appium_find_element: { enabled: true }
  - appium_page_source:  { enabled: true }
  - appium_screenshot:   { enabled: true }

text_detection:
  - easyocr: { enabled: true }

image_detection:
  - templatematch: { enabled: false }

log_level: INFO

Enabling the LLM features

The llm_models block powers both natural-language mode in optics live (Ctrl-N) and AI self-heal. Install the extra (pip install "optics-framework[llm]"), then add:

llm_models:
  - gemini:
      enabled: true
      capabilities:
        model: gemini-2.5-flash    # optional; this is the default
        # use_vertexai: true       # optional; else read from the environment
        # project: my-gcp-project  # optional (Vertex)
        # location: us-east4       # optional (Vertex)

ai_self_heal: true                 # opt into the LLM backstop; default false

Credentials are read from the environment by the google-genai SDK — GEMINI_API_KEY (or GOOGLE_API_KEY) for the Gemini Developer API, or GOOGLE_GENAI_USE_VERTEXAI + GOOGLE_CLOUD_PROJECT / GOOGLE_CLOUD_LOCATION / GOOGLE_APPLICATION_CREDENTIALS for Vertex AI. Never commit keys to config.yaml. With every capabilities key omitted the SDK auto-detects the backend. google-genai is imported only when gemini is enabled, and a misconfigured LLM degrades to "no self-heal" rather than a hard failure.

Full reference: Configuration. Adding your own engine is a file drop plus an interface — see Extending the Framework.

Results

An optics execute run writes to <project>/execution_output/:

• junit_output.xml — written incrementally, so CI sees progress as it happens
• logs.json — structured logs when json_log: true
• screenshots — pre/post action frames, plus strategy-annotated and AOI overlays
• detected_errors_<session_id>.json — on-screen error detection

Drop an error_definitions.csv into test_data/ and Optics scans visible text for crash dialogs, Session expired, network errors, and the like — no assertions required. Matches also land in the JUnit XML as a synthetic failing testcase, so CI fails a build on "the app crashed mid-test" the same way it fails a normal assertion. See Error Detection.

CLI reference

optics init        Scaffold a new project (positional NAME; --template, --path, --force, --git-init)
optics quickstart  Guided walkthrough: engines + project + config + doctor check
optics configure   Write/edit a project's config.yaml (guided prompts; --edit opens a TUI)
optics doctor      Diagnose engines, tooling and a project's config (--check for CI)
optics setup       Install engine backends (--list, --install); bare command opens a TUI
optics dry_run     Validate a project without touching a device
optics execute     Run a project (--runner test_runner|pytest)
optics live        Interactive keyword session against a live target
optics generate    Emit pytest or Robot Framework code from a project
optics list        Print every discoverable keyword
optics serve       Start the REST API server (--host, --port, --workers)
optics mcp         Start the MCP server (--transport stdio|http)
optics completion  Install shell autocompletion
optics --version   Print the installed version

Details in the CLI guide.

Contributing

Contributions welcome — bug reports, docs, and code are all first-class.

• Bigger items are queued under help wanted.
• Found a bug or missing a keyword? Open an issue.

The contribution guide covers development setup, step-by-step recipes (adding a keyword or engine backend), commit conventions, and the pull-request format reviewers expect. Tooling details live in the Developer Guide; behaviour expectations in our Code of Conduct.

git clone git@github.com:mozarkai/optics-framework.git
cd optics-framework
pipx install poetry
poetry install --with dev,test,docs

poetry run pytest                    # tests + coverage
poetry run ruff check --fix .        # lint
poetry run pre-commit run --all-files
poetry run mkdocs serve              # docs preview

Commits follow Conventional Commits, enforced by commitizen in the commit-msg hook.

Security issues follow SECURITY.md rather than public issues.

License & support

Apache 2.0 — see LICENSE.

Questions and bugs: GitHub Issues. Anything else: lalit@mozark.ai.