Most frameworks assume a UI element has one true locator, and a test breaks the moment that locator changes. Optics assumes an element has several plausible identities — its XPath, its visible text, what it looks like on screen — and tries all of them before giving up.

That idea runs through the whole framework: locators fall back, drivers fall back, element values fall back. Tests are data (CSV or YAML), so non-coders can write them, and the same keywords are reachable from Python, Robot Framework, HTTP, and MCP.

Why Optics

A locator ladder, not a locator. Every element-based keyword walks a priority-ordered chain until one strategy succeeds:

Cheap strategies run first, so vision only costs you time when the tree can't help. Steps 1–4 are LocatorStrategy registrations; step 5 is a separate recovery layer, bounded to five turns and a six-keyword allowlist so it re-enters the ladder rather than tapping blind coordinates. Two more fallback axes sit alongside: multiple values per element name, and multiple enabled drivers or element sources, each tried in config order.

Beyond the ladder:

• Tests are data — elements, modules and test cases as plain CSV or YAML. No IDE, no programming.
• Targets — Android, iOS, web (Selenium/Playwright), Android TV, Samsung Tizen, LG webOS.
• Non-intrusive — the ble driver drives production devices as a Bluetooth HID mouse/keyboard where debugging and screenshots are blocked. Coordinate-only, so pair it with camera_screenshot and the vision strategies.
• Agent-ready — optics mcp exposes every keyword as a typed MCP tool and device state as MCP resources.

Install

Optics needs Python 3.12+. The core install ships no drivers, OCR, or LLM backends — you add only what you need:

python3 -m venv venv && source venv/bin/activate
pip install "optics-framework[appium,easyocr]"

Extra names match the config.yaml source keys, so the word you install is the word you enable:

Or install them by name — optics setup pins to your installed Optics version, and bare optics setup opens a TUI picker:

optics setup --list
optics setup --install appium easyocr

[!IMPORTANT] A driver extra installs only the Python client. Mobile testing also needs the Appium server, a device/emulator, and platform tooling (Node.js, Android SDK/adb, JDK). See the Installation & Prerequisites guide.

[!WARNING] Conda is not supported for easyocr + optics-framework together (conflicting NumPy 1.x/2.x requirements). Use a standard venv.

Quickstart

optics init --name my_test_project --template contact
# point my_test_project/config.yaml at your device/app, start the Appium server, then:
optics dry_run my_test_project    # validate keywords, elements and module refs — no device needed
optics execute my_test_project

--template scaffolds a working project from a bundled sample: contact, calendar, youtube (Appium/Android), clock (Android + image templates), gmail_web (Selenium), playwright (Playwright). Omit it for an empty scaffold with a commented starter config.yaml.

Write a test as data

my_test_project/
├── config.yaml
├── test_cases/test_cases.csv
├── modules/modules.csv
└── test_data/
    ├── elements.csv
    ├── error_definitions.csv      # optional
    └── input_templates/*.png      # optional, for image matching

test_data/elements.csv — names mapped to locators. Doubles as a general variable store; repeating a name builds a fallback list.

Element_Name,Element_ID
Add_Contact_Button,//android.widget.Button[@content-desc="Create contact"]
First_Name_element,//android.widget.EditText[@text="First name"]
Save_Button,Save
First_Name,John

A locator can be an XPath, text=…, css=…, a plain string, an image filename from input_templates/, or TEXT_ONLY:… to force a vision-based search.

modules/modules.csv — a reusable sequence of keywords; ${name} resolves against elements.csv.

module_name,module_step,param_1,param_2
Add Contact,Press Element,${Add_Contact_Button}
Add Contact,Enter Text,${First_Name_element},${First_Name}
Add Contact,Press Element,${Save_Button}

test_cases/test_cases.csv — modules sequenced into scenarios. A test case whose name contains suite + setup (or teardown) is hoisted to run around the whole suite.

test_case,test_step
Suite Setup,Launch Contact Application
Add Contact with Contact App,Add Contact
Add Contact with Contact App,Verify Contact is Added

Six ways to run the same keywords

/save <test_case> <module_name>

from optics_framework import Optics

optics = Optics()
optics.setup(
    driver_sources=[{"appium": {"enabled": True, "url": "http://localhost:4723"}}],
    elements_sources=[{"appium_find_element": {"enabled": True}}],
)

optics.launch_app("com.example.app")
optics.enter_text("username_field", "testuser")
optics.press_element("submit_button")
optics.validate_element("welcome_message")
optics.quit()

{ "mcpServers": { "optics": { "command": "optics", "args": ["mcp"] } } }

Keywords

Every public method on the four API classes is automatically a keyword, on every surface above. CSV/YAML uses Title Case (Press Element → press_element).

Run optics list for the live catalogue with signatures, or read the Keyword Usage guide for parameters and examples. Location keywords accept percentage-based Area-of-Interest bounds (aoi_x/y/width/height, 0–100) to scope a vision search to part of the screen.

[!NOTE] Press Checkbox and Press Radio Button still resolve but are deprecated aliases of Press Element — use Press Element directly. Add API is available on the Optics Python class only, not to the CSV/YAML runner; define APIs in an api.yaml and call them with Invoke API instead.

Configure once, in config.yaml

Every section is a priority-ordered list and every entry has an enabled flag. Enable a second driver and it becomes a fallback.

driver_sources:
  - appium:
      enabled: true
      url: "http://localhost:4723"
      capabilities:
        platformName: Android
        automationName: UiAutomator2
        deviceName: emulator-5554
        appPackage: com.google.android.contacts
        appActivity: com.android.contacts.activities.PeopleActivity

elements_sources:
  - appium_find_element: { enabled: true }
  - appium_page_source:  { enabled: true }
  - appium_screenshot:   { enabled: true }

text_detection:
  - easyocr: { enabled: true }

image_detection:
  - templatematch: { enabled: false }

log_level: INFO

Enabling the LLM features

The llm_models block powers both natural-language mode in optics live (Ctrl-N) and AI self-heal. Install the extra (pip install "optics-framework[llm]"), then add:

llm_models:
  - gemini:
      enabled: true
      capabilities:
        model: gemini-2.5-flash    # optional; this is the default
        # use_vertexai: true       # optional; else read from the environment
        # project: my-gcp-project  # optional (Vertex)
        # location: us-east4       # optional (Vertex)

ai_self_heal: true                 # opt into the LLM backstop; default false

Credentials are read from the environment by the google-genai SDK — GEMINI_API_KEY (or GOOGLE_API_KEY) for the Gemini Developer API, or GOOGLE_GENAI_USE_VERTEXAI + GOOGLE_CLOUD_PROJECT / GOOGLE_CLOUD_LOCATION / GOOGLE_APPLICATION_CREDENTIALS for Vertex AI. Never commit keys to config.yaml. With every capabilities key omitted the SDK auto-detects the backend. google-genai is imported only when gemini is enabled, and a misconfigured LLM degrades to "no self-heal" rather than a hard failure.

Full reference: Configuration. Adding your own engine is a file drop plus an interface — see Extending the Framework.

Results

An optics execute run writes to <project>/execution_output/:

• junit_output.xml — written incrementally, so CI sees progress as it happens
• logs.json — structured logs when json_log: true
• screenshots — pre/post action frames, plus strategy-annotated and AOI overlays
• detected_errors_<session_id>.json — on-screen error detection

Drop an error_definitions.csv into test_data/ and Optics scans visible text for crash dialogs, Session expired, network errors, and the like — no assertions required. Matches also land in the JUnit XML as a synthetic failing testcase, so CI fails a build on "the app crashed mid-test" the same way it fails a normal assertion. See Error Detection.

CLI reference

optics init        Scaffold a new project (--template, --path, --force, --git-init)
optics setup       Install engine backends (--list, --install); bare command opens a TUI
optics dry_run     Validate a project without touching a device
optics execute     Run a project (--runner test_runner|pytest)
optics live        Interactive keyword session against a live target
optics generate    Emit pytest or Robot Framework code from a project
optics list        Print every discoverable keyword
optics serve       Start the REST API server (--host, --port, --workers)
optics mcp         Start the MCP server (--transport stdio|http)
optics config      Manage global configuration (interactive)
optics completion  Install shell autocompletion
optics --version   Print the installed version

Details in the CLI guide.

Contributing

git clone git@github.com:mozarkai/optics-framework.git
cd optics-framework
pipx install poetry
poetry install --with dev,test,docs

poetry run pytest                    # tests + coverage
poetry run ruff check --fix .        # lint
poetry run pre-commit run --all-files
poetry run mkdocs serve              # docs preview

Commits follow Conventional Commits, enforced by commitizen in the commit-msg hook. Read the Contributing Guidelines, the Developer Guide, and our Code of Conduct before opening a PR. Looking for a place to start? See Help Wanted.

Security issues: please follow SECURITY.md rather than opening a public issue.

License & support

Apache 2.0 — see LICENSE.

Questions and bugs: GitHub Issues. Anything else: lalit@mozark.ai.