okareo-vision

okareo-vision is the browser side of an Okareo simulation of an agent that lives behind a web page: you describe the page in a small Python class, and it opens the page in Chromium, runs the moves Okareo sends each turn, and answers with what the screen shows. Okareo plays the customer with a Driver you write, reads every screen, and scores the conversation; nothing in this package reads the screen or calls a model.

Install

pip install okareo-vision-beta
playwright install chromium

The first releases are published as okareo-vision-beta while the name okareo-vision moves to Okareo's own PyPI account; the import is okareo_vision either way.

Python 3.10 to 3.12, in a virtualenv of your own. It installs the Okareo SDK (okareo) and Playwright; playwright install chromium downloads the browser once (on Linux, playwright install --with-deps chromium also installs the libraries it needs). A Run needs an Okareo account and its API key.

Run the example

The example is in the repository, beside a local copy of a made-up store's page (Shopwise, with a shopping helper called Sprig), so it needs no site of your own.

git clone https://github.com/okareo-ai/okareo-vision.git
cd okareo-vision/examples/retail_chat
export OKAREO_API_KEY=<your Okareo API key>
export OKAREO_BASE_PATH=https://api.okareo.com   # leave it unset for a local Okareo server at http://localhost:8000
python run_retail_chat.py --debug

It creates a Scenario with two rows, a Driver and a Target, runs both conversations in a headless browser, and prints the link to the finished Run in the Okareo app (on a local server, the app at http://localhost:3000). --debug keeps a record of the run under okareo_runs/ (see The run record). --headed shows the browser, --max-turns N caps each conversation (default 20), and --name names the Run, its Scenario and the Target (the Driver is always "Shopwise shopper"); rerunning under the same name reuses the Scenario, so change it after editing scenarios.jsonl. OKAREO_PROJECT picks an Okareo project by name or id; unset, the default project.

Its three files are everything you write for a site of your own:

File	What it is
run_retail_chat.py	ShopwiseHelper, a VisionTarget subclass (the page, a note about it, a prepare step), and the SDK calls that start the Run
driver.md	the Driver: who the customer is and how they behave
scenarios.jsonl	the Scenario rows: per line, an input with the facts the Driver uses and a result saying what a finished conversation looks like

The example's page is a made-up store's page served from a file, so it needs no site and no sign-in.

How a turn works

Okareo does the looking. Each turn it sends a batch of commands; okareo-vision runs them in order and answers with a frame: a JPEG of the screen, its hash, the page's visible text, the controls on screen (role, name, state and box, from the browser's accessibility tree), whether the screen changed, and the outcome of each command. A command that fails says why in its result; the turn goes on, and so does the Run. There are six commands:

Command	What it does
click	Presses a control by its place on the screen, by its name, or as an entry from the control list. With a name and a place, what is under the place must carry that name; otherwise nothing is pressed and the result says what is there.
type	Types into the focused field, replacing what it held, and reads the value back. Nothing focused, or a value the page rewrote, is a failed command.
press	Presses one key: Enter, Tab, Escape, ArrowDown.
scroll	Turns the mouse wheel over a place on the screen (an inner list moves, not the page), or over the middle of the screen.
compose	Puts a message in the chat's message box and sends it: Enter, then the send button beside the box if Enter only added a new line.
wait	Waits for the screen to settle.

The screen has settled when captures taken a moment apart stay identical for a quiet window, 1.2 seconds, for at most settle_timeout_s (Okareo can ask for other values in a wait). The first frame of every conversation waits the same way. A spinner that never stops is reported as not settled; what to do about it is the Driver's call.

VisionTarget

Configure it by subclassing, with class attributes, or with constructor arguments; an argument wins over the attribute.

from okareo_vision import VisionTarget, click_selector

class SupportChat(VisionTarget):
    url = "https://support.example.com/"
    app_notes = "The chat opens from the 'Help' button at the bottom right."

target = SupportChat(headless=False, max_parallel=3)

# the same page, no subclass
target = VisionTarget(url="https://support.example.com/",
                      app_notes="The chat opens from the 'Help' button at the bottom right.",
                      prepare=click_selector("#help-button"))

Setting	Default	What it does
url	required	The page every conversation opens. {session_index} (1, 2, … in start order) and {scenario_input.KEY} (a field of the row's input) are filled per conversation. Without a url the constructor raises TypeError.
app_notes	""	What Okareo is told about the page: where the chat is, what the side panels are, what has to happen before a request can start.
prepare(page)	does nothing	Gets the page ready; see prepare. A method on the subclass, or prepare= on the constructor.
name	the class name	The Target's name in Okareo. Constructor only.
viewport	(1280, 900)	The browser window's size; frames come in it.
max_parallel	10	How many conversations Okareo may hold open at once. They share one browser, each in its own context (its own cookies and storage).
settle_timeout_s	30.0	The longest a wait for the screen to settle lasts.
headless	True	False shows the browser.
channel	None	Which browser to launch. None is Playwright's Chromium; "chrome", "msedge", "chrome-beta", "msedge-beta" are browsers installed on the machine (or fetched once with playwright install chrome).
user_agent	None	None presents a normal Chrome of the running version; "" keeps Playwright's own; any other string is used as is.
browser_args	()	Extra Chromium command-line flags.
storage_state	None	A Playwright storage-state file (a saved sign-in); every conversation starts from it.
screen_selector	None	A CSS selector for the one element Okareo is shown as the screen, for a chat panel docked over a busy page. Frames, visible text, the control list and Okareo's points are that element's. It must be on the page whenever a frame is taken; when it is not, the turn fails. None is the whole window.
composer_selector, send_selector	""	The chat's message box and send button. When set, compose fills that box and clicks that button instead of using Okareo's point and the Enter key.
debug	off	Records a video, a HAR and a screenshot per turn; see The run record.
run_dir	None	A RunDir (from okareo_vision.diagnostics) to write the run record to. Debug mode makes one when none is given.

VisionTarget.preflight() launches and closes the browser once, so a missing browser fails in your terminal instead of once per conversation inside a Run.

Environment variables, for settings without code:

Variable	What it does
OKAREO_WEB_TARGET_DEBUG=1	Debug mode on.
OKAREO_WEB_TARGET_HAR=full	The HAR keeps cookies, headers and security details (otherwise Playwright's minimal HAR).
OKAREO_RUN_DIR	Where run directories go instead of okareo_runs/.
OKAREO_WEB_BROWSER	The channel, e.g. chrome.
OKAREO_WEB_PERCEPTION	The perception mode Okareo runs this Target with, vision or ax, sent when the Target registers; a Run's own simulation settings outrank it.

The package also exports Perception, Resolver, WizardExecutor and parse_driver_values, a separate rule-based engine that VisionTarget does not use and this page does not cover.

prepare

prepare(page) runs once per conversation, after the page loads and before Okareo's first look. The Playwright Page is yours: close a banner, sign in, open the chat, and wait for the message box so the first frame has the chat in it. Raising ends that conversation with the error on its row; the Run goes on. The example's, from run_retail_chat.py:

class ShopwiseHelper(VisionTarget):
    url = REPLICA.as_uri() + "?plain=1&notice=1&greet=1"   # the store's page, opened from a file
    app_notes = (
        "This is Shopwise, an online store for the backyard. Its shopping helper, Sprig, opens as a "
        "drawer on the right side of the page. The message box at the bottom of the drawer reads "
        "'Ask anything'; a message is sent with the arrow button beside it (Enter only adds a new "
        "line). ..."
    )

    async def prepare(self, page: Page) -> None:
        # A cookie notice can cover the page on first load; its X closes it.
        notice = page.get_by_role("dialog", name="Cookies on Shopwise")
        if await notice.count() and await notice.is_visible():
            await notice.get_by_role("button", name="Close", exact=True).click()
            await notice.wait_for(state="hidden", timeout=5000)
        # The helper opens from "Ask Shopwise", or from the round button in the corner.
        pill = page.get_by_role("button", name="Ask Shopwise")
        if await pill.count() and await pill.is_visible():
            await pill.click()
        else:
            await page.locator('[data-testid="chat-bubble"]').click(force=True)
        await page.get_by_role("textbox", name="Ask anything").wait_for(state="visible", timeout=10000)
        # The helper can remember an earlier visit; Reset chat starts clean.
        reset = page.get_by_role("button", name="Reset chat")
        if await reset.count() and await reset.is_visible():
            await reset.click()
            await page.get_by_text("I'm Sprig").first.wait_for(state="visible", timeout=10000)

When one click is all it takes, click_selector("#help-button") is a ready prepare: it clicks the control, and carries on with a warning when the control is not there.

The Driver is yours

The package ships no Driver. A Driver is a prompt you write and register with the Okareo SDK: who the customer is, what they want, how they behave, and when they are done. Okareo adds what every Driver in a web Run needs to read a screen and make moves, so your text is about the customer only. {scenario_input.KEY} is filled from the row's input. The start of the example's driver.md:

## Persona

-   **Identity:** You are role-playing a shopper on a store's website, chatting with the store's shopping helper to get one thing done.
-   **Mindset:** Friendly and to the point. You know what you want and how much you will spend; you answer the helper's questions from the Scenario Details and stop as soon as your request is done.

## Scenario Details

What you want: {scenario_input.task}
Budget: {scenario_input.budget}
Notes: {scenario_input.notes}

## Objectives

1. Open by telling the helper what you are shopping for, in one short sentence in your own words.
2. Answer the helper's questions from the facts above, and pick from what it shows you.
3. Stop once the helper has answered everything your request asks for. A question from the helper is something to answer, not the end.

It goes on with tactics, hard rules and when to end. A row that fills it:

{"input": {"test_case": "SHOP-02", "task": "Find out which bag sizes the lump charcoal comes in.", "budget": "under $40", "notes": "Ask for lump charcoal first, then press the 'Show bag sizes' suggestion under Sprig's reply."}, "result": "Sprig suggests lump charcoal under $40 and says its bags come in 15 lb and 20 lb."}

And the SDK calls that run it, shortened from the example (the example reads the server, the project and the names from its flags and environment):

import json, os
from okareo import Okareo
from okareo.model_under_test import Driver, Target
from okareo_api_client.models import ScenarioSetCreate

okareo = Okareo(api_key=os.environ["OKAREO_API_KEY"], base_path="https://api.okareo.com")
rows = [json.loads(line) for line in open("scenarios.jsonl") if line.strip()]
scenario = okareo.create_scenario_set(ScenarioSetCreate(
    name="Shopwise shopper", seed_data=Okareo.seed_data_from_list(rows)))
driver = Driver(name="Shopwise shopper", prompt_template=open("driver.md").read(), temperature=0.0)

ShopwiseHelper.preflight()
run = okareo.run_simulation(
    name="Shopwise shopper",
    scenario=scenario,
    target=Target(name="Shopwise shopper", target=ShopwiseHelper()),
    driver=driver,
    checks=["result_completed"],
    max_turns=20,
    first_turn="driver",
)
print(run.app_link)

run_simulation serves the browser side from this process until the Run finishes. The browser runs where this process runs, so the page can be on your own network.

The run record

With --debug in the example, debug=True on the Target, or OKAREO_WEB_TARGET_DEBUG=1, the Target writes a record of its conversations to a directory under okareo_runs/ in the working directory (or under $OKAREO_RUN_DIR), named for the time the first one started, in UTC. A RunDir passed as run_dir= writes the same record without the video and the HAR.

okareo_runs/20261006-063959/
  turns.jsonl            one line per turn, and one per conversation's video
  shots/                 turn00-9d0efd23.jpg: each frame Okareo saw
  ax/                    turn00-9d0efd23.json: the control list read from the page
  dom/                   turn00-9d0efd23.html.gz: the page's HTML at that frame
  video/                 9d0efd23.webm per conversation (and .mp4 with ffmpeg); debug mode
  9d0efd23.har           the page's network traffic per conversation; debug mode

9d0efd23 is a conversation, turn00 its first frame, taken before any turn and so with no line of its own; turn N's files are turnNN-9d0efd23.*. The example's first turn in that conversation, one line of turns.jsonl shown across several:

{
  "turn": 1,
  "session": "9d0efd23",
  "commands": [
    {"op": "compose", "text": "I’m looking to buy a bag of lump charcoal for grilling, under $40.", "x": 1073, "y": 867},
    {"op": "wait", "quiet_s": 1.2, "timeout_s": 30.0}
  ],
  "results": [
    {"op": "compose", "ok": true},
    {"op": "wait", "ok": true, "settled": true}
  ],
  "frame_sha1": "7fb9cafdec473e0df1dfc424887afe59c2349b44",
  "video_offset_s": 10.29,
  "browser_ms": 1354,
  "at": "2026-10-06T06:40:09+00:00"
}

Field	What it holds
turn, session	the turn's number in its conversation, and the conversation
commands	the commands as Okareo sent them
results	one per command: op, ok, error when it failed, and what the command reports (for click: how, which is name, point or typed, with name, at and scrolled; landed and value for type; settled for wait)
frame_sha1	the SHA-1 of the capture the settle wait compared, the screenshot before it is encoded as the JPEG in shots/; it is not the saved file's hash
video_offset_s	where this turn is in the conversation's video, in seconds
browser_ms	the time this turn spent in the browser
at	when the line was written, UTC

When a conversation ends in debug mode, one more line names its video: {"session": "9d0efd23", "video": ".../video/9d0efd23.webm", "at": "..."}.

Three more files are written when your runner asks for them, with the helpers in okareo_vision.diagnostics and a RunDir passed to the Target: setup(run_dir) logs to run.log; preflight(run_dir, okareo_base=..., app_url=...) writes preflight.json (Python, Playwright, proxy settings, and whether Okareo and the page can be reached); report(run_dir, run_link=...) writes report.txt, a per-turn summary with no credentials in it.

The record holds whatever the page showed and sent: the screenshots, the HTML and the video show all of it. The HAR is recorded in Playwright's minimal mode (no cookies, headers or security details) unless OKAREO_WEB_TARGET_HAR=full. The video is written when its conversation ends.

The server it needs

okareo-vision speaks version 1 of Okareo's web simulation protocol, and says so when the Target registers. Okareo's cloud (https://api.okareo.com) speaks it, and a local Okareo server works the same; the example talks to http://localhost:8000 unless OKAREO_BASE_PATH says otherwise. The oldest Okareo SDK it works with is 0.0.151, the floor of the okareo it installs.

The traffic goes two ways. To Okareo: HTTPS to the API, and NATS over a WebSocket to wss://connect.ngs.global:443, which every Run needs, a local server included, so the machine needs that outbound connection. To the page: the browser, directly. Chromium does not read HTTPS_PROXY, so a page that must be reached through a proxy needs it as a browser flag, browser_args=["--proxy-server=http://proxy.example.com:8080"].

When it goes wrong

The page never shows a chat

• Watch it: headless=False (the example's --headed), or --debug and open shots/turn00-*.jpg, the first frame Okareo saw.
• prepare is where the chat gets opened. End it by waiting for the message box, as the example does, so the first frame has the chat in it; when prepare raises, the error is on the row.
• Say in app_notes where the chat is and what opens it.
• A chat in a panel over a busy page: screen_selector shows Okareo the panel alone.
• When Okareo cannot find the message box: composer_selector and send_selector name it.

The site refuses headless Chrome

The browser already presents itself as a normal Chrome: the automation marker (navigator.webdriver) is off and the user agent has no "HeadlessChrome" in it. When the site still refuses, show the browser and use the real Chrome: headless=False and channel="chrome" (or OKAREO_WEB_BROWSER=chrome; run playwright install chrome if it is not on the machine). Sites with a bot gate commonly admit only the real Chrome, headed. A headed browser needs a display; on a Linux server, run under xvfb-run.

Sign-in needs MFA

A scripted sign-in cannot answer an MFA prompt. Sign in by hand once, save the session, and start every conversation from it:

from okareo_vision import VisionTarget
from okareo_vision.auth import ensure_session

with ensure_session("https://portal.example.com/", storage_state="portal_session.json",
                    login_timeout=300) as (session, page):
    print("signed in at", session["landed_url"])

class Portal(VisionTarget):
    url = "https://portal.example.com/"
    storage_state = "portal_session.json"

With no recipe, ensure_session opens a visible browser at the url and waits up to login_timeout seconds (600 unless you say) for you to sign in, MFA included, then saves the session to storage_state with file mode 0600. Run it before each Run: it reuses the saved session and asks you to sign in again only when it no longer works (verify=, a function of (page, api_base) returning a bool, says what "works" means for your site). The file is a signed-in session; keep it out of version control. form_login(...) and entra_login(...) in the same module are scripted sign-ins for sites without MFA.

What is not in the package

• A Driver. Yours, registered with the Okareo SDK.
• Model calls. This process holds no model key; Okareo reads the screens and picks the moves.
• Scenarios and Checks. Made with the Okareo SDK, as in the example.
• The example and test pages. They are in the repository, not in the wheel.
• Anything about a particular site. Its url, sign-in and selectors belong in your subclass.

Documentation: https://docs.okareo.com/simulation/web-ui. Changes: CHANGELOG.md. License: Apache-2.0.