okareo-vision

okareo-vision is the browser side of an Okareo simulation of an agent that lives behind a web page: you describe the page in a small Python class, and it opens the page in Chromium, runs the moves Okareo sends each turn, and answers with what the screen shows. Okareo plays the customer with a Driver you write, reads every screen, and scores the conversation; nothing in this package reads the screen or calls a model.

Install

pip install okareo-vision
playwright install chromium

Python 3.10 to 3.12, in a virtualenv of your own. It installs the Okareo SDK (okareo) and Playwright; playwright install chromium downloads the browser once (on Linux, playwright install --with-deps chromium also installs the libraries it needs). A Run needs an Okareo account and its API key.

Run the example

The example is in the repository, beside a local copy of a made-up store's page (Shopwise, with a shopping helper called Sprig), so it needs no site of your own.

git clone https://github.com/okareo-ai/okareo-vision.git
cd okareo-vision/examples/retail_chat
export OKAREO_API_KEY=<your Okareo API key>
export OKAREO_BASE_PATH=https://api.okareo.com   # leave it unset for a local Okareo server at http://localhost:8000
python run_retail_chat.py --debug

It creates a Scenario with two rows, a Driver and a Target, runs both conversations in a headless browser, and prints the link to the finished Run in the Okareo app (on a local server, the app at http://localhost:3000). --debug keeps a record of the run under okareo_runs/ (see The run record). --headed shows the browser, --max-turns N caps each conversation (default 20), and --name names the Run, its Scenario and the Target (the Driver is always "Shopwise shopper"); rerunning under the same name reuses the Scenario, so change it after editing scenarios.jsonl. OKAREO_PROJECT picks an Okareo project by name or id; unset, the default project.

Its three files are everything you write for a site of your own:

The example's page is a made-up store's page served from a file, so it needs no site and no sign-in.

How a turn works

Okareo does the looking. Each turn it sends its moves; okareo-vision runs them in order and answers with a frame: a JPEG of the screen, its hash, the page's visible text, the controls on screen (role, name, state and box, from the browser's accessibility tree), whether the screen changed, and the outcome of each command. A command that fails says why in its result; the turn goes on, and so does the Run. A failed type stops the rest of its turn: the moves after it are not run (a wait still is), so nothing is submitted on a wrong value. There are six moves:

A turn is done the way browser automation usually decides it, with the picture keeping the last word. The page and its frames must first go half a second with no request of the page's own site in flight, no change in the DOM (open shadow roots included), and nothing drawn in the accessibility tree saying it is busy (the busy state, or a progress bar with no value; a bar at opacity 0 is not drawn). A request is over when the browser says so, when its response came with a known length, or when the page it belonged to went to another page (the browser's error page too) or its frame was taken off the page; never because it is old. Then two captures 400 ms apart must match, with the box being typed in left out of both (a rich-text editor that is a whole document is left out as a whole): a blinking cursor is not a change, and a canvas app's paint still holds the wait. A signal that keeps changing with no half-second pause through the first 2 seconds (analytics beacons, an animation in the DOM) is left out of that wait; one request held open, or one busy node, holds it up to the limit, by design, and the record names it. Nothing is configured per site. A wait lasts at most settle_timeout_s, or the limit Okareo sends with it, and the first frame of every conversation waits the same way. A spinner that never stops is reported as not settled; what to do about it is the Driver's call.

Each wait's result says how it ended, beside settled: ended_by (picture, or ceiling), held_by (the signals that held it), held (what held it: the request open longest, by its host and path, and the busy node, by its role and name), dropped (the signals left out for churning), waited_ms, and what the signals saw (requests, mutations, busy_ms). The log has a line for every wait, naming what held it.

A type works on a box that formats itself as it is typed into (a date box that puts in its own slashes, a phone or card mask, a currency box) the way a person does: it types the letters and digits in runs, and before each separator it looks at the box, skipping its own where the box already put one. Once a look shows the box holding exactly what was typed, it is not formatting, and the rest of the value is typed in one burst: a long answer is as quick as a short one. The read-back accepts the box's formatting: 555-123-4567 is held by (555) 123-4567 and 1234.50 by 1,234.50, while a box that dropped or moved one of the value's separators, or holds a different date, has not taken it. A suggestion the box selected after the typed text is deleted. A miss is cleared and typed once more, a key every 40 ms, or in one burst for a value over 25 characters; a box whose maxlength cannot hold the value is not typed again. A date is written in the shape the box names in its placeholder or its name (MM/DD/YY, MM-DD-YYYY), month first only, and a two-digit year from the Driver is written as four, which the result says (typed 1992 for 92); a box that names nothing and takes two digits of the year gets two. The box read back is the one typing began in, even after the page moves focus on. A row of one-character boxes (a code's digit boxes, each moving focus to the next) is read as one value from its first box: the boxes from the one typing began in, forward in document order, while each is of the first one's kind (each declaring the first one's maxlength, or none when it declares none, and each declaring maxlength 1 or holding one character), wherever focus went after the last digit (the next box, a Submit button). Any of them holding something is cleared before the code is typed. A value of one letter or digit never reads or clears a row. A middle-initial box is out of a first-name box's row when it declares maxlength 1 and the first-name box does not: two adjacent boxes with no maxlength (or the same one, over 1), each holding one character, look like a code row and are read as one. The result has landed and value, and when they apply typed_as, retried and note. The browser's own date input is set through its value setter instead.

A click by name alone presses the one control in the control list that answers to the name, scrolled into view when it is off screen or covered. A row of an open list that the page has not built yet (a list that builds only the rows near what it shows, as Flutter's and react-window's do) is reached by paging the list: the wheel turns over the list a page at a time, down then up, at most 12 pages each way, and the row is pressed once it appears. A list is open when a row of it is on top, or when a control that says it is expanded names it (aria-controls); a list that only stands on the page is never paged or scrolled to. A list an expanded control names that opened past the edge of the scroller it sits in is scrolled into view first. The result then has scrolled and a note saying how many pages it took. Nothing else is pressed, and a name that never appears fails the command: no control named ....

VisionTarget

Configure it by subclassing, with class attributes, or with constructor arguments; an argument wins over the attribute.

from okareo_vision import VisionTarget, click_selector

class SupportChat(VisionTarget):
    url = "https://support.example.com/"

target = SupportChat(headless=False, max_parallel=3)

# the same page, no subclass
target = VisionTarget(url="https://support.example.com/", prepare=click_selector("#help-button"))

VisionTarget.preflight() launches and closes the browser once, so a missing browser fails in your terminal instead of once per conversation inside a Run.

Environment variables, for settings without code:

prepare

prepare(page) runs once per conversation, after the page loads and before Okareo's first look. The Playwright Page is yours: close a banner, sign in, open the chat, and wait for the message box so the first frame has the chat in it. Raising ends that conversation with the error on its row; the Run goes on. The example's, from run_retail_chat.py:

class ShopwiseHelper(VisionTarget):
    url = REPLICA.as_uri() + "?plain=1&notice=1&greet=1"   # the store's page, opened from a file

    async def prepare(self, page: Page) -> None:
        # A cookie notice can cover the page on first load; its X closes it.
        notice = page.get_by_role("dialog", name="Cookies on Shopwise")
        if await notice.count() and await notice.is_visible():
            await notice.get_by_role("button", name="Close", exact=True).click()
            await notice.wait_for(state="hidden", timeout=5000)
        # The helper opens from "Ask Shopwise", or from the round button in the corner.
        pill = page.get_by_role("button", name="Ask Shopwise")
        if await pill.count() and await pill.is_visible():
            await pill.click()
        else:
            await page.locator('[data-testid="chat-bubble"]').click(force=True)
        await page.get_by_role("textbox", name="Ask anything").wait_for(state="visible", timeout=10000)
        # The helper can remember an earlier visit; Reset chat starts clean.
        reset = page.get_by_role("button", name="Reset chat")
        if await reset.count() and await reset.is_visible():
            await reset.click()
            await page.get_by_text("I'm Sprig").first.wait_for(state="visible", timeout=10000)

When one click is all it takes, click_selector("#help-button") is a ready prepare: it clicks the control, and carries on with a warning when the control is not there.

cleanup(page) is its counterpart: it runs at the end of each conversation, before the page closes. A failure in it is logged; the page closes anyway.

Signing in once per Run

A site behind a sign-in signs in once per Run, not once per conversation: override sign_in(page) and sign_out(page), and run the simulation inside a with block.

class Portal(VisionTarget):
    url = "https://portal.example.com/support"

    async def sign_in(self, page: Page) -> None:
        await page.goto("https://portal.example.com/login")
        await page.fill("#username", os.environ["PORTAL_USER"])
        await page.fill("#password", os.environ["PORTAL_PASS"])
        await page.click("#sign-in")
        await page.get_by_text("Your account").wait_for()    # raises if it did not land

    async def sign_out(self, page: Page) -> None:
        await page.goto("https://portal.example.com/logout")
        await page.get_by_text("You are signed out").wait_for()

with Portal() as target:     # sign_in runs here, before anything is submitted
    okareo.run_simulation(..., target=Target(name="Portal", target=target), ...)
# leaving the block runs sign_out, even when the Run failed

• sign_in runs when the block opens, in a short-lived browser launched with the Target's own settings (headless, channel, browser_args), on a thread of its own. The session it leaves (cookies and local storage, and IndexedDB on Playwright 1.51 or later) is held in memory, never written to disk, and every conversation of the Run starts from it. If the sign-in raises, sign_out runs on the same page, best effort, and the error is raised: nothing has been submitted. Ctrl-C during the sign-in lets it finish, signs out, then stops.
• sign_out runs when the block closes (target.close()), in a short-lived browser holding the session, even when the Run failed. Raise in it when the sign-out did not land. Leaving the block never raises from the sign-out: target.signed_out is True when it landed and False when it raised (the error is logged). Ctrl-C during the sign-out lets it finish, then stops. A Target that does not override sign_out launches nothing when the block closes, and signed_out stays None.
• Okareo's SDK does not tell a Target that its Run is over; the end of the block says it. On Ctrl-C during the Run, stop the Run's listener (okareo.model_under_test.stop_listener_for_run(run_id)) and let the block sign out.
• Nothing signs in again during the Run. A conversation that lands signed out ends with its error.
• The sign-in, the conversations and the sign-out present the same Chrome: each context's user agent is read from the browser it is opened in.

Changing how a command acts

A Target may override the commands Okareo sends, for a page the defaults cannot work (a chat painted on a canvas, say): click(st, x, y, label=None, near=None), type_text(st, text), press(st, key), scroll(st, x, y, dy), compose(st, text, x, y), and settle(st, *, quiet_s, timeout_s, poll_s, until_change), the wait. settle returns a quiet.Done: the last capture (shot), its sha1, whether the turn settled, and the record of how the wait ended; an override returns one too. Call super() for the default. st is the conversation's state: st["page"] is its Playwright page, and nothing else in st is part of the package's surface. Points arrive in the page's pixels; with a screen_selector, Okareo's points have been moved onto the page already.

controls(st) reads the screen's control list as it is now, the one sent to Okareo with each frame. Each entry has ref, role, names (each a text and the source it came from), display (the first name), value, states, within (the list, menu or dialog it sits in, as role and id, or none) and bbox (x, y, w, h). At most 80 entries, then one whose role is _dropped: its dropped lists the controls off screen or covered, each with its bbox and a why. Boxes are in the pixels of the screen Okareo is shown, the screen_selector element's when there is one; screen_origin(st) is where that screen sits on the page, (0, 0) without one. Add it to a box to press or turn the wheel there with the page's mouse.

The Driver is yours

The package ships no Driver. A Driver is a prompt you write and register with the Okareo SDK: who the customer is, what they want, how they behave, and when they are done. Okareo adds what every Driver in a web Run needs to read a screen and make moves, so your text is about the customer only. {scenario_input.KEY} is filled from the row's input. The start of the example's driver.md:

## Persona

-   **Identity:** You are role-playing a shopper on a store's website, chatting with the store's shopping helper to get one thing done.
-   **Mindset:** Friendly and to the point. You know what you want and how much you will spend; you answer the helper's questions from the Scenario Details and stop as soon as your request is done.

## Scenario Details

What you want: {scenario_input.task}
Budget: {scenario_input.budget}
Notes: {scenario_input.notes}

## Objectives

1. Open by telling the helper what you are shopping for, in one short sentence in your own words.
2. Answer the helper's questions from the facts above, and pick from what it shows you.
3. Stop once the helper has answered everything your request asks for. A question from the helper is something to answer, not the end.

It goes on with tactics, hard rules and when to end. A row that fills it:

{"input": {"test_case": "SHOP-02", "task": "Find out which bag sizes the lump charcoal comes in.", "budget": "under $40", "notes": "Ask for lump charcoal first, then press the 'Show bag sizes' suggestion under Sprig's reply."}, "result": "Sprig suggests lump charcoal under $40 and says its bags come in 15 lb and 20 lb."}

And the SDK calls that run it, shortened from the example (the example reads the server, the project and the names from its flags and environment):

import json, os
from okareo import Okareo
from okareo.model_under_test import Driver, Target
from okareo_api_client.models import ScenarioSetCreate

okareo = Okareo(api_key=os.environ["OKAREO_API_KEY"], base_path="https://api.okareo.com")
rows = [json.loads(line) for line in open("scenarios.jsonl") if line.strip()]
scenario = okareo.create_scenario_set(ScenarioSetCreate(
    name="Shopwise shopper", seed_data=Okareo.seed_data_from_list(rows)))
driver = Driver(name="Shopwise shopper", prompt_template=open("driver.md").read(), temperature=0.0)

ShopwiseHelper.preflight()
run = okareo.run_simulation(
    name="Shopwise shopper",
    scenario=scenario,
    target=Target(name="Shopwise shopper", target=ShopwiseHelper()),
    driver=driver,
    checks=["result_completed"],
    max_turns=20,
    first_turn="driver",
)
print(run.app_link)

run_simulation serves the browser side from this process until the Run finishes. The browser runs where this process runs, so the page can be on your own network.

The run record

With --debug in the example, debug=True on the Target, or OKAREO_WEB_TARGET_DEBUG=1, the Target writes a record of its conversations to a directory under okareo_runs/ in the working directory (or under $OKAREO_RUN_DIR), named for the time the first one started, in UTC. A RunDir passed as run_dir= writes the same record without the video and the HAR.

okareo_runs/20261006-063959/
  turns.jsonl            one line per turn, and one per conversation's video
  shots/                 turn00-9d0efd23.jpg: each frame Okareo saw
  ax/                    turn00-9d0efd23.json: the control list read from the page
  dom/                   turn00-9d0efd23.html.gz: the page's HTML at that frame
  video/                 9d0efd23.webm per conversation (and .mp4 with ffmpeg); debug mode
  9d0efd23.har           the page's network traffic per conversation; debug mode

9d0efd23 is a conversation, turn00 its first frame, taken before any turn and so with no line of its own; turn N's files are turnNN-9d0efd23.*. The example's first turn in that conversation, one line of turns.jsonl shown across several:

{
  "turn": 1,
  "session": "9d0efd23",
  "commands": [
    {"op": "compose", "text": "I’m looking to buy a bag of lump charcoal for grilling, under $40.", "x": 1073, "y": 867},
    {"op": "wait", "quiet_s": 1.2, "timeout_s": 30.0}
  ],
  "results": [
    {"op": "compose", "ok": true},
    {"op": "wait", "ok": true, "settled": true}
  ],
  "frame_sha1": "7fb9cafdec473e0df1dfc424887afe59c2349b44",
  "video_offset_s": 10.29,
  "browser_ms": 1354,
  "at": "2026-10-06T06:40:09+00:00"
}

When a conversation ends in debug mode, one more line names its video: {"session": "9d0efd23", "video": ".../video/9d0efd23.webm", "at": "..."}.

Three more files are written when your runner asks for them, with the helpers in okareo_vision.diagnostics and a RunDir passed to the Target: setup(run_dir) logs to run.log; preflight(run_dir, okareo_base=..., app_url=...) writes preflight.json (Python, Playwright, proxy settings, and whether Okareo and the page can be reached); report(run_dir, run_link=...) writes report.txt, a per-turn summary with no credentials in it. The three are Diagnostics(run_dir)'s setup(), preflight(okareo_base=..., app_url=...) and report(run_link=...), the same calls.

The record holds whatever the page showed and sent: the screenshots, the HTML and the video show all of it. The HAR is recorded in Playwright's minimal mode (no cookies, headers or security details) unless OKAREO_WEB_TARGET_HAR=full. The video is written when its conversation ends.

The server it needs

okareo-vision speaks version 1 of Okareo's web simulation protocol, and says so when the Target registers. Okareo's cloud (https://api.okareo.com) speaks it, and a local Okareo server works the same; the example talks to http://localhost:8000 unless OKAREO_BASE_PATH says otherwise. The oldest Okareo SDK it works with is 0.0.151, the floor of the okareo it installs.

The traffic goes two ways. To Okareo: HTTPS to the API, and NATS over a WebSocket to wss://connect.ngs.global:443, which every Run needs, a local server included, so the machine needs that outbound connection. To the page: the browser, directly. Chromium does not read HTTPS_PROXY, so a page that must be reached through a proxy needs it as a browser flag, browser_args=["--proxy-server=http://proxy.example.com:8080"].

When it goes wrong

The page never shows a chat

• Watch it: headless=False (the example's --headed), or --debug and open shots/turn00-*.jpg, the first frame Okareo saw.
• prepare is where the chat gets opened. End it by waiting for the message box, as the example does, so the first frame has the chat in it; when prepare raises, the error is on the row.
• A chat in a panel over a busy page: screen_selector shows Okareo the panel alone.
• A chat in an iframe needs nothing set: typing, sending, pressing and the visible text reach into the frame. One served from another site entirely does not list its controls yet; press it at a point.
• A Flutter app needs nothing set: the library turns on its accessibility layer in every page and frame, so its controls are listed.
• When Okareo cannot find the message box: composer_selector and send_selector name it.

The site refuses headless Chrome

The browser already presents itself as a normal Chrome: the automation marker (navigator.webdriver) is off and the user agent has no "HeadlessChrome" in it. When the site still refuses, show the browser and use the real Chrome: headless=False and channel="chrome" (or OKAREO_WEB_BROWSER=chrome; run playwright install chrome if it is not on the machine). Sites with a bot gate commonly admit only the real Chrome, headed. A headed browser needs a display; on a Linux server, run under xvfb-run.

Sign-in needs MFA

A sign-in whose code can be scripted (a test account's fixed code, say) belongs in sign_in, above: once per Run, held in memory. One that needs a person cannot be scripted. Sign in by hand once, save the session, and start every conversation from it:

from okareo_vision import VisionTarget
from okareo_vision.auth import ensure_session

with ensure_session("https://portal.example.com/", storage_state="portal_session.json",
                    login_timeout=300) as (session, page):
    print("signed in at", session["landed_url"])

class Portal(VisionTarget):
    url = "https://portal.example.com/"
    storage_state = "portal_session.json"

With no recipe, ensure_session opens a visible browser at the url and waits up to login_timeout seconds (600 unless you say) for you to sign in, MFA included, then saves the session to storage_state with file mode 0600. Run it before each Run: it reuses the saved session and asks you to sign in again only when it no longer works (verify=, a function of (page, api_base) returning a bool, says what "works" means for your site). The file is a signed-in session; keep it out of version control. form_login(...) and entra_login(...) in the same module are scripted sign-ins for sites without MFA.

What is not in the package

• A Driver. Yours, registered with the Okareo SDK.
• Model calls. This process holds no model key; Okareo reads the screens and picks the moves.
• Scenarios and Checks. Made with the Okareo SDK, as in the example.
• The example and test pages. They are in the repository, not in the wheel.
• Anything about a particular site. Its url, sign-in and selectors belong in your subclass.

Documentation: https://docs.okareo.com/simulation/web-ui. Changes: CHANGELOG.md. License: Apache-2.0.