tamash-playwright

tamash-playwright is a plug and play self-healing and reporting solution for Playwright + pytest. Install it, add your AI API key details, and wire in one fixture override.

That's it. No changes needed to your actual test functions if you're following standard Playwright/pytest best practices.

Why you need this

Websites change often. A button gets renamed or moved, and your test can't find it anymore — even though the app still works fine for real users. Normally, that just means a broken test.

tamash-playwright fixes this automatically. When a test action can't find an element, it reads an AI-optimized accessibility snapshot of the current page and asks an AI model to point at the element directly, then works out the most durable way to describe it for next time — preferring a real identity (test id, ARIA role + name, label, placeholder) over anything positional or fragile. If it succeeds, your test keeps going. If not, it fails normally, just like before.

Once a fix has been proven to work, tamash-playwright apply-heals can rewrite it straight into your source — so the next run doesn't need AI at all for that line.

On top of that, it gives you a step-by-step HTML report of everything your tests actually did — every action, assertion, network call, and fixture, in order, with timing and full detail on anything that got healed or failed — something Playwright's own pytest-playwright plugin doesn't provide at all (confirmed not planned).

Here are the detailed steps to use this package.

Step 1: Install it

pip install tamash-playwright

This pulls in pytest-playwright as a dependency, so if you're starting fresh you'll also need the Playwright browsers:

playwright install

Some providers need one extra install:

# Anthropic (Claude) via API key
pip install "tamash-playwright[anthropic]"

# Claude subscription — no API key, uses your Claude Code / Claude subscription login instead
pip install "tamash-playwright[claude-subscription]"

# GitHub Copilot subscription — no API key, uses your Copilot CLI login instead (needs Python 3.11+)
pip install "tamash-playwright[copilot-subscription]"

Step 2: Connect an AI model

tamash-playwright needs an AI model to decide where a broken element actually went. You have two kinds of provider to choose from:

• API key — Ollama, OpenAI, Anthropic (Claude), or Google Gemini. Pay-per-token, set the key directly.
• Subscription — an existing Claude Code / Claude subscription, or an existing GitHub Copilot subscription. No API key at all: auth comes from a CLI login you already have (or a token env var for CI).

Create a file named .env in your project folder:

# Master on/off switch. Leave this as true, or remove the line entirely.
HEALER_ENABLED=true

# Pick one: ollama | openai | anthropic | gemini | claude-subscription | copilot-subscription
HEALER_PROVIDER=ollama

# --- Ollama Cloud (https://ollama.com) ---
OLLAMA_MODEL=gpt-oss:120b
OLLAMA_API_KEY=

# --- OpenAI ---
# OPENAI_MODEL=gpt-4.1-mini
# OPENAI_API_KEY=

# --- Anthropic (Claude) ---
# ANTHROPIC_MODEL=claude-haiku-4-5
# ANTHROPIC_API_KEY=

# --- Google Gemini ---
# GEMINI_MODEL=
# GEMINI_API_KEY=

# --- Claude subscription (requires: pip install tamash-playwright[claude-subscription]) ---
# No API key — authenticates via an existing `claude login` session, or CLAUDE_CODE_OAUTH_TOKEN
# for CI (see `claude setup-token`).
# CLAUDE_SUBSCRIPTION_MODEL=haiku

# --- GitHub Copilot subscription (requires: pip install tamash-playwright[copilot-subscription]) ---
# No API key — authenticates via an existing `copilot` CLI login, or GITHUB_TOKEN for CI.
# COPILOT_SUBSCRIPTION_MODEL=mai-code-1-flash-picker

Just fill in the API key and model for whichever one you want to use, and leave the rest as-is (or delete them). For a subscription provider, there's no key to fill in at all — if you can already run claude or copilot from your terminal, you're already authenticated.

Getting a free Ollama key (fastest way to get started)

Ollama Cloud is a quick, free way to get an API key without signing up for OpenAI/Anthropic/Gemini billing.

1. Go to ollama.com and create an account.
2. Once signed in, go to ollama.com/settings/keys.
3. Create a new API key and copy it.
4. Paste it into your .env file:

HEALER_ENABLED=true
HEALER_PROVIDER=ollama
OLLAMA_MODEL=gpt-oss:120b
OLLAMA_API_KEY=paste_your_key_here

That's all you need — no other variables required.

Step 3: Wire it in

pytest's plugin model means the reliable way to activate self-healing is one line in your project's conftest.py, added once — not per test file:

# conftest.py
from tamash_playwright.plugin import page  # noqa: F401

Why this line, and not nothing at all: tamash-playwright registers itself as a pytest plugin automatically on install, and its page fixture may already override pytest-playwright's own page fixture depending on plugin load order — but that order isn't something pytest guarantees across environments. A conftest.py fixture, on the other hand, is always preferred by pytest over a same-named fixture from an installed plugin, so re-exporting it there is the one setup step that's guaranteed to work everywhere, every time.

With that line in place, every test using the page fixture — no matter how many test files you have — automatically gets self-healing and reporting. Nothing else changes:

from tamash_playwright import expect

def test_login(page):
    page.goto("/")
    page.get_by_placeholder("Username").fill("Admin")  # healed automatically if this breaks
    page.get_by_role("button", name="Login").click()
    expect(page.get_by_role("heading", name="Dashboard")).to_be_visible()  # recorded in the report too

Any other page your test opens itself — a popup, a manually opened tab, a target="_blank" link — is automatically healing/reporting-aware too, with no extra code. So is context, if your test uses it directly (context.new_page(), context.route(), etc.).

Step 4: Turn on the report

Add one flag when you run pytest:

pytest --tamash-report=report.html

Open report.html and you'll see, for every test: duration, a pass/healed/failed badge, and every step it took — in order, with timing. Steps are split into three categories (filterable in the report itself):

• action — clicks, fills, navigation, drag-and-drop, mouse/keyboard input, network requests, everything a test does
• assert — every expect(...) check
• fixture — fixture/hook setup and teardown, including any custom fixture named directly in a test's own signature

Anything that got healed shows which AI provider recovered it, what it recovered to, and token usage. A heal that anchored on nearby text rather than a stable identity of its own is flagged needs review — not a failure, just worth a quick human look before being trusted long-term (see "How self-healing finds a replacement" below). Anything that failed outright shows the real Playwright error message and a screenshot at the moment of failure.

This works alongside pytest-html (pip install pytest-html, then add --html=report.html) rather than replacing it — the two are complementary, not overlapping. tamash-report.html shows what Playwright did and whether it healed. pytest-html's report shows why an assertion failed (the exact line and value diff), which matters for the checks below that can't go through expect() at all — see "API testing" and "What gets reported" for why. Running both is recommended:

pytest --html=report.html --self-contained-html --tamash-report=tamash-report.html

Step 5: Check your setup

Run the built-in doctor command to confirm everything's wired up correctly:

tamash-playwright doctor

It checks:

1. AI connectivity — confirms HEALER_ENABLED/HEALER_PROVIDER are set correctly and actually calls your configured provider to make sure it's reachable (API key + model, or subscription login). Also reports whether the configured model looks vision-capable — see "How self-healing finds a replacement" below for what that unlocks.
2. A sane action timeout — without one, a broken locator can retry silently for a long time before ever raising, leaving self-healing no time to run at all. Checks conftest.py / pytest.ini / pyproject.toml / setup.cfg / tox.ini for a default timeout and recommends one if it can't find it.
3. Missing .describe() labels — scans your test files (tests/ by default, or pass --dir <path>) for locators that don't have a .describe('...') label, flagging the ones most worth fixing (raw CSS/XPath selectors first).
4. Locators written directly in test files — flags any locator defined inline in a test rather than inside a Page Object class, a Playwright best practice regardless of self-healing.

If it finds issues, the fastest fix is to open the project in an AI coding assistant (Claude Code, Cursor, GitHub Copilot, etc.) and ask it to address what it flagged. You can also add a standing rule to that assistant's instructions/skill file (e.g. CLAUDE.md, .cursor/rules, .github/copilot-instructions.md) so it follows both practices automatically on any new test code going forward.

A quick tip for better results

If you're using plain CSS selectors (like page.locator('input[name="username"]')) rather than Playwright's more descriptive locators (get_by_role, get_by_placeholder, etc.), it helps to add a short, human-readable label so the healer knows what it's actually looking for. Chain .describe('...') right onto the locator:

def test_login_using_css_selectors(page):
    page.goto("https://example.com/auth/login")

    txt_username = page.locator('input[name="username"]').describe("User Name Textbox")
    txt_username.fill("testadmin")

    txt_password = page.locator('input[placeholder="Password"]').describe("Password Textbox")
    txt_password.fill("secret")

    btn_login = page.locator('button[type="submit"]').describe("Login Button")
    btn_login.click()

This step is optional, but recommended. If you skip it, the healer doesn't just guess blind — it falls back to reading the variable name your locator was assigned to (txt_username, btn_login, ...) straight from your own source line, which is usually still a decent signal. An explicit .describe() is simply the most reliable option, and the only one that survives a variable being renamed or the locator being reassigned.

Writing tests

Both of the patterns below work exactly as they would with plain Playwright — the page fixture is the only thing that changed (see Step 3), so nothing here is tamash-playwright-specific syntax.

A normal test (no Page Object Model)

from tamash_playwright import expect

def test_login(page):
    page.goto("https://example.com/login")

    page.get_by_placeholder("Username").fill("testadmin")
    page.get_by_placeholder("Password").fill("secret")
    page.get_by_role("button", name="Login").click()

    expect(page.get_by_role("heading", name="Dashboard")).to_be_visible()

A Page Object Model test

Page Object classes just take page in their constructor like normal — .describe() on each locator is optional but recommended (see "A quick tip for better results" above):

# pages/login_page.py
class LoginPage:
    def __init__(self, page):
        self.page = page
        self.txt_username = page.get_by_placeholder("Username").describe("Username Textbox")
        self.txt_password = page.get_by_placeholder("Password").describe("Password Textbox")
        self.btn_login = page.get_by_role("button", name="Login").describe("Login Button")

    def login(self, username, password):
        self.txt_username.fill(username)
        self.txt_password.fill(password)
        self.btn_login.click()

# pages/dashboard_page.py
from tamash_playwright import expect

class DashboardPage:
    def __init__(self, page):
        self.page = page
        self.header = page.get_by_role("heading", name="Dashboard").describe("Dashboard Header")

    def verify_loaded(self):
        expect(self.header).to_be_visible()

# tests/test_login.py
from pages.dashboard_page import DashboardPage
from pages.login_page import LoginPage

def test_login_with_pom(page):
    page.goto("https://example.com/login")

    login_page = LoginPage(page)
    login_page.login("testadmin", "secret")

    dashboard_page = DashboardPage(page)
    dashboard_page.verify_loaded()

Healing and reporting apply the same way regardless of which style you use — the page object passed into a Page Object's constructor is the same wrapped page your test received, so every locator built from it is tracked and healable whether it's created directly in the test function or inside a Page Object method.

A "base test" pattern (your own fixtures on top of page)

A common next step beyond plain Page Objects is a project's own conftest.py fixtures that hand tests a ready-to-use Page Object instead of constructing one inline every time:

# conftest.py
import pytest
from pages.dashboard_page import DashboardPage
from pages.login_page import LoginPage

@pytest.fixture
def login_page(page):
    return LoginPage(page)

@pytest.fixture
def dashboard_page(page):
    return DashboardPage(page)

# tests/test_login.py
def test_login_with_base_fixtures(page, login_page, dashboard_page):
    page.goto("https://example.com/login")
    login_page.login("testadmin", "secret")
    dashboard_page.verify_loaded()

This needs no tamash-playwright-specific code at all — it's the same fixture-composition pattern you'd write against plain Playwright. The reason it's worth calling out explicitly: login_page/dashboard_page both depend on page, and by Step 3 page is already the healing/reporting-wrapped one — so any fixture layer you build on top of it inherits healing and reporting automatically, no matter how deep the composition goes. If your project already has fixtures like this before adopting tamash-playwright, you don't need to touch them at all — the one conftest.py line from Step 3 is the only change anywhere in your test suite.

API testing

For a browser-driven test that also makes an API call, page.request and context.request are already reporting-aware — no setup needed, same as page itself. (Reporting only, not healing — there's no locator involved in an HTTP call, so nothing for the AI to recover.)

For pure API testing (no browser at all), use the api_request_context fixture:

def test_members_api(api_request_context):
    response = api_request_context.get(
        "https://example.com/api/members",
        headers={"Authorization": "Basic ..."},
    )
    assert response.status == 200

It's named api_request_context rather than request: pytest already has its own built-in request fixture (test/fixture introspection metadata, a completely different thing), and pytest-playwright doesn't ship a request-context fixture of its own to override the way it does for page. api_request_context gives you a standalone, wrapped APIRequestContext (playwright.request.new_context() under the hood) with no browser involved.

There's nothing to heal for API calls — no locator was ever involved — but every get/post/put/patch/delete/head/fetch call shows up in the report with its URL, duration, and status code.

If you build your own APIRequestContext some other way (e.g. via playwright.request.new_context() directly, outside this fixture), wrap it yourself with bind_api_request_context() to get the same tracking:

from tamash_playwright import bind_api_request_context

context = playwright.request.new_context()
bind_api_request_context(context)

expect() vs plain assert for API responses

Use tamash_playwright.expect(response).to_be_ok() when a check only cares "did this succeed" — it's real Playwright API (APIResponseAssertions has exactly to_be_ok()/not_to_be_ok()), so it shows up in the report like any other assertion. For anything more specific — an exact status code, a JSON body field, a header value — use a plain assert: Playwright's expect() only accepts Locator/Page/APIResponse objects, not plain Python values, so expect(response.status) or expect(some_string) raises ValueError: Unsupported type. That's not a gap in this package — it's a hard limit of Playwright's own Python expect() — and it's deliberately not "fixed" with a custom assertion helper, since that would force test code to deviate from standard Playwright/Python just to get report coverage. Plain-assert failures still show up in your report as an overall test failure; pair with pytest-html (see Step 4) if you want the exact failing line and value diff too.

How self-healing finds a replacement

When an action fails, tamash-playwright doesn't just hand the AI a blind "guess a CSS selector" task. It reads an AI-optimized accessibility snapshot of the page — every element tagged with its own id, real parent/child/sibling structure preserved — and asks the model to point at the element directly. That resolves to the exact node, no guessing about uniqueness.

Once the right element is found (and the original action has actually been retried against it successfully), a second step works out the most durable way to describe it for next time, in order of preference:

1. A real identity of its own — a test id, ARIA role + accessible name, label, placeholder, alt text, or title. This is what Playwright's own tooling already considers best practice.
2. Nearby text, when the element has none of the above — the same way a sighted person reads a label next to a field before clicking into it. This is flagged needs review in the report: it's a working fix, just inherently a bit less stable than a real identity, so it's worth a quick look rather than being silently trusted forever.
3. If neither works, the least-fragile selector Playwright itself could find — also flagged for review.

If text-based healing can't find a plausible match at all, and the configured model looks vision-capable (tamash-playwright doctor tells you), it falls back to looking at an actual screenshot of the page and locating the element visually — a genuine second attempt, not a first resort.

Once a selector's been confirmed to work for a specific line of your source code, it's remembered on disk (.tamash-playwright/heals.jsonl). The next time that exact line breaks the same way — this run or a future one — it's retried directly first, with no AI call and no token cost, before falling back to a fresh attempt if the cached fix has stopped working too.

What gets healed

Any Playwright action or read that resolves a selector and genuinely raises on failure is healed — verified empirically per method, not assumed, since some Playwright methods are designed to degrade gracefully instead of throwing (see below):

• Actions: click, dblclick, tap, hover, fill, clear, press, check, uncheck, select_option, set_input_files, focus, blur, type, press_sequentially, set_checked, scroll_into_view_if_needed, dispatch_event, select_text, wait_for (Locator.wait_for() specifically — a bad selector means it never reaches the state you're waiting for, same underlying problem as a bad click).
• Reads: text_content, inner_text, inner_html, get_attribute, input_value, is_checked, is_enabled, is_disabled, is_editable, bounding_box, element_handle, aria_snapshot, screenshot — these resolve a selector exactly like click does, and a successful heal returns the actual value instead of raising (e.g. a healed text_content() call still returns the real text, not None).

drag_to is tracked but never healed — it needs two locators (source + target), and guessing a replacement drop target is too risky to attempt safely.

Six read methods are deliberately not healable, because they're designed by Playwright to degrade gracefully rather than throw — confirmed directly (not assumed) by testing each one against a locator matching nothing: is_visible/is_hidden return False/True, count returns 0, all/all_inner_texts/all_text_contents return []. There's no exception for the healer to ever catch, so no matter how broken the locator is, these can't be healing candidates — they're still tracked in the report, just never as a heal attempt.

expect(...) assertions (to_have_text, to_be_visible, etc.) are not healed — they use Playwright's own built-in auto-retrying assertions, which are a separate mechanism this package doesn't touch. If a locator only ever appears inside an expect(...) call and never in an action, .describe() on it is a readability nicety, not something that affects healing.

Content inside an <iframe> (via frame_locator()) is healed the same way as anything else — the healer snapshots and rebuilds from inside the correct frame, not the outer page.

When healing doesn't fully recover

Not every heal attempt succeeds, and the report tells you exactly how far it got rather than collapsing everything into a generic "failed":

• Never triggered at all — HEALER_ENABLED=false, no AI provider configured (missing API key/model, or a subscription that isn't logged in), or the page's accessibility snapshot couldn't be captured. Nothing was ever sent to the AI.
• Triggered, but didn't recover — the AI call itself failed (network error, wrong API key — a real 401/403 fails the heal, not your test; the original Playwright error is still what your test fails with), the AI explicitly found nothing plausible in the snapshot (or, for a vision fallback, the screenshot), or — the most informative case — the AI did suggest a specific replacement and it was actually tried, but that failed too. That last case shows you the exact selector the AI guessed, not just that healing didn't work.

Every one of these shows up in both the console ([self-healer] ...) and tamash-report.html's failed-step detail, with a plain-language reason and, where relevant, the AI's actual suggestion and token usage — even on a failed attempt, since token spend on a wrong guess is still worth seeing.

Making a heal permanent: apply-heals

Self-healing fixes a broken selector for that run. To fix it in your actual source code — so future runs don't need AI at all for that line — run:

tamash-playwright apply-heals

This reads every eligible heal recorded during your last test run(s) and rewrites the exact broken locator call to the selector that was proven to work — nothing else on the line changes; a chained .describe('...') survives untouched. A few things to know:

• --dry-run shows exactly what would change, with a before/after diff, without touching any files.
• Only heals with a known source location are eligible — a vision-based heal (the AI located the element in a screenshot, not the accessibility tree) has no reusable source form, so it's skipped rather than guessed at.
• Applying writes two reports (JSON and Markdown) under .tamash-playwright/, plus a ready-to-run verification script (.tamash-playwright/verify_heals.py) that re-runs exactly the tests affected — with HEALER_ENABLED=false — so a pass proves the rewritten selector works completely on its own, not just "worked while healing was propping it up."
• A fix that came from anchoring on nearby text rather than a real identity (see "How self-healing finds a replacement") is marked needs review in the output — worth a quick look before committing, same flag as in the HTML report.
• --logs-dir <path> merges heals.jsonl files found recursively under <path> instead of the local .tamash-playwright/heals.jsonl — for a sharded CI setup where each shard uploads its own heal log as an artifact and a separate job applies them all at once.

What gets reported

Beyond the core click/fill/navigate actions and expect() assertions, all of the following show up in the report automatically, with no extra setup:

• Multi-page and popups — any second page a test opens (context.new_page(), expect_popup(), a target="_blank" link) is auto-wrapped the moment it appears, via the browser context's own "page" event.
• iframes — frame_locator() and anything resolved through it, including nested iframes.
• Mouse/keyboard — page.mouse.* and page.keyboard.* calls (reporting only, nothing to heal since there's no locator involved).
• Network interception — page.route()/context.route(), including whichever terminal action the handler takes (fulfill/abort/continue_/fallback).
• Dialogs — page.on("dialog", ...) and page.once("dialog", ...), including whichever terminal method the handler calls (accept/dismiss).
• Downloads — captured automatically via an always-on listener, whether your test uses expect_download() or its own on("download", ...) handler.
• JS execution — evaluate()/evaluate_handle() on Page, Frame, and Locator (reporting only — a JS error is almost always a bug in the expression itself, not a "couldn't find the element" problem, so it's not treated as a healing candidate).
• Explicit waits — wait_for_load_state, wait_for_url, wait_for_timeout, wait_for_event, wait_for_function on Page, Frame, and Locator (reporting only, no locator involved in what they're waiting for). Locator.wait_for() is different — see "What gets healed" above.
• Non-throwing read methods — is_visible, is_hidden, count, all, all_inner_texts, all_text_contents (reporting only — see "What gets healed" for why these specifically can't be healing candidates).
• API calls — see "API testing" above.

page.on("request"/"response", ...) and any other event name aren't special-cased — they pass straight through unaffected, so plain Playwright event-handling code keeps working exactly as written.

Known limitations

.first / .last don't get healing or reporting: they're Python properties, not method calls, so there's no way to patch the Locator they return without globally patching Playwright's own Locator class — which this package deliberately never does (everything it touches is scoped to objects it handed back to your own code). Use .nth(0) / .nth(-1) instead — same result, and a real method call that gets full healing and reporting.

claude-subscription on Windows needs a native Claude Code install, not the npm install -g @anthropic-ai/claude-code shim — the underlying SDK deliberately refuses to launch that shim on Windows (a real command-injection class it's guarding against, not overcaution). Install natively instead: irm https://claude.ai/install.ps1 | iex (PowerShell). macOS/Linux aren't affected.

License

Free to use, including commercially. The source code may not be copied, modified, redistributed, or resold without prior written permission. See the LICENSE file included in this package for the full terms.

Support

For questions or concerns, contact us at support@vibetestq.com.