Testorim for Python

Testorim is an AI QA service: describe a web test in plain English, or pick a saved one, and Testorim runs it in a real browser and returns a verdict with the evidence. This package runs Testorim tests from Python code and from pytest.

Install

pip install testorim

Python 3.9 or newer. The package has no dependencies.

Quick start

Create an API key in Testorim under Settings, API keys, then:

export TESTORIM_API_KEY=tst_live_...

from testorim import Testorim

client = Testorim()  # reads TESTORIM_API_KEY

run = client.run_test(
    "shop.example.com",  # a project's name, address or id
    "Add the Blue Top to the cart, open the cart and check that it shows 1 item.",
)
print(run.summary())
run.assert_passed()  # raises TestorimRunFailed unless the verdict is passed

run_test starts the run and waits for the verdict, polling every 3 seconds for up to 600 seconds. Name the buttons, fields and text as they appear on the page.

More of the client:

client.projects()                                   # [Project(id, name, base_url), ...]
client.create_project("https://shop.example.com")   # returns the existing project if the address is already tested
client.tests("Shop")                                # the project's saved tests

# Replay a saved test: by name within a project, or by id
run = client.run_saved("Checkout", project="Shop")
run = client.run_saved("5f0c...")

# A negative test passes when the app refuses what the description tries
client.run_test("Shop", "Sign in with a wrong password and check that an error is shown.", expect_failure=True)

# Run against another address, for example a pull request's preview
client.run_test("Shop", "...", base_url="https://pr-42.preview.shop.example.com")

# Start without waiting, then wait, read or stop it
run = client.run_test("Shop", "...", wait=False)
run = client.wait_for(run.id, timeout=900, on_status=lambda r: print(r.status))
run = client.get_run(run.id)
client.cancel(run.id)

When the wait runs out, run_test, run_saved and wait_for return the run as it is (run.done is False) instead of raising. Pass raise_on_timeout=True to get TestorimTimeout instead.

The site has to be reachable from the internet: Testorim's browsers refuse localhost and private addresses. To test work in progress, run against a preview deployment or expose your dev server through a tunnel. Every run counts against your workspace's plan.

The run

summary() leaves the evidence links out, because they open without signing in and the summary often lands in CI logs.

pytest

Installing the package adds a pytest plugin:

import pytest

@pytest.mark.testorim
def test_checkout(testorim):
    testorim.run_saved("Checkout", project="Shop").assert_passed()

@pytest.mark.testorim
def test_sign_up(testorim):
    testorim.run_test(
        "Shop",
        "Sign up with a new email address and check that the dashboard says Welcome.",
    ).assert_passed()

• testorim fixture: a client for the session. Without TESTORIM_API_KEY the test fails with a message saying how to create a key; set TESTORIM_SKIP_WITHOUT_KEY=1 to skip it instead.
• @pytest.mark.testorim: select these tests with pytest -m testorim, or leave them out with pytest -m "not testorim".
• --testorim-base-url URL: every run made through the fixture opens this address instead of the project's own. A base_url passed to a single run still wins.

A failed run fails the test with its summary:

E   testorim.errors.TestorimRunFailed: Verdict: FAILED (1 passed, 1 failed, 1 skipped, 41s)
E   Run: https://app.testorim.com/runs/...
E   Failed step 2: click "Create account": The page showed Something went wrong [the app did not behave as described]

GitHub Actions: test each pull request's preview

name: Browser tests
on: pull_request

jobs:
  testorim:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install testorim pytest
      - name: Run the Testorim tests against the preview
        env:
          TESTORIM_API_KEY: ${{ secrets.TESTORIM_API_KEY }}
          TESTORIM_SKIP_WITHOUT_KEY: "1" # pull requests from forks get no secrets: skip there
        run: pytest -m testorim --testorim-base-url "https://pr-${{ github.event.number }}.preview.example.com"

Put your preview's real address in --testorim-base-url, and make sure the preview is deployed before this job runs. A saved test that types a saved password runs only on its project's own host or on one of the project's environments, so Testorim refuses to replay it on a preview at another host.

Verdicts and blame

Each failed step carries a blame:

unconfirmed is True when a step found text that differs from what was expected but cannot tell whether the page or the expected text is wrong.

assert_passed() raises for every verdict but passed, so a run that needs review stops CI for a person. To let it through, check run.failed yourself.

Errors

Every exception derives from TestorimError.

RefusedError.code is one of:

A 429 with no code is a rate limit; retry_after says how many seconds to wait. The client never retries a trigger by itself: a retried trigger is a second run.

Configuration

Links

• Documentation: https://docs.testorim.com
• Coding agents (Claude Code, Codex, Cursor, VS Code and others): the Testorim MCP server, npx -y @testorim/cli mcp, in @testorim/cli, which is also the command-line tool