aiexpect

Assertions for non-deterministic AI text. Drop into the tests you already have.

from aiexpect import expect

def test_refund_policy(bot):
    reply = bot.ask("What is your refund policy?")

    expect(reply).to_mean("you can return items within 30 days")   # semantic, no API key
    expect(reply).to_be_grounded_in(policy_doc)                    # no hallucination
    expect(reply).to_not_contain_pii().to_have_length(max=600)      # deterministic rules

Run pytest as usual. You get normal pass/fail plus a Trust Score and a self-contained HTML report:

================================= aiexpect =================================
Trust Score: 87/100
  Accuracy 92 · Groundedness 85 · Relevance 90 · Safety 100 · Consistency 80 · Format 75
  41/46 checks passed (89%)
  report: /your/project/aiexpect-report.html

→ Open the sample report

pip install aiexpect

Zero dependencies. Works offline out of the box.

Why

Chatbot and LLM output changes every run. assert reply == "..." is useless, and most eval frameworks want you to adopt a whole new platform. aiexpect is just an assertion library: it slots into pytest next to your existing tests, and the results roll up into metrics a non-ML person can read.

Three tiers, free first

Know the limits of Tier 2. Embeddings measure topical similarity, not truth: "sale items cannot be returned" scores 0.64 against "you can return within 30 days" (measured with all-MiniLM-L6-v2). Use to_contain/to_not_contain for must-have facts and to_be_grounded_in / to_be_consistent_with (Tier 3) when negation or contradiction matters.

aiexpect never proxies your traffic. You bring the key; you own the bill. Judge verdicts are cached on disk, so re-running an unchanged suite costs nothing. Is a free local model good enough? Measured answer in docs/judges.md: a 3B Ollama model got 43/44 probe verdicts right at ~2 s each.

Configure a judge (only needed for Tier 3)

# free, local (llama3.2 is 2 GB and fits an 8 GB laptop; use llama3.1 with 16 GB+)
ollama pull llama3.2
export AIEXPECT_JUDGE=ollama:llama3.2

# or a cloud model
export ANTHROPIC_API_KEY=...                        # auto-detected; uses claude-opus-5 at low effort
export AIEXPECT_JUDGE=anthropic:claude-haiku-4-5    # cheaper
export AIEXPECT_JUDGE=openai:gpt-4o-mini
export AIEXPECT_JUDGE=openai-compatible:qwen2.5@http://localhost:8000/v1   # vLLM, LM Studio, Groq...

or in conftest.py:

import aiexpect
aiexpect.configure(judge="ollama:llama3.2", judge_threshold=0.7)

Flaky by nature? Measure it.

import aiexpect

@aiexpect.consistent(samples=5, min_pass_rate=0.8)
def test_greeting(bot):
    expect(bot.ask("hi")).to_have_tone("friendly")

Runs the body 5 times and passes on the pass-rate, not a single coin flip. Feeds the Consistency sub-score.

Semantic snapshots

def test_refund_policy(bot):
    expect(bot.ask("refund policy?")).to_match_snapshot()

First run stores the reply in __aisnapshots__/. Later runs compare by meaning, so rewording passes and a real change in what the bot says fails. Refresh with pytest --aiexpect-update-snapshots; forbid silent creation in CI with --aiexpect-snapshot-mode=strict.

Hallucination probe pack

from aiexpect import probes

@pytest.mark.parametrize("probe", probes.all(), ids=lambda p: p.id)
def test_hallucination_probe(bot, probe):
    probes.check(bot.ask(probe.question), probe)

22 curated questions that reliably expose fabrication: false premises ("Name the current King of France"), true-but-surprising premises ("Are sharks older than trees?") and plain facts. Keyword answer keys work offline; with a judge configured, paraphrased corrections are recognised too.

The report

pytest writes aiexpect-report.html (and .json) every run:

• Trust Score (0–100) = mean of six plain-English sub-scores: Accuracy, Groundedness, Relevance, Safety, Consistency, Format
• pass rate per assertion type, score distribution, per-test table, Trust Score trend across runs
• every check with the judge's reason, expandable, filterable (failed only / LLM-judged)
• single file, no CDN, light and dark mode, colour-blind-safe palette

CI gate: pytest --aiexpect-min-trust=80 fails the run when the Trust Score drops below 80.

GitHub Action: Trust Score as a PR comment

- run: pytest --aiexpect-json=aiexpect-report.json
- uses: dmsehgal/aiexpect@v0.3.1
  with:
    min-trust: 80          # optional gate
  # needs: permissions: { pull-requests: write }

Posts (and updates) one comment per PR with the Trust Score, sub-scores and failed checks, and writes the same table to the job summary.

CLI

aiexpect check "Return within 30 days" --contain "30 days" --no-pii --mean "30-day returns"
aiexpect report aiexpect-report.json -o report.html
aiexpect summary aiexpect-report.json --min-trust 80
aiexpect judge     # which judge would be used?

Testing a real web page (Playwright)

from aiexpect import expect

def test_search_snippet_is_relevant(page):          # `page` comes from pytest-playwright
    page.goto("https://en.wikipedia.org/w/index.php?fulltext=1&search=python+testing+framework")
    snippet = page.locator(".mw-search-result .searchresult").first.inner_text()

    expect(snippet).to_be_relevant_to("software testing framework")   # text changes; meaning shouldn't
    expect(snippet).to_contain_any("test", "testing").to_not_contain_pii()

pip install pytest-playwright && playwright install chromium, then pytest. Same idea for a chat widget: locate the bot's last message bubble and hand its text to expect(). Full example in examples/playwright.

Soft mode

e = expect(reply, soft=True).to_contain("30 days").to_not_contain_pii().to_be_json()
e.verify()   # raises once with every failure listed

How it compares

They are good tools with different centres of gravity. If you already run evals in one of them, keep doing so; aiexpect is for the tests next to your product code.

Roadmap

• TypeScript port with Jest/Vitest matchers, Playwright fixture, Cypress commands
• Judge agreement benchmark across more models (see docs/judges.md for the first result)
• More probe packs (multi-turn contradiction, instruction following)

Contributing

git clone https://github.com/dmsehgal/aiexpect && cd aiexpect
uv venv && uv pip install -e ".[dev]" && pytest

The test suite is fully offline (fake judge, lexical embeddings). Adapters for other frameworks and new probe packs are the easiest first contributions — see CONTRIBUTING.md.

MIT © Deep Sehgal