pytest-llm-assert

Natural language assertions for pytest.

Testing a text-to-SQL agent? Validating LLM-generated code? Checking if error messages are helpful? Now you can:

def test_sql_agent_output(llm):
    sql = my_agent.generate("Get names of users over 21")
    
    assert llm(sql, "Is this a valid SQL query that selects user names filtered by age > 21?")

The LLM evaluates your criterion and returns pass/fail — no regex, no parsing, no exact string matching.

Features

Semantic assertions — Assert meaning, not exact strings
Multiple LLM providers — OpenAI, Azure, Anthropic, Gemini, Groq via Pydantic AI
pytest native — Works as a standard pytest plugin/fixture
Response introspection — Access tokens and reasoning via llm.response
Type-safe — Built with Pydantic for structured outputs

Installation

pip install pytest-llm-assert

Quick Start

# conftest.py
import pytest
from pytest_llm_assert import LLMAssert

@pytest.fixture
def llm():
    return LLMAssert(model="openai:gpt-4o-mini")

# test_my_agent.py
def test_generated_sql_is_correct(llm):
    sql = "SELECT name FROM users WHERE age > 21 ORDER BY name"
    assert llm(sql, "Is this a valid SELECT query that returns names of users over 21?")

def test_error_message_is_helpful(llm):
    error = "ValidationError: 'port' must be an integer, got 'abc'"
    assert llm(error, "Does this explain what went wrong and how to fix it?")

def test_summary_captures_key_points(llm):
    summary = generate_summary(document)
    assert llm(summary, "Does this mention the contract duration and parties involved?")

Setup

Works out of the box with cloud identity — no API keys to manage:

# Azure (Entra ID)
export AZURE_API_BASE=https://your-resource.openai.azure.com
az login

# Google Cloud (Vertex AI)
gcloud auth application-default login

# AWS (Bedrock)
aws configure  # Uses IAM credentials

Supports multiple providers via Pydantic AI — including API key auth for OpenAI, Anthropic, and more.

Documentation

Documentation — Full documentation with examples
Configuration — All providers, CLI options, environment variables
API Reference — Full API documentation
Comparing Judge Models — Evaluate which LLM works best for your assertions
Examples — Working pytest examples
Integration Tests — Running tests with real LLM APIs

pytest-aitest — Full framework for testing MCP servers, CLIs, and AI agents
Contributing — Development setup and guidelines

Requirements

Python 3.11+
pytest 9.0+
An LLM (OpenAI, Azure, Anthropic, etc.) or local Ollama

Security

Sensitive data: Test content is sent to LLM providers — consider data policies

License

MIT

Development Status
- 3 - Alpha
Framework
- Pytest
Intended Audience
- Developers
License
- OSI Approved :: MIT License
Programming Language

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pytest_llm_assert-0.2.4.tar.gz (243.3 kB view details)

Uploaded Mar 31, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

pytest_llm_assert-0.2.4-py3-none-any.whl (7.7 kB view details)

Uploaded Mar 31, 2026 Python 3

File details

Details for the file pytest_llm_assert-0.2.4.tar.gz.

File metadata

Download URL: pytest_llm_assert-0.2.4.tar.gz
Upload date: Mar 31, 2026
Size: 243.3 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.7

File hashes

Hashes for pytest_llm_assert-0.2.4.tar.gz
Algorithm	Hash digest
SHA256	`a1c306ffb03db5599dd93389f51ac48cedda83492d64b2d0173bd181c0a6a68a`
MD5	`4674f3b65f0df0e8e3e97011510e04b7`
BLAKE2b-256	`4a3b8ff3e098895008c751b5742eabd600f8b86e83649b7572768de05eebec8c`

See more details on using hashes here.

Provenance

The following attestation bundles were made for pytest_llm_assert-0.2.4.tar.gz:

Publisher: release.yml on sbroenne/pytest-llm-assert

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: pytest_llm_assert-0.2.4.tar.gz
- Subject digest: a1c306ffb03db5599dd93389f51ac48cedda83492d64b2d0173bd181c0a6a68a
- Sigstore transparency entry: 1202822457
- Sigstore integration time: Mar 31, 2026
Source repository:
- Permalink: sbroenne/pytest-llm-assert@c0d3f81a90ab0ba02d8278d5bfbbbef125e2fec3
- Branch / Tag: refs/heads/main
- Owner: https://github.com/sbroenne
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@c0d3f81a90ab0ba02d8278d5bfbbbef125e2fec3
- Trigger Event: workflow_dispatch