pytest-codex

Your AGENTS.md says “use uv.” Did Codex actually use it?

Turn agent instructions into pytest assertions. Check the commands Codex ran, the files it changed, and the MCP tools it called. Record a real run once; replay it in CI with zero model calls.

Illustration: a synthetic command trace, rendered from actual pytest failure output. No model call was used to make this image. A separate real recording is included and passes offline.

中文 · Full reference

0.3.0 adds fresh task regression comparisons, available from the GitHub source release. PyPI currently provides 0.2.0.

Start with one test

python -m pip install pytest-codex

Version 0.2.0 is available on PyPI, including version-checked replay.

Point pytest at the small project you want Codex to work on:

# pyproject.toml
[tool.pytest.ini_options]
codex_project = "sample_repo"

# tests/test_conventions.py
def test_agent_follows_project_rules(codex):
    run = codex.exec("Add requests as a project dependency.")
    assert run.ran("uv add", ok=True), run
    assert not run.ran("pip install"), run
    assert run.modified("pyproject.toml"), run
    assert not run.modified("legacy/*"), run

# Record once — needs Codex CLI + login and uses model quota.
python -m pytest tests/test_conventions.py --codex-record
# Replay in CI — no Codex binary, credentials, or model calls.
python -m pytest tests/test_conventions.py

Each call gets a fresh project copy. Shell edits count as file changes too. Missing or stale recordings fail with a re-record instruction; the default mode never falls back to a paid call.

Check reliability across attempts

import pytest

@pytest.mark.codex(runs=5, pass_rate=0.8)
def test_dependency_convention(codex):
    run = codex.exec("Add requests as a dependency.")
    assert run.ran("uv add", ok=True), run

Record five attempts, require four to pass, and save each one separately. Replay checks those saved observations; re-record to measure a new model or CLI version.

Why a pytest plugin?

• A linter checks instruction text. This checks the commands and file changes from an execution, including edits made through shell tools.
• A one-off agent script needs its own test machinery. This fits pytest fixtures, parametrization, failure reports, and your existing CI.
• Calling a model on every PR spends quota. Checked-in recordings let ordinary PR checks replay without model calls. Use explicit live runs for fresh evidence.

Try the included recording

git clone https://github.com/Czj027/pytest-codex.git
cd pytest-codex
python -m pip install .
python -m pytest examples --codex-bin=not-installed -q

This is a real Windows recording: successful uv add, changed dependency declarations, and untouched legacy code. The deliberately missing CLI proves the example can replay without launching Codex.

The included recordings also cover our own project rules with two models and two attempts each, a local MCP call, and a Skill's file output. These are small smoke checks, not estimates of general model reliability. See recording evidence.

Catch stale recordings

Pin both the model and the CLI version for portable offline replay:

[tool.pytest.ini_options]
codex_model = "your-model-id"
codex_cli_version = "0.163.0"

Use --codex-check-cli locally to compare with the installed codex --version. Version or input mismatches fail before applying recorded file changes; strict checks require an explicit model. Default replay stays offline and needs no CLI. An unchanged model ID cannot detect a provider updating that model; re-record when checking current behavior.

The daily latest-CLI check uses a local canned response endpoint without login or model calls. For fresh behavior checks, use the opt-in live CI template.

Learn more

run.succeeded means the CLI turn completed; use command status and file assertions to check the work. Recordings may contain sensitive output, so review them before committing. MIT licensed; independent community project.