pytest-triage

Structured failure triage for pytest. It collects a machine-readable report of every failed test and — optionally — enriches each failure with an LLM verdict (regression / flaky / environment / test bug), so an on-call engineer gets a hypothesis and a ready-to-run rerun command instead of a wall of tracebacks.

Built to feed airflow-pytest-operator: a nightly DAG that fails at 3 a.m. can alert with a diagnosis instead of two thousand lines of traceback.

Package

Badge	What it tells you
	Latest release on PyPI — pip install pytest-triage
	Supported Python versions (3.10+)
	A pytest plugin — requires pytest 7.0+
	Distributed under the Apache-2.0 licence

Quality & build

Badge	What it tells you
	Lint, types and the unit matrix (Python 3.10–3.13) on main
	Test coverage of the package
	Fully type-checked with mypy --strict
	Linted & formatted with Ruff
	OpenSSF supply-chain security score

AI providers — bring your own key; each is an optional extra:

Everything is opt-in. Installing pytest-triage changes the behaviour of zero existing suites. Nothing runs, no report is written, and no network is touched until you pass an explicit --ai-* flag. See CHANGELOG.md.

Table of contents

• Why
• Install
• Quick start
• The report
• Configuration
• Providers
  • Anthropic
  • OpenAI
    • OpenAI-compatible endpoints
  • GigaChat (for the RU segment)
• What a run costs
• Write your own provider
• How it works
• Safety: the four invariants
• Using it from Airflow
• Development
• Scope of 0.1.x
• License

Why

A failed test in CI or a nightly DAG gives you a traceback. Somebody still has to read it and answer the only question that matters: is this my fault, the test's fault, or the environment's fault?

pytest-triage turns that judgement into structured data:

• a machine-readable JSON report of every failure — stable, versioned schema, with a ready-to-run rerun command;
• an optional LLM verdict per failure — one of regression, flaky, env, test_bug, or unknown, plus a one-line hypothesis and a suggested fix;
• secret redaction before anything leaves the process, so tracebacks with tokens or passwords don't end up in an artifact or a prompt.

It is deliberately narrow. It does not generate tests and it is not an LLM-evaluation framework — those niches are already well served.

Install

pip install pytest-triage

The core package depends only on pytest. LLM providers are optional extras:

pip install "pytest-triage[anthropic]"   # adds the Anthropic provider

pip install "pytest-triage[openai]"      # adds the OpenAI provider

pip install "pytest-triage[gigachat]"    # adds the GigaChat provider (RU)

Quick start

Everything is off until you ask for it. Three levels of opt-in:

1. Just the report — no AI, no network:

pytest --ai-report=failures.json

Runs your suite exactly as before, and additionally writes failures.json describing each failure. Every verdict is null — no provider is involved.

2. Add an LLM verdict per failure:

pip install "pytest-triage[anthropic]"
export ANTHROPIC_API_KEY=sk-ant-...
pytest --ai-triage=on --ai-provider=anthropic

Turning triage on writes the report to .triage.json automatically (override the path with --ai-report=PATH).

…or, for a Russian-language verdict from GigaChat:

pip install "pytest-triage[gigachat]"
export GIGACHAT_CREDENTIALS=...
pytest --ai-triage=on --ai-provider=gigachat

3. Try it with zero setup using the built-in deterministic fake:

pytest --ai-triage=on --ai-provider=fake

The fake provider hits no network and returns a deterministic verdict keyed off the exception type — ideal for wiring up your pipeline before committing to a real model. After any triaged run you also get a one-line terminal summary:

pytest-triage: 1 env, 2 test_bug

New here? Try it on failures that fail on purpose. examples/ is a small suite where every test breaks for a different, believable reason — a real server 500, a wrong assertion, a product bug, a DNS timeout — so you can see what each verdict looks like before pointing it at your own suite:

pip install requests            # the examples make real HTTP calls
pytest examples/ --ai-triage=on --ai-provider=fake --ai-report=.triage.json

Swap fake for anthropic or gigachat (with a key) to compare real verdicts.

The report

The report is a single JSON object with a versioned schema (schema_version) that only ever grows. One failure looks like this:

{
  "schema_version": 1,
  "created_at": "2026-07-23T15:03:30Z",
  "ai_model": "claude-sonnet-5",
  "triage_duration": 4.12,
  "pytest_args": [
    "tests/test_shop.py::test_checkout_total",
    "tests/test_shop.py::test_db_connection"
  ],
  "failures": [
    {
      "nodeid": "tests/test_shop.py::test_db_connection",
      "pytest_args": ["tests/test_shop.py::test_db_connection"],
      "phase": "call",
      "outcome": "failed",
      "exc_type": "ConnectionError",
      "exc_message": "could not connect to db at 10.0.0.5:5432",
      "traceback": "def test_db_connection():\n>       raise ConnectionError(...)\n...",
      "duration": 0.00004,
      "stdout_tail": "",
      "stderr_tail": "",
      "log_tail": "",
      "verdict": {
        "category": "env",
        "hypothesis": "ConnectionError in tests/test_shop.py::test_db_connection",
        "confidence": "low",
        "suggested_fix": null
      }
    }
  ],
  "total_failures": 2,
  "total_verdicts": 1
}

Key fields:

The report is written atomically (temp file + rename) and owner-only (0o600) — even after redaction it may hold residual sensitive output and must not be world-readable on a shared CI host. A write failure prints a warning and never changes the run's outcome.

Configuration

Every option has a CLI flag and a matching ini key. CLI overrides ini. All names are namespaced --ai- / ai_.

Triage runs only when --ai-triage=on and a provider is set — the two are deliberately separate, so you can configure ai_provider centrally in ini yet keep triage (and its cost) off until a specific run turns it on. A half-set combination (--ai-triage=on with no provider, or a provider with triage off) emits a config-time warning rather than silently doing nothing.

When triage is on, the report is written to .triage.json automatically — pass --ai-report=PATH to choose another path. Without triage, --ai-report=PATH still writes a report on its own, with every verdict set to null.

Set defaults for a project in pyproject.toml:

[tool.pytest.ini_options]
ai_triage = "on"
ai_provider = "anthropic"
ai_report = ".triage.json"
ai_budget = "20"

…or pytest.ini / tox.ini:

[pytest]
ai_triage = on
ai_provider = anthropic
ai_report = .triage.json

Triage runs only when --ai-triage=on and a provider is set. --ai-report works on its own (verdicts stay null).

Providers

A provider is the thin transport that turns a failure into a verdict. Two are built in and require no extra dependency:

Three more ship as optional extras: anthropic, openai, and gigachat.

Anthropic

The anthropic provider ships in the [anthropic] extra and talks to the Anthropic Messages API through the official anthropic SDK:

pip install "pytest-triage[anthropic]"
export ANTHROPIC_API_KEY=sk-ant-...
pytest --ai-triage=on --ai-provider=anthropic

It calls the API with strict tool use (record_verdict) so the model returns a structured verdict directly. Deployments or answers that don't fit are handled gracefully: a non-conforming response is parsed by the tolerant base parser, and anything unrecognisable becomes category="unknown" rather than an error. Retries are disabled so a rate-limited call fails fast instead of retrying past the wall-clock cap.

pytest-triage never touches your credentials. Everything below is read straight from the environment:

The model defaults to claude-sonnet-5. Override it with the provider's own env var — it is Anthropic-specific, so there is nothing shared to clash:

export ANTHROPIC_MODEL=claude-haiku-4-5

The anthropic SDK is imported lazily: if the extra isn't installed you get a clear configuration-time error — never a surprise ImportError mid-run.

OpenAI

The openai provider ships in the [openai] extra and talks to the OpenAI Chat Completions API through the official openai SDK:

pip install "pytest-triage[openai]"
export OPENAI_API_KEY=sk-...
pytest --ai-triage=on --ai-provider=openai

It forces a function call (record_verdict, strict schema) so the model returns a structured verdict directly. A non-conforming answer is parsed by the tolerant base parser, and anything unrecognisable becomes category="unknown" rather than an error. Retries are disabled so a rate-limited call fails fast instead of retrying past the wall-clock cap.

pytest-triage never touches your credentials. Everything below is read straight from the environment:

The model defaults to gpt-4o-mini (cheap and reliable for classifying a traceback). Override it with the provider's own env var:

export OPENAI_MODEL=gpt-4o

The openai SDK is imported lazily: if the extra isn't installed you get a clear configuration-time error — never a surprise ImportError mid-run.

OpenAI-compatible endpoints

The openai provider is not tied to OpenAI's own servers. A growing number of APIs speak the same protocol, so you can point it at any of them with OPENAI_BASE_URL and a model — no new provider, no new code:

# Kimi (Moonshot) — same provider, different endpoint
export OPENAI_API_KEY=<moonshot-key>
export OPENAI_BASE_URL=https://api.moonshot.ai/v1
export OPENAI_MODEL=kimi-k2-0711-preview
pytest --ai-triage=on --ai-provider=openai

For a local model the key can be any non-empty placeholder (OPENAI_API_KEY=ollama). Function-calling support varies by endpoint; where a model doesn't honour the forced call, the tolerant parser reads its plain-text answer and falls back to category="unknown" — the run is never affected.

Azure OpenAI is not a plain base-url swap — it needs the SDK's AzureOpenAI client (api-version, deployment names). Wrap it in a small custom provider (see Write your own provider) rather than OPENAI_BASE_URL.

GigaChat (for the RU segment)

The gigachat provider ships in the [gigachat] extra and talks to GigaChat through the official gigachat SDK:

pip install "pytest-triage[gigachat]"
export GIGACHAT_CREDENTIALS=...          # your GigaChat authorization key
pytest --ai-triage=on --ai-provider=gigachat

The prompt and the verdict text are Russian. GigaChat is a Russian model and reasons noticeably better in its own language, so hypothesis and suggested_fix come back in Russian:

{
  "category": "env",
  "hypothesis": "Тест не смог подключиться к базе: сервис недоступен в CI.",
  "confidence": "high",
  "suggested_fix": "Проверить, что контейнер postgres поднят до старта прогона."
}

category and confidence stay in English — they are the machine-readable part of the frozen Verdict contract, and downstream consumers match on them.

The provider forces a function call (record_verdict) so the model returns a structured verdict. Deployments that don't honour a forced call still work: a plain-text JSON answer is parsed by the tolerant base parser, and anything unrecognisable becomes category="unknown" rather than an error.

pytest-triage never touches your credentials. Everything below is read by the SDK straight from the environment:

The SDK reads a few more (GIGACHAT_FLAGS, GIGACHAT_MAX_CONNECTIONS) that triage has no use for — it sends one request at a time.

GIGACHAT_MAX_RETRIES has no effect — this provider pins it to 0, because retries inside the SDK only fight the wall-clock cap and leave abandoned requests billing; --ai-timeout, --ai-budget and the circuit breaker own resilience instead. GIGACHAT_RETRY_BACKOFF_FACTOR and GIGACHAT_RETRY_ON_STATUS_CODES are inert as a consequence, since the SDK gates both on max_retries > 0.

Dev contours. GIGACHAT_BASE_URL and GIGACHAT_AUTH_URL point the SDK at an internal gateway or a staging installation — nothing has to change in pytest-triage, which pins neither:

export GIGACHAT_BASE_URL=https://gigachat.dev.internal/api/v1
export GIGACHAT_AUTH_URL=https://ngw.dev.internal:9443/api/v2/oauth
export GIGACHAT_CA_BUNDLE_FILE="$PWD/certs/dev-ca.pem"   # keep verification on
pytest --ai-triage=on --ai-provider=gigachat

The environment is the documented path; GigaChatClient takes the same options as constructor arguments for callers that build the client themselves, and anything left unset falls through to GIGACHAT_* untouched.

Certificate trust. The root CA that signs the GigaChat endpoints does not ship in certifi, so out of the box you will see CERTIFICATE_VERIFY_FAILED, surfaced as a triage provider error (the run itself still passes or fails exactly as before). Fetch the official Russian Trusted Root CA into a git-ignored certs/ folder — verification stays on:

curl -fsSL -o certs/root.crt https://gu-st.ru/content/lending/russian_trusted_root_ca_pem.crt
curl -fsSL -o certs/sub.crt  https://gu-st.ru/content/lending/russian_trusted_sub_ca_pem.crt
awk 1 certs/root.crt certs/sub.crt > certs/russian_trusted_bundle.pem
export GIGACHAT_CA_BUNDLE_FILE="$PWD/certs/russian_trusted_bundle.pem"

A corporate contour (GIGACHAT_API_CORP/B2B) may also want a client certificate: GIGACHAT_CERT_FILE, GIGACHAT_KEY_FILE, and GIGACHAT_KEY_FILE_PASSWORD if that key is encrypted.

Disabling verification is possible via the SDK's own GIGACHAT_VERIFY_SSL_CERTS, but pytest-triage will never turn it off for you. On a dev contour with a self-signed certificate, prefer GIGACHAT_CA_BUNDLE_FILE with that contour's CA: GIGACHAT_VERIFY_SSL_CERTS=false sends your credentials over a connection nobody authenticated, and the failure context travels the same wire.

What a run costs

Triage is billed per call, so the plugin is built to keep the number of calls small and visible. Every triaged run prints what it actually spent:

pytest-triage: 1 env, 2 test_bug
pytest-triage: 2 provider call(s), 1 from cache

Four things bound the bill:

• --ai-budget=N (default 10) — a hard cap on provider calls per run. A suite with 300 failures still makes at most 10 calls; the rest come back as unknown with the reason triage budget exhausted.
• Caching — identical failures (normalized traceback + exception type) are analyzed once. One broken fixture that fails 40 tests is one call.
• A circuit breaker — the first timeout, or two consecutive provider errors, stops triage for the rest of the run. A provider that is down cannot burn the whole budget, and a hung one cannot turn budget × timeout into a five-minute suite.
• Bounded prompts — the context is truncated by bytes before it is sent (4 KB of traceback, 2 KB of each output tail, 1 KB of the exception message), and empty sections are dropped rather than sent as empty labels. A worst-case prompt is roughly 8 KB; a typical one is well under 2 KB.

So a default run costs at most 10 calls of ≤ ~8 KB prompt and ≤ 512–1024 response tokens each, and usually far less. Set --ai-budget=0 to keep the report while spending nothing.

Write your own provider

Most teams have a private model (an internal gateway, a self-hosted GigaChat, a local model). Implement ~80 lines of transport and you inherit prompt rendering, tolerant JSON parsing, budget, caching, timeout, and the circuit breaker for free.

# myco/testing/triage.py
from pytest_triage.providers import BaseTriageClient


class MyClient(BaseTriageClient):
    def __init__(self) -> None:
        self._session = ...  # your HTTP client / SDK

    def _request(self, prompt: str) -> str:
        # Send `prompt` to your model and return its raw text reply.
        # BaseTriageClient parses the JSON out of it tolerantly; an
        # unparseable reply becomes category="unknown", never an exception.
        return self._session.complete(prompt)

    def close(self) -> None:
        self._session.close()

If you write your own _render_prompt, use the two public helpers so your provider keeps the same cost and safety properties:

from pytest_triage.providers import redact_nodeid, render_sections


def _render_prompt(self, ctx):
    return render_sections(
        [
            # Parametrized ids embed argument values (test_login[sk-live-...]);
            # redact_nodeid scrubs the [...] part before it leaves the machine.
            ("nodeid: ", redact_nodeid(ctx.nodeid)),
            # render_sections drops empty sections — an empty stdout tail still
            # costs prompt tokens on every single call.
            ("traceback:\n", ctx.traceback),
            ("stdout tail:\n", ctx.stdout_tail),
        ]
    )

Register it either way (both are supported):

# Entry point — for teams that package their provider.
[project.entry-points."pytest_triage.providers"]
myco = "myco.testing.triage:MyClient"

# Import string — for teams without private package infrastructure.
pytest --ai-triage=on --ai-provider=myco.testing.triage:MyClient

Verify it against the public conformance kit:

from pytest_triage.testing import assert_conforms
from myco.testing.triage import MyClient


def test_my_provider_conforms() -> None:
    assert_conforms(MyClient())

The public contract lives at pytest_triage.providers (TriageClient, BaseTriageClient, Verdict, PROVIDER_API_VERSION, redact_nodeid, render_sections) and pytest_triage.context.FailureContext. Verdict is deliberately flat — weak models produce invalid JSON on nested schemas, so keep it flat.

How it works

      test fails
          │
          ▼
pytest_exception_interact ──►  FailureContext          (context.py)
  (controller only)            · truncate by bytes + marker
                               · redact secrets (strict)
          │
          ▼
pytest_sessionfinish  ──────►  triage each failure      (wrappers.py)
                               CachingClient
                                └─ CircuitBreakerClient
                                    └─ BudgetedClient
                                        └─ TimedOutClient
                                            └─ your provider
          │                    every call fenced: raise/timeout → "unknown"
          ▼
pytest_triage_report  ──────►  write .triage.json        (report.py)
pytest_terminal_summary ────►  "pytest-triage: 1 env, 2 test_bug"
                               "pytest-triage: 2 provider call(s), 1 from cache"

• Collection happens in pytest_exception_interact, where both the report and the live exception are available (pytest_runtest_logreport fires too early to see the exception type).
• Triage is composed in a single factory, build_triage_client. Cross-cutting concerns are decorators over the provider, never logic inside it: CachingClient (dedupes identical failures) wraps CircuitBreakerClient (stops after a timeout or two errors in a row) wraps BudgetedClient (caps provider calls) wraps TimedOutClient (hard wall-clock cap). Cache is outermost, so a cache hit costs neither budget nor time; the breaker sits above the budget, so a tripped breaker spends nothing at all.
• Every provider call is fenced. A provider that raises, hangs, or returns garbage yields a category="unknown" verdict — the exception never escapes. So does a failure whose context cannot be collected, and a downstream pytest_triage_report implementer that raises.
• Under xdist, triage and reporting run on the controller only; workers return from pytest_configure before a provider is ever built. Worker-failure aggregation is out of scope for 0.1.x, and a report run under -n warns and stays empty.

Safety: the four invariants

These are enforced by tests, not just documented:

1. AI never affects the test verdict. The plugin lives strictly in the reporting layer. A provider raising, timing out, or returning garbage leaves the run byte-identical to a run without the plugin — same exit code, same outcome, same collection.
2. Disabled by default. Installing the plugin changes the behaviour of zero existing suites. Everything is opt-in.
3. The report schema is versioned from day one and evolves additively only.
4. FailureContext and Verdict are frozen public contracts. New fields get defaults; nothing is renamed or removed.

Secret redaction (--ai-redact=strict, the default) scrubs JWTs and JWEs, URL-embedded credentials, PEM private keys, AWS access keys, Authorization headers (raw or inside an SDK error's header repr), bearer/basic tokens, secret-named assignments, and known-secret environment values before a failure is written or sent to a provider. It is best-effort and deliberately over-redacts; every pattern is linear (no ReDoS). Set --ai-redact=off only for local debugging.

Two things to know about what redaction does not cover:

• A test's nodeid stays verbatim in the report. It is the rerun selector, and pytest already prints it to the terminal and to JUnit XML, so scrubbing it in the artifact would buy nothing. The copy that goes to a provider is scrubbed: redact_nodeid cleans the [...] parametrization section, where argument values actually live.
• Model output is not a trusted channel. A verdict is capped in length and written as-is; treat hypothesis and suggested_fix as text a model wrote after reading your traceback, not as a fact.

Using it from Airflow

The intended downstream consumer is airflow-pytest-operator: a nightly DAG runs your suite with --ai-report=.triage.json, the operator reads the report and pushes the verdicts to XCom, and the alert carries a diagnosis and a rerun command instead of a raw traceback. Because the schema is versioned and the contracts are frozen, the operator can rely on the shape of the report across plugin upgrades.

Development

python -m pip install -e ".[dev]"

Run the suite, lint, and types the way CI does:

pytest                 # unit matrix; the `live` marker is excluded by default
ruff check . && ruff format --check .
mypy

Notes:

• Coverage must be measured with coverage run -m pytest, not pytest --cov. pytest-triage is a startup-loaded pytest11 plugin, so pytest-cov starts tracing too late and reports import-time lines as uncovered.
• The live marker hits a real provider endpoint and is excluded from the default run via addopts = -m "not live". Run it explicitly with an API key: pytest -m live.
• Behavioural tests use pytest's own pytester fixture and runpytest_subprocess() where exit-code fidelity matters.

Scope of 0.1.x

Intentionally not in this release: flaky-run detection, a persistent cache, git blame/context, an OpenAI provider, an MCP server, parallel triage, and xdist support beyond the controller guard.

License

Apache-2.0. See LICENSE and NOTICE.