testrelic-pytest 0.4.0

Generic pytest reporter that uploads test results to the TestRelic AI cloud platform.

testrelic-pytest

Generic pytest reporter that ships your test results to the TestRelic AI cloud platform. Works with any pytest project — unit tests, API tests, integration tests — and stays out of the way when no API key is configured.

Install

pip install testrelic-pytest

The plugin auto-registers via the pytest11 entry point. Nothing to import.

Configure

export TESTRELIC_API_KEY=tr_live_...
pytest

The plugin silently no-ops when TESTRELIC_API_KEY is unset, so it is safe to add to any project.

Options

All options can be set via CLI flag, environment variable, or a JSON file at .testrelic/testrelic-config.json. Precedence (highest first): CLI flag → environment → JSON file → defaults.

CLI flags use a --testrelic-pytest-* prefix so they don't collide with the testrelic-playwright plugin if both are installed.

CLI flag	Env var	Default	Purpose
--testrelic-pytest-disable	TESTRELIC_DISABLE=1	off	Force-disable the plugin.
--testrelic-pytest-api-key VALUE	TESTRELIC_API_KEY	—	API key from the TestRelic dashboard.
--testrelic-pytest-endpoint URL	TESTRELIC_CLOUD_ENDPOINT	https://platform.testrelic.ai/api/v1	Cloud base URL.
--testrelic-pytest-upload-strategy {batch,realtime,both,none}	TESTRELIC_UPLOAD_STRATEGY	batch	When to send results.
--testrelic-pytest-quiet	TESTRELIC_QUIET=1	off	Suppress banner output.
--testrelic-pytest-run-type {smoke,regression,nightly,ci}	TESTRELIC_RUN_TYPE	—	Dashboard bucket.
--testrelic-pytest-project-name NAME	TESTRELIC_PROJECT_NAME	git remote or pyproject name	Stable project identity.
--testrelic-pytest-artifact-threshold-kb N	—	32	Captured output beyond N KB is truncated inline.
--testrelic-pytest-output PATH	—	—	Optional JSON dump for debugging.

Config file (.testrelic/testrelic-config.json)

{
  "apiKey": "$TESTRELIC_API_KEY",
  "endpoint": "https://platform.testrelic.ai/api/v1",
  "upload": "batch",
  "projectName": "my-suite",
  "artifactThresholdKb": 32,
  "uploadArtifacts": true,
  "queueDirectory": ".testrelic/queue",
  "metadata": { "team": "platform" }
}

$ENV_VAR strings are expanded against the process environment at startup.

What gets captured

For each test the plugin captures:

• nodeId, file, className, testName, parametrizeId
• Outcome: passed | failed | skipped | xfailed | xpassed | error
• Duration (ms) and per-phase breakdown (setup/call/teardown)
• Markers (e.g. @pytest.mark.smoke), keywords, parametrize ids
• Skip reasons and pytest warnings
• Failure tracebacks and assertion repr
• Captured stdout / stderr / caplog (inline with redaction; truncated above --testrelic-artifact-threshold-kb, hard cap 1 MB per stream — artifact upload for large streams is planned for v0.2)

Test runs include CI metadata auto-detected from GitHub Actions, GitLab CI, Jenkins, and CircleCI, plus git branch/commit/author from the working tree.

pytest-xdist

Parallel runs via pytest-xdist (pytest -n 4) are supported out of the box. Each worker forwards its TestReport objects through the xdist channel; only the controller node uploads. The result is exactly one cloud run with all tests aggregated.

Offline mode

Any upload that fails after retries is written to ~/.testrelic/queue/ as a JSON file. Re-run later:

testrelic-pytest drain

The drainer re-authenticates with TESTRELIC_API_KEY and replays each entry to its original endpoint.

See also

• testrelic-deepeval — DeepEval / LLM evaluation bridge
• testrelic-playwright — Playwright reporter for pytest