testrelic-deepeval

The TestRelic Python SDK for DeepEval. Capture LLM evaluation results from your existing DeepEval test suite and ship them to your TestRelic workspace in one line of install.

pip install testrelic-deepeval
testrelic login
deepeval test run tests/

That's it. The pytest plugin captures DeepEval's in-memory TestRun at session finish and uploads cases + metrics to TestRelic's /api/v1/evals/* endpoints.

What this is (and isn't)

• Is: a pytest plugin + Python client for sending DeepEval results to TestRelic
• Is: a drop-in testrelic.evaluate() wrapper for programmatic eval runs
• Is: a CLI (testrelic login, testrelic test, testrelic view)
• Is not: a replacement for DeepEval (use DeepEval to author tests; we just receive the results)
• Is not: a Confident AI proxy — we use TestRelic's own API, not Confident's

Install

pip install testrelic-deepeval                  # core
pip install "testrelic-deepeval[deepeval]"      # also pull deepeval if not already installed
pip install "testrelic-deepeval[otel]"          # OTel tracing (Phase 4, preview)

Supported Python versions: 3.9, 3.10, 3.11, 3.12.

Authenticate

testrelic login --api-key tr_yourkey_here

Credentials are stored at ~/.testrelic/credentials.toml with 0600 permissions on POSIX. Or pass credentials via environment:

export TESTRELIC_API_KEY=tr_yourkey
export TESTRELIC_BASE_URL=https://platform.testrelic.ai/api/v1/evals  # only override if self-hosted

Precedence (highest to lowest): explicit configure() args → env vars → credentials file → built-in defaults.

Use with pytest

No code changes needed. After install + login:

deepeval test run tests/
# or, if you want the wrapper that double-checks credentials:
testrelic test tests/

Every test run uploads as a new eval run in TestRelic, annotated with the current branch, commit, and CI run URL (auto-detected for GitHub Actions, GitLab, Jenkins, CircleCI, and Buildkite).

Use programmatically

from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from testrelic import evaluate

results = evaluate(
    test_cases=[LLMTestCase(input="Hi", actual_output="Hello")],
    metrics=[AnswerRelevancyMetric(threshold=0.7)],
)
# results is whatever deepeval.evaluate() returns; upload is automatic

Datasets

from testrelic import datasets

# Pull a dataset version into a deepeval EvaluationDataset
ds = datasets.pull("customer-support-qa", label="latest")

# Push goldens up to a new version
datasets.push(
    alias="customer-support-qa",
    goldens=[{"input": "...", "expected_output": "..."}],
    label="v2",
    description="Refreshed Q1 2026 examples",
)

Migrating from Confident AI

See docs/migration-from-confident-ai.md. The TL;DR is:

1. pip install testrelic-deepeval
2. testrelic login
3. Remove CONFIDENT_API_KEY from CI
4. Run your existing tests unchanged

Or, in one shot: testrelic migrate-from-confident.

Offline / flaky networks

Failed uploads land in ~/.testrelic/queue/. Replay them with:

testrelic drain

The plugin never fails your test run because of an upload error — uploads run at pytest_sessionfinish(trylast=True) and swallow exceptions with a warning log.

CLI

Development

git clone https://github.com/testrelic-ai/testrelic-python-sdk
cd testrelic-python-sdk
pip install -e ".[dev]"
ruff check src tests
mypy src/testrelic
pytest

License

GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later). See LICENSE.

Note for integrators: AGPL-3.0 is a strong copyleft license. If you distribute or run a modified version of this SDK as part of a network service, the AGPL's network-use clause requires you to make the corresponding source available to users of that service. If you only use the SDK as a library to upload results to TestRelic, you are not distributing or modifying it and the AGPL terms do not encumber your application code.