pytest-charisma

A pytest plugin that streams test results to the Charisma ingestion API as tests execute.

Results are batched and sent in a background thread so test execution is not blocked by network I/O. The plugin implements retry-once semantics and a circuit breaker (3 consecutive failures disables streaming for the rest of the session).

Installation

pip install pytest-charisma

No token, no index URL — works on any laptop and in any CI. The package is published to public PyPI; the Charisma repo itself stays private (only this client plugin wheel is public).

For local development from a checkout of the repo:

pip install -e packages/pytest-charisma

Publishing (maintainers)

Releases publish to public PyPI automatically from the merge-queue workflow using a PyPI API token.

Trusted publishing (OIDC) is not used because this repo runs on GitHub Enterprise Server, whose OIDC issuer PyPI does not trust. A stored API token is required instead.

One-time setup (done once by a PyPI project owner):

1. Sign in to PyPI with the team-owned account (2FA enabled).
2. Go to Account settings → API tokens → Add API token. Scope it to the pytest-charisma project once the project exists; for the very first publish use an account-scoped token, then re-scope it to the project afterward.
3. Copy the token (starts with pypi-).
4. In the GitHub Enterprise repo, add it as an Actions secret named PYPI_API_TOKEN (org-level secret recommended so both packages share it).
5. Bump version in pyproject.toml and merge — the workflow builds and publishes on the next release.

Rotate the token periodically and store it in the team secret manager.

Quick start

pytest \
  --charisma-url https://api.charisma.example.com \
  --charisma-token $CHARISMA_TOKEN \
  --charisma-project-alias my-project

The plugin activates automatically when charisma_url is configured. If the URL is absent, the plugin stays silent and does nothing.

Configuration

Options are resolved with priority: CLI flag > pytest.ini / pyproject.toml > environment variable.

Configuration via pyproject.toml

[tool.pytest.ini_options]
charisma_url = "https://api.charisma.example.com"
charisma_project_alias = "my-project"
charisma_batch_size = "10"
charisma_branch = "main"
# Token should come from env var for security:
# export CHARISMA_TOKEN=your-token

Configuration via pytest.ini

[pytest]
charisma_url = https://api.charisma.example.com
charisma_project_alias = my-project
charisma_batch_size = 10

Environment variables only (CI-friendly)

export CHARISMA_URL=https://api.charisma.example.com
export CHARISMA_TOKEN=your-api-token
export CHARISMA_PROJECT_ALIAS=my-project
export CHARISMA_BUILD_ID=$CI_BUILD_NUMBER
export CHARISMA_COMMIT_SHA=$CI_COMMIT_SHA
export CHARISMA_BRANCH=$CI_BRANCH
export CHARISMA_SOURCE_URL=$CI_BUILD_URL

pytest

How it works

1. Collection — After pytest collects tests, the plugin opens a launch session via POST /api/v1/launches.
2. Execution — As each test completes, results are buffered. When the buffer reaches batch_size, the batch is submitted to a background worker thread.
3. Flush — At session end, remaining results are flushed and the worker drains its queue (up to 30s timeout).
4. Resilience — Each batch gets one retry on failure. After 3 consecutive batch failures, the circuit breaker trips and streaming is disabled for the rest of the session. Test execution is never affected.

Outcome mapping

Requirements

• Python ≥ 3.10
• pytest ≥ 7.0, < 9.0
• httpx ≥ 0.24, < 1.0

Development

cd packages/pytest-charisma
pip install -e ".[dev]"
pytest

License

MIT