pytest-flakiness 0.18.0

Pytest reporter for Flakiness.io

pytest-flakiness

The official Flakiness.io reporter for pytest.

[!TIP]

• Report demo is available at https://flakiness.io/flakiness/pytest-flakiness
• Supported features: features.md

Installation

Install using uv (recommended):

uv add --dev pytest-flakiness

Or via standard pip:

pip install pytest-flakiness

Usage

Once installed, simply run pytest. The reporter will automatically activate, aggregate test results, and create Flakiness Report in the flakiness-report directory.

pytest

The generated report can be viewed interactively via the Flakiness CLI Tool:

flakiness show

[!TIP] Make sure to add flakiness-report directory to your .gitignore

flakiness-report/

If Flakiness Access Token is passed, then the reporter will upload the report to Flakiness.io. You will see a confirmation in your terminal summary:

...
PASSED [100%]
==============================
✅ [Flakiness] Report uploaded: https://flakiness.io/your_org/your_proj/run/1
==============================

Uploading Reports to Flakiness.io

Github Actions

When running in GitHub Actions, the reporter can authenticate using GitHub's OIDC token — no access token needed.

For this to work:

1. The GitHub Actions workflow must have id-token: write permission.
2. The --flakiness-project option (or FLAKINESS_PROJECT env variable) must be set to your Flakiness.io project identifier (org/project).
3. The Flakiness.io project must be bound to the GitHub repository that runs the GitHub Actions workflow.

permissions:
  id-token: write

steps:
  - name: Run Tests
    run: pytest --flakiness-project="my-org/my-project"

You can also use the FLAKINESS_PROJECT environment variable instead of the CLI flag:

permissions:
  id-token: write

steps:
  - name: Run Tests
    env:
      FLAKINESS_PROJECT: my-org/my-project
    run: pytest

Access Token

Alternatively, you can authenticate using your project's Access Token. You can find this in your project settings on flakiness.io.

Set the Access Token using either an environment variable or command-line flag:

export FLAKINESS_ACCESS_TOKEN="flakiness-io-..."
pytest --flakiness-access-token="flakiness-io-..."

All Configuration Options

All options can be set via environment variables or command-line flags:

Flag	Environment Variable	Description
--flakiness-name	FLAKINESS_NAME	Name for this environment. Defaults to pytest
--flakiness-title	FLAKINESS_TITLE	Optional human-readable report title. Typically used to name a CI run, matrix shard, or other execution group
--flakiness-output-dir	FLAKINESS_OUTPUT_DIR	Local directory to save JSON report. Defaults to flakiness-report
--flakiness-project	FLAKINESS_PROJECT	Flakiness.io project identifier (e.g. org/project). Required for GitHub OIDC authentication
--flakiness-access-token	FLAKINESS_ACCESS_TOKEN	Your Flakiness.io access token for upload
--flakiness-endpoint	FLAKINESS_ENDPOINT	Flakiness.io service endpoint. Defaults to https://flakiness.io
--flakiness-disable-upload	FLAKINESS_DISABLE_UPLOAD	Disable uploading the report to Flakiness.io. The JSON report is still written to the output directory

Custom Environment Data

You can add custom metadata to your test runs using FK_ENV_* environment variables. These might be handy to capture properties that affect system-under-test.

export FK_ENV_GPU_TYPE="H100"
export FK_ENV_DEPLOYMENT="staging"

The FK_ENV_ prefix is removed and keys are lowercased, e.g. FK_ENV_DEPLOYMENT becomes deployment, and FK_ENV_GPU_TYPE becomses gpu_type.

Local Development

To save reports locally, pass --flakiness-output-dir:

pytest --flakiness-output-dir=./flakiness-reports

This will create a report.json file and an attachments/ directory in the specified folder.

CI/CD Example (GitHub Actions)

Using GitHub OIDC (recommended — no secrets needed):

permissions:
  id-token: write

steps:
  - name: Run Tests
    run: pytest --flakiness-project="my-org/my-project"

Alternatively, using an access token:

- name: Run Tests
  env:
    FLAKINESS_ACCESS_TOKEN: ${{ secrets.FLAKINESS_ACCESS_TOKEN }}
  run: pytest

Contributing

See CONTRIBUTING.md for development setup, running checks, and publishing new versions.

License

MIT