pytest-flakiness 1.1.0

Pytest reporter for Flakiness.io

pytest-flakiness

The official Flakiness.io reporter for pytest.

[!TIP]

• Report demo is available at https://flakiness.io/flakiness/pytest-flakiness
• Supported features: features.md

Installation

1. Install using uv (recommended):

   uv add --dev pytest-flakiness
   

   Or via standard pip:

   pip install pytest-flakiness
   

2. Set your Flakiness.io project identifier (org/project) in your pytest config:

   # pyproject.toml
   [tool.pytest]
   flakiness_project = "my-org/my-project"
   

   See Project Configuration for other config file formats, or use the --flakiness-project flag / FLAKINESS_PROJECT env variable instead.

Usage

Once installed, simply run pytest. The reporter will automatically activate, aggregate test results, and create Flakiness Report in the flakiness-report directory.

pytest

The generated report can be viewed interactively via the Flakiness CLI Tool:

flakiness show

[!TIP] Make sure to add flakiness-report directory to your .gitignore

flakiness-report/

If Flakiness Access Token is passed, then the reporter will upload the report to Flakiness.io. You will see a confirmation in your terminal summary:

...
PASSED [100%]
==============================
✅ [Flakiness] Report uploaded: https://flakiness.io/your_org/your_proj/run/1
==============================

Project Configuration

Options that are stable across a project are best set once in your pytest config instead of being passed on every run. With pytest 9, add them to the [tool.pytest] table in pyproject.toml:

# pyproject.toml
[tool.pytest]
flakiness_project = "my-org/my-project"

The same keys work in [pytest] of pytest.ini / tox.ini / setup.cfg. See All Configuration Options for the full list — CLI flags and FLAKINESS_* environment variables still override these per run.

Uploading Reports to Flakiness.io

Github Actions

When running in GitHub Actions, the reporter can authenticate using GitHub's OIDC token — no access token needed.

For this to work:

1. The GitHub Actions workflow must have id-token: write permission.
2. The --flakiness-project option (or FLAKINESS_PROJECT env variable) must be set to your Flakiness.io project identifier (org/project).
3. The Flakiness.io project must be bound to the GitHub repository that runs the GitHub Actions workflow.

permissions:
  id-token: write

steps:
  - name: Run Tests
    run: pytest --flakiness-project="my-org/my-project"

You can also use the FLAKINESS_PROJECT environment variable instead of the CLI flag:

permissions:
  id-token: write

steps:
  - name: Run Tests
    env:
      FLAKINESS_PROJECT: my-org/my-project
    run: pytest

Access Token

Alternatively, you can authenticate using your project's Access Token. You can find this in your project settings on flakiness.io.

Set the Access Token using either an environment variable or command-line flag:

export FLAKINESS_ACCESS_TOKEN="flakiness-io-..."
pytest --flakiness-access-token="flakiness-io-..."

All Configuration Options

Each option can be set via a command-line flag, an environment variable, or an ini option (see Ini File Configuration below). When the same option is set in more than one place, the highest-priority source wins:

CLI flag > environment variable > ini file > built-in default

Flag	Environment Variable	Ini Option	Description
--flakiness-name	FLAKINESS_NAME	flakiness_name	Name for this environment. Defaults to pytest
--flakiness-title	FLAKINESS_TITLE	flakiness_title	Optional human-readable report title. Typically used to name a CI run, matrix shard, or other execution group
--flakiness-output-dir	FLAKINESS_OUTPUT_DIR	flakiness_output_dir	Local directory to save JSON report. Defaults to flakiness-report
--flakiness-project	FLAKINESS_PROJECT	flakiness_project	Flakiness.io project identifier (e.g. org/project). Required for GitHub OIDC authentication
--flakiness-access-token	FLAKINESS_ACCESS_TOKEN	—	Your Flakiness.io access token for upload. Not available as an ini option, since ini files are usually committed to version control
--flakiness-endpoint	FLAKINESS_ENDPOINT	flakiness_endpoint	Flakiness.io service endpoint. Defaults to https://flakiness.io
--flakiness-disable-upload	FLAKINESS_DISABLE_UPLOAD	flakiness_disable_upload	Disable uploading the report to Flakiness.io. The JSON report is still written to the output directory
--flakiness-commit-id	FLAKINESS_COMMIT_ID	flakiness_commit_id	Commit ID of the repository under test. Defaults to the current git commit
--flakiness-git-root	FLAKINESS_GIT_ROOT	flakiness_git_root	Root directory used to normalize all paths. Defaults to the git repository root

Ini File Configuration

Options can also be set in your pytest configuration file, which is handy for values that are stable across a project (such as flakiness_project). All of the ini option names above are recognized in any file pytest reads — pytest.ini, tox.ini, setup.cfg, or pyproject.toml.

pytest.ini (or tox.ini / setup.cfg under [pytest]):

[pytest]
flakiness_project = my-org/my-project
flakiness_name = pytest
flakiness_disable_upload = false

pyproject.toml — pytest 9 reads native TOML types from the [tool.pytest] table (recommended):

[tool.pytest]
flakiness_project = "my-org/my-project"
flakiness_name = "pytest"
flakiness_disable_upload = false

If you target older pytest, or prefer the string-based ini format, use [tool.pytest.ini_options] instead (don't combine both tables — pytest errors if it sees them together):

[tool.pytest.ini_options]
flakiness_project = "my-org/my-project"
flakiness_name = "pytest"
flakiness_disable_upload = "false"

Note: The access token is intentionally not available as an ini option — keep secrets out of version-controlled config files and pass --flakiness-access-token or FLAKINESS_ACCESS_TOKEN instead.

Custom Environment Data

You can add custom metadata to your test runs using FK_ENV_* environment variables. These might be handy to capture properties that affect system-under-test.

export FK_ENV_GPU_TYPE="H100"
export FK_ENV_DEPLOYMENT="staging"

The FK_ENV_ prefix is removed and keys are lowercased, e.g. FK_ENV_DEPLOYMENT becomes deployment, and FK_ENV_GPU_TYPE becomses gpu_type.

Local Development

To save reports locally, pass --flakiness-output-dir:

pytest --flakiness-output-dir=./flakiness-reports

This will create a report.json file and an attachments/ directory in the specified folder.

CI/CD Example (GitHub Actions)

Using GitHub OIDC (recommended — no secrets needed):

permissions:
  id-token: write

steps:
  - name: Run Tests
    run: pytest --flakiness-project="my-org/my-project"

Alternatively, using an access token:

- name: Run Tests
  env:
    FLAKINESS_ACCESS_TOKEN: ${{ secrets.FLAKINESS_ACCESS_TOKEN }}
  run: pytest

Contributing

See CONTRIBUTING.md for development setup, running checks, and publishing new versions.

License

MIT