pytest-flakiness

The official Flakiness.io reporter for pytest.

[!TIP]

• Report demo is available at https://flakiness.io/flakiness/pytest-flakiness
• Supported features: features.md

Installation

1. Install using uv (recommended):

   uv add --dev pytest-flakiness
   

   Or via standard pip:

   pip install pytest-flakiness
   

2. Set your Flakiness.io project identifier (org/project) in your pytest config:

   # pyproject.toml
   [tool.pytest]
   flakiness_project = "my-org/my-project"
   

   See Project Configuration for other config file formats, or use the --flakiness-project flag / FLAKINESS_PROJECT env variable instead.

Usage

Once installed, simply run pytest. The reporter will automatically activate, aggregate test results, and create Flakiness Report in the flakiness-report directory.

pytest

The generated report can be viewed interactively via the Flakiness CLI Tool:

flakiness show

[!TIP] Make sure to add flakiness-report directory to your .gitignore

flakiness-report/

If Flakiness Access Token is passed, then the reporter will upload the report to Flakiness.io. You will see a confirmation in your terminal summary:

...
PASSED [100%]
==============================
✅ [Flakiness] Report uploaded: https://flakiness.io/your_org/your_proj/run/1
==============================

Project Configuration

Options that are stable across a project are best set once in your pytest config instead of being passed on every run. With pytest 9, add them to the [tool.pytest] table in pyproject.toml:

# pyproject.toml
[tool.pytest]
flakiness_project = "my-org/my-project"

The same keys work in [pytest] of pytest.ini / tox.ini / setup.cfg. See All Configuration Options for the full list — CLI flags and FLAKINESS_* environment variables still override these per run.

Uploading Reports to Flakiness.io

Github Actions

When running in GitHub Actions, the reporter can authenticate using GitHub's OIDC token — no access token needed.

For this to work:

1. The GitHub Actions workflow must have id-token: write permission.
2. The --flakiness-project option (or FLAKINESS_PROJECT env variable) must be set to your Flakiness.io project identifier (org/project).
3. The Flakiness.io project must be bound to the GitHub repository that runs the GitHub Actions workflow.

permissions:
  id-token: write

steps:
  - name: Run Tests
    run: pytest --flakiness-project="my-org/my-project"

You can also use the FLAKINESS_PROJECT environment variable instead of the CLI flag:

permissions:
  id-token: write

steps:
  - name: Run Tests
    env:
      FLAKINESS_PROJECT: my-org/my-project
    run: pytest

GitLab CI/CD

When running in GitLab CI/CD, the reporter can authenticate using a GitLab ID token — no access token needed.

GitLab mints ID tokens when the job starts, rather than on demand like GitHub Actions, so the job must declare one named FLAKINESS_ID_TOKEN whose audience matches your project identifier:

test:
  id_tokens:
    FLAKINESS_ID_TOKEN:
      aud: my-org/my-project   # must match --flakiness-project
  script:
    - pytest --flakiness-project="my-org/my-project"

For this to work:

1. The job must declare a FLAKINESS_ID_TOKEN ID token whose aud is your Flakiness.io project identifier (org/project).
2. The --flakiness-project option (or FLAKINESS_PROJECT env variable) must be set to that same identifier.
3. The Flakiness.io project must be bound to the GitLab project that runs the pipeline.

Access Token

Alternatively, you can authenticate using your project's Access Token. You can find this in your project settings on flakiness.io.

Set the Access Token using either an environment variable or command-line flag:

export FLAKINESS_ACCESS_TOKEN="flakiness-io-..."
pytest --flakiness-access-token="flakiness-io-..."

All Configuration Options

Each option can be set via a command-line flag, an environment variable, or an ini option (see Ini File Configuration below). When the same option is set in more than one place, the highest-priority source wins:

CLI flag > environment variable > ini file > built-in default

Ini File Configuration

Options can also be set in your pytest configuration file, which is handy for values that are stable across a project (such as flakiness_project). All of the ini option names above are recognized in any file pytest reads — pytest.ini, tox.ini, setup.cfg, or pyproject.toml.

pytest.ini (or tox.ini / setup.cfg under [pytest]):

[pytest]
flakiness_project = my-org/my-project
flakiness_name = pytest
flakiness_disable_upload = false

pyproject.toml — pytest 9 reads native TOML types from the [tool.pytest] table (recommended):

[tool.pytest]
flakiness_project = "my-org/my-project"
flakiness_name = "pytest"
flakiness_disable_upload = false

If you target older pytest, or prefer the string-based ini format, use [tool.pytest.ini_options] instead (don't combine both tables — pytest errors if it sees them together):

[tool.pytest.ini_options]
flakiness_project = "my-org/my-project"
flakiness_name = "pytest"
flakiness_disable_upload = "false"

Note: The access token is intentionally not available as an ini option — keep secrets out of version-controlled config files and pass --flakiness-access-token or FLAKINESS_ACCESS_TOKEN instead.

Custom Environment Data

You can add custom metadata to your test runs using FK_ENV_* environment variables. These might be handy to capture properties that affect system-under-test.

export FK_ENV_GPU_TYPE="H100"
export FK_ENV_DEPLOYMENT="staging"

The FK_ENV_ prefix is removed and keys are lowercased, e.g. FK_ENV_DEPLOYMENT becomes deployment, and FK_ENV_GPU_TYPE becomses gpu_type.

Local Development

To save reports locally, pass --flakiness-output-dir:

pytest --flakiness-output-dir=./flakiness-reports

This will create a report.json file and an attachments/ directory in the specified folder.

CI/CD Example (GitHub Actions)

Using GitHub OIDC (recommended — no secrets needed):

permissions:
  id-token: write

steps:
  - name: Run Tests
    run: pytest --flakiness-project="my-org/my-project"

Alternatively, using an access token:

- name: Run Tests
  env:
    FLAKINESS_ACCESS_TOKEN: ${{ secrets.FLAKINESS_ACCESS_TOKEN }}
  run: pytest

CI/CD Example (GitLab CI/CD)

Using a GitLab ID token (recommended — no secrets needed):

test:
  id_tokens:
    FLAKINESS_ID_TOKEN:
      aud: my-org/my-project
  script:
    - pytest --flakiness-project="my-org/my-project"

Alternatively, using an access token stored as a masked CI/CD variable:

test:
  script:
    - pytest
  variables:
    FLAKINESS_ACCESS_TOKEN: $FLAKINESS_ACCESS_TOKEN

Contributing

See CONTRIBUTING.md for development setup, running checks, and publishing new versions.

License

MIT