pytest-subset

A pytest plugin for deterministically sampling a subset of a heavily parametrized test's variants.

Installation

uv pip install pytest-subset

Usage

import pytest

@pytest.mark.subset(fraction=0.2)
@pytest.mark.parametrize("value", range(1000))
def test_something(value):
    ...

Running pytest normally will run ~20% of test_something's variants, chosen deterministically. Pass --full (or -F) to run every variant, e.g. in CI before a release.

Marker reference

@pytest.mark.subset(fraction=..., seed=None)

• fraction: float in (0, 1]. Fraction of parametrized variants to keep. Optional if subset_default_fraction is set in your pyproject.toml.
• seed: optional int. A seed value used to generate a pseudo-random slice of the parametric test space. Pins the selection for this test. If not provided, a default seed is generated based on the test module and Node ID. In either case, a given seed value will always generate the same subset of tests (i.e. deterministic)

CLI options

• -F, --full — ignore every subset marker and run all variants.
• --salt=SEED — controls the slice for tests that don't pin their own seed=. Accepts:
  • an integer, so different values give different, still-reproducible slices;
  • random, to pick a fresh seed each run. It's printed after the run so you can reuse it later.

import pytest

@pytest.mark.subset(fraction=0.2, seed=42)
@pytest.mark.parametrize("value", range(1000))
def test_something(value):
    ...

@pytest.mark.subset(fraction=0.2)
@pytest.mark.parametrize("value", range(1000))
def test_something_else(value):
    ...

test_something always uses seed 42 — an explicit seed= superceeds --salt every time. test_something_else has no seed of its own, so --salt decides its slice:

pytest --salt=22       # test_something_else samples using salt 22
pytest --salt=random   # a fresh seed is picked and printed after the run

Note that --salt=random picks each run's slice independently, so slices across runs are not guaranteed to be disjoint.

Distributed runs (pytest-xdist)

pytest-subset is compatible with pytest-xdist; -n needs no extra flags.

xdist has known limitations, requiring every worker to collect the same tests in the same order. To account for this, the plugin resolves the salt once on the controller and hands it to each worker. That includes --salt=random: all workers share the one freshly picked seed, and the value printed after the run reproduces the distributed selection exactly.

Note: xdist support was only added in v0.2.0.

pytest -n auto --salt=random

Ini options

Set this under [tool.pytest.ini_options] in your pyproject.toml (or the equivalent section of pytest.ini/setup.cfg):

• subset_default_fraction — fraction used by @pytest.mark.subset when it's applied bare, with no fraction= argument. This value is applied globally across the test suite.

  [tool.pytest.ini_options]
  subset_default_fraction = "0.3"
  

Then, in your test modules,

import pytest

@pytest.mark.subset  # 0.3 applied here by default.
@pytest.mark.parametrize("value", range(1000))
def test_something(value):
    ...