pytest-imply 0.1.7

Pytest plugin for test implication — skip tests implied by stronger ones

pytest-imply

A pytest plugin for test implication — skip tests that are implied by stronger ones.

The problem

Parametrized test suites often have an ordered severity axis:

test_count[c100]   PASSED   2.58s
test_count[c500]   PASSED   4.62s
test_count[c1000]  PASSED   7.13s
test_count[c5000]  PASSED  27.45s
test_count[c10000] PASSED  52.90s

If c10000 passes, the smaller counts will almost certainly pass too — running them all wastes CI time.

The solution

pytest-imply lets the developer declare implication relationships between tests. When a stronger test passes, weaker tests are short-circuited with a synthetic PASSED result:

test_count[c10000] PASSED  52.90s
test_count[c5000]  IMPLIED
test_count[c1000]  IMPLIED
test_count[c500]   IMPLIED
test_count[c100]   IMPLIED

For named-token implication, reports now display the provider(s):

test_complex    PASSED
test_simple     IMPLIED by [test_complex]

• implied_by: reports exactly one provider
• implied_by_any: reports the first matched provider
• implied_by_all: reports all providers

(Full details on these markers are in the Markers section below.)

If the strongest test fails, the plugin works downward to find the threshold — no time wasted on tests below the passing level.

Installation

pip install pytest-imply

Markers

monotonic — parametrized implication

For tests parametrized along an ordered axis:

@pytest.mark.monotonic("count")
@pytest.mark.parametrize("count", [100, 500, 1000, 5000, 10000])
def test_stress(count):
    run_workload(count)

The first parametrize item runs first in asc mode, and the last item runs first in desc mode (default).

Example ordering behavior with non-sorted parametrization:

@pytest.mark.monotonic("n")              # default direction="desc"
@pytest.mark.parametrize("n", [1, 5, 3, 7, 2])
def test_range(n):
    assert n > 0

desc (default): the input list is reversed (a list reversal, not a numeric sort), then candidate items are probed in binary-search order over that reversed sequence. For [1, 5, 3, 7, 2], the reversed sequence is [2, 7, 3, 5, 1], so the first probe is 2, then the remaining items are visited in binary-search order within [2, 7, 3, 5, 1].

asc:

@pytest.mark.monotonic("n", direction="asc")
@pytest.mark.parametrize("n", [1, 5, 3, 7, 2])
def test_range_asc(n):
    assert n > 0

asc: the input list is used in its original order, then candidate items are probed in binary-search order over that sequence. For [1, 5, 3, 7, 2], the first probe is 1, then the remaining items are visited in binary-search order within [1, 5, 3, 7, 2].

A range() style parametrization is also perfectly valid:

@pytest.mark.monotonic("count")
@pytest.mark.parametrize("count", range(1, 17))
def test_stress(count):
    run_workload(count)

By default, monotonic uses bisect=True: if the first candidate fails, it probes remaining items in binary-search order over the current sequence, rather than checking each item in sequence.

Disable this binary-search probing with bisect=False to use strict linear ordering instead:

@pytest.mark.monotonic("count", bisect=False)
@pytest.mark.parametrize("count", [100, 500, 1000, 5000, 10000])
def test_stress(count):
    run_workload(count)

implies / implied_by / implied_by_any / implied_by_all — named tokens

For arbitrary implication relationships between tests:

@pytest.mark.implies("full_stack_ok")
def test_complex():
    """Complex test — if this passes, the simpler test is implied."""
    ...

@pytest.mark.implied_by("full_stack_ok")
def test_simple():
    """Implied by the passing complex test — no need to run."""
    ...

Use implied_by_any for OR semantics (implied if any token was recorded):

@pytest.mark.implied_by_any("tcp_ok", "udp_ok")
def test_loopback():
    """Implied if either transport passed."""
    ...

Use implied_by_all for AND semantics (implied only if all tokens were recorded):

@pytest.mark.implied_by_all("tcp_ok", "udp_ok")
def test_both_transports():
    """Implied only if both transports passed."""
    ...

Ordering

The plugin builds a dependency graph from all implication relationships and topologically sorts the test items using Kahn's algorithm. This guarantees that implying tests always run before their implied dependents. Tests without implication markers keep their original collection order.

Caveats

• Transitive propagation: when a test is implied (short-circuited), its implies tokens are still propagated, so downstream tests see them. This means chains like A→B→C work as expected.

• Stacked markers: multiple markers of the same type on one test are all honoured. For example, stacking two @pytest.mark.implies decorators records both sets of tokens.

• Token namespacing: tokens live in a single flat namespace. In large projects, use a prefix convention (e.g., "mymodule::token") to avoid accidental collisions between independently-authored test modules.

• Fixtures: when a test is implied, its fixtures do not run. If an implied test has a session- or module-scoped fixture whose side effects other tests depend on, those tests may break. Only mark tests as implied when their fixtures are not needed by other tests. The plugin warns about non-function-scoped fixtures on implied tests. See Suppressing fixture warnings for ways to silence these when the fixtures are known to be safe.

• pytest-xdist: the plugin does not support parallel workers. Implication state is per-process. When xdist is active with workers, implication logic is automatically disabled and a warning is emitted. Use -p no:imply to suppress the warning.

• Plugin interoperability: when a test is implied, the plugin returns True from pytest_runtest_protocol, which tells pytest the item is fully handled. Other plugins that wrap the test protocol (e.g., pytest-cov, pytest-timeout) will not see implied tests. This is inherent to the short-circuit design.

• Ordering plugins (e.g., pytest-randomly, pytest-ordering): pytest-imply topologically sorts test items to satisfy dependency constraints. If another plugin reorders items after the sort, pytest-imply detects the change at the start of execution and emits a PytestImplyWarning. Implication stays active: tests whose providers still happen to run first are implied as usual, while others simply run normally. Correctness is never affected.

• Orphan tokens: if an implied_by (or variant) references a token that no test provides via implies, a warning is emitted at collection time.

• Dependency cycles: if implication markers form a cycle, the plugin falls back to original collection order for the affected tests and emits a warning.

• xfail interaction: a test marked @pytest.mark.xfail with strict=False that passes unexpectedly (xpass — an unexpected pass) does record its implies tokens. With strict=True, an xpass is treated as a failure and tokens are not recorded.

Disabling implication

--no-imply

Disable all implication logic; all tests are executed unconditionally:

pytest --no-imply

Use this for exhaustive nightly or release-gate runs to verify that the developer's implication assumptions still hold.

imply_enabled ini option

Disable implication via configuration instead of a CLI flag:

[tool.pytest.ini_options]
imply_enabled = false

Suppressing fixture warnings

The plugin warns when an implied test uses a non-function-scoped fixture, since that fixture's side effects will be skipped. For fixtures that are safe to skip (e.g. session-scoped autouse directory creation), suppress the warning in two ways:

imply_ignore_fixtures ini option (global)

List fixture names that should never trigger the warning:

[tool.pytest.ini_options]
imply_ignore_fixtures = ["_prepare_log_dir", "db_schema"]

@pytest.mark.imply_suppress (per-test)

Suppress the warning for a specific test or test class:

@pytest.mark.imply_suppress
@pytest.mark.monotonic("count")
@pytest.mark.parametrize("count", [100, 1000, 10000])
def test_stress(count, session_fixture):
    ...

Apply it at the class level to suppress for all tests in the class:

@pytest.mark.imply_suppress
class TestSuite:
    @pytest.mark.monotonic("count")
    @pytest.mark.parametrize("count", [100, 1000])
    def test_a(self, count, session_fixture):
        ...

How it works

1. Collection — pytest_collection_modifyitems builds the dependency graph and topologically sorts items.
2. Execution — pytest_runtest_protocol checks whether a test is implied; if so, it emits synthetic TestReport objects and returns True.
3. Recording — pytest_runtest_makereport records tokens and monotonic passes after a test succeeds.
4. Reporting — pytest_report_teststatus renders implied tests as IMPLIED PASS / IMPLIED FAIL with a cyan i or red I marker.

Verbosity levels

By default (no -v), implied tests show only the short letter (i/I) in the dots line. With -v, a concise label is shown including the provider for token-based implications:

test_load[1000]  IMPLIED PASS
test_load[5000]  IMPLIED FAIL
test_simple      IMPLIED PASS by [test_complex]

With -vv, the full reason (including bisect details) is shown:

test_load[1000]  IMPLIED PASS monotonic bisect (count, desc)
test_load[5000]  IMPLIED FAIL monotonic bisect (count, desc)
test_simple      IMPLIED PASS by [test_complex]

License

MIT