prunepytest 0.6.0

only test relevant files

prunepytest

Do you have lots of Python tests?

Do you find yourself wondering whether you really need to run the full test suite for every tiny change?

Do you wish to know exactly which tests to run to assess the soundness of any given change?

Would you plausibly save a significant amount of time (and possibly money if your CI budget is significant) by skipping irrelevant tests?

Then prunepytest might be for you! Make sure you read the LICENSE first though.

Who can use this ?

As per the terms of the LICENSE, this software can be freely used as long as the code under test is either publicly available under an open source license, or private and never shared with anyone.

If you wish to use this on code that more than one individual can access, but that is not publicly available under an OSI or FSF approved license, you may take advantage of the trial period for up to 15 days per calendar year. After that, you MUST either obtain my written permission to continue using the software, or wait for usage restrictions to be lifted, 4 years after the initial public release.

Contact me

Installation

pip install prunepytest

How does it work?

prunepytest operates by first building an accurate module-level import graph for your Python code.

Then, based on that import graph, and a set of modified files (either provided explicitly, or inferred from a version control system), it is able to determine a safe set of test files that should be run.

The main interface is a pytest plugin that is intended to work out-of-the-box for most simple Python projects.

pytest --prune

The import graph can be computed very quickly by using a native extension, written in Rust with PyO3, which leverages the fast Python parser from ruff, and the parallel directory walker from ripgrep.

The combination of efficient native code with built-in support for parallelism means that a large codebase that would take over a minute to parse with Python's own ast module can be fully parsed, and have its transitive import closure computed in a matter of seconds!

Validating the import graph

Unfortunately, the Python import machinery is notoriously complex, due to the possibility arbitrary code execution at import time, and the ability to perform dynamic imports at run-time. For projects that depend on those features, it is important to first validate the accuracy of the import graph before relying on prunepytest for test selection.

import-time validation

As a first step, it is often useful to do a quick import-time validation, which works by:

• computing a static import graph by parsing the Python code
• hooking into the import machinery, then importing all test code, to obtain an accurate picture of the actual import graph.
• validating that the set of static imports obtained from parsing is a superset of the set obtained by importing the code

python -m prunepytest validate

test-time validation

By default, prunepytest does the safe thing, and reports all unexpected dynamic imports within a test context as failures. However, this will only validate tests that are being run. Before rolling out prunepytest in a CI system, it is highly advisable to do a first full-scale test-time validation, by disabling test pruning, via the --prune-no-select flag:

pytest --prune --prune-no-select

Test-time validation can be explicitly disabled via the --prune-no-validate flag

pytest --prune --prune-no-validate

Alternatively, the errors can be downgraded to warnings, via the --prune-no-fail flag.

pytest --prune --prune-no-fail

Sensible defaults, and overriding them

Both the import-time validator and the pytest plugin have sensible defaults: they scan the repository under test to automatically detect relevant Python packages, and attempt to detect common configuration files (such as pyproject.toml) to automatically adjust to slight deviations from conventions.

For more complicated repository layouts, the default behavior might not quite work. This can be addressed by creating a Python file that contains an implementation of prunepytest.api.PluginHook or prunepytest.api.ValidatorHook and point prunepytest to this file:

pytest --prune --prune-hook hook.py

For slight deviations, it is possible to subclass prunepytest.DefaultHook instead of starting from scratch, to leverage sensible defaults as a starting point.

For more details, refer to docstrings in api.py

Dealing with dynamic imports

If validation points to inaccuracies in the static import graphs, there are a few ways to deal with that.

Providing hints to the import parser

To ensure maximum accuracy, the import parser goes deep, and considers all import statements, not just at-module level, but arbitrarily nested, whether inside functions or conditional blocks.

This makes it extremely easy to provide hints about dynamic imports to the parser, by gating import statements behind an always-False conditional:

if False:  # for the import parser
  import foo.bar  # noqa: F401

For improved usability, the import parser also departs from the specification of the import machinery when it comes to wildcard imports. Specifically, while the Python interpreter depends on an explicit listing of submodules in the __all__ variable of a package's __init__.py, prunepytest will instead scan the filesystem, and resolve a wildcard to all existing modules next to the __init__.py. This makes it more practical to provide hints that encompass a large number of submodules, without having to update the hints every time modules are added or removed.

Considering typechecking-only imports

One notable exception to the "parser goes deep" outlined above is that, by default, typechecking-only imports are excluded from the import graph. If that exclusion is undesirable, it can be changed through a hook:

from prunepytest.api import DefaultHook

class Hook(DefaultHook):
   def include_typechecking(self) -> bool:
      return True

Specifying extra dependencies via a custom Hook

There are two main hooks to specify extra dependencies:

• dynamic_dependencies(), returns a mapping from module (specified as Python import path or filepath) to set of extra imports (specified as Python import paths, possibly including wildcards).
  Those extra dependencies are incorporated into the graph before computing the transitive closure.

• dynamic_dependencies_at_leaves(), returns a sequence of entries to be incorporated after computing of the transitive closure of the import graph.
  This can be useful when, for instance, a common method triggers dynamic import based on a configuration file, and that file differs across source roots.
  In such a case, it would be possible to explicitly specifying extra dependencies for each test file relying on that method but that might be tedious and brittle. Assigning varying dependencies to the common module offers a more succinct and robust way to specify the same dynamic dependency information.

Sample hooks

Sample implementations for some real-world open source projects are available in prunepytest-validation:

• pydantic v1
• pydantic v2

Reusing the import graph

In cases where the import graph is large and slow to build, it can be worth saving it in a serialized form to reuse across multiple commands.

# collect and save import graph
# NB: this is a no-op if a valid graph exists
python -m prunepytest graph --prune-graph graph.bin

# use the existing graph instead of parsing source code again
pytest --prune --prune-graph graph.bin

Limitations

Because it only relies on a statically derived import graph, prunepytest can be used immediately, without needing a first full run of the test suite to collect dependency information (although it might be worth doing a validation run, as explained earlier), and saving/transferring that information across builds.

However, that also means that it will not catch more complicated test dependencies, such as dependencies on config files, or data-driven test cases.

Handling such dependencies in the general case is possible, for instance through syscall-tracing, but intentionally out-of-scope of this project. A future release might add some facility to explicitly annotate non-Python dependencies, until then, the handling of non-Python dependencies in the context of test selection is left as an exercise for the user.

The pytest plugin does a limited best-effort handling of data-driven test cases, inspired by the desire to support mypy's test suite. Contributions intended to expand this to a broader set of data-driven test cases would be welcome.

FAQ

• What environments are supported?
  The minimum required Python version is 3.7
  The minimum required pytest version is 7.2
  Linux, macOS, and Windows are supported, and covered in CI.

• Is this compatible with xdist?
  Yes!
  The pytest plugin will automatically detect whether xdist is being used, and make necessary adjustments to avoid redundant import graph computation, and ensure continued correctness of test-time validation.
  NB: this has only been tested with pytest-xdist==3.6.1

• What about <other pytest plugin>?
  While there is no a-priori expectation of incompatibility, it is not practical to test interactions with all popular pytest plugins. Feel free to report any issue.

• How does this compare to pytest-testmon?
  testmon is the only other attempt I am aware of to solve the problem of selecting a minimal safe set of tests to run based on modified files. It's design is based on detailed code coverage data, which has the potential to more aggressively prune the test set. However, it has a number of significant drawbacks:

  • testmon requires enabling code coverage with coverage.py, which typically incurs 2-4x slowdown. With prunepytest, you are free to leverage the amazing slipcover instead.
  • testmon requires a full initial test run to build the dependency database. For prunepytest, this is only relevant as a validation step for codebases that rely on dynamic imports.
  • testmon still needs to parse the Python source code, which can add considerable overhead at test-selection time for large codebases.
  • In most cases, testmon's database is considerably larger than prunepytest's serialized import graph, which makes a big difference for distributed test runs.
  • testmon does not offer meaningful validation of its test selection logic, whereas prunepytest takes validation seriously, and has a comprehensive test suite that achieves a high level of code coverage.

• What's this weird license about?
  Large companies using this software in their CI system might unlock significant cost savings.
  In fact, I started prototyping this code to pitch a certain financial software company on hiring me as a contractor to save upward of $100k/month on EC2 costs alone! They turned me down, and I want to make sure they were serious when they told me they were not interested :)
  At the same time, I believe this tool could be tremendously useful for many open source projects. This weird license is my attempt to strike a balance between serving the needs of the open source ecosystem while incentivizing large companies to pay for software that they derive significant value from.

• So it's not actually Open Source?
  No.
  This license is not OSI-approved, and the presence of usage restrictions go against the fundamental "freedom 0", so it is unlikely to ever get the stamp of approval from OSI or FSF.

• But it will eventually be Open Source?
  Yes.
  Four years after the initial public release.

• Can I use it in the CI system of some Open Source project?
  Yes.

• Can I use it in the CI system of some closed source project?
  You need to contact me to get written permission.

• What if I make changes?
  The license still applies to any modified version. So using a modified version on open source code is fine, but using the modified version on code that is not open source would still require written permission from the copyright owner.

• Is anyone actually going to pay for this?
  Some people think that capitalism implies that corporations are rational actors, or at least tend to act more rationally that many individuals. If that were so, I would expect that any company with a significant Python codebase that incurs a large CI expenditure, and lacking the time or expertise to build equivalent software in-house, would be excited to pay for prunepytest.
  Pay how much? At least as much as they would expect to save in the amount of time it would take them to build an equivalent in-house solution. And that would be arguably be a screaming deal, because they would be saving on the payroll required to fund an in-house alternative, avert the risk of the in-house project failing, and derive usability benefits beyond the CI cost savings.
  Whether there are any companies that would both benefit from this software, and can rationally assess its value to them, is an open question.

• Can I contribute?
  Maybe.
  You have to be willing to make your contribution under a sufficiently permissive license for me to integrate it, and ensure that any prospective private user only have to get written permission from a single entity. Basically that means BSD, MIT, Apache, or Unlicense/Public Domain.

• My codebase is too gnarly, help?!?
  If your codebase is open source, submit an issue in the tracker.
  If your codebase is not open source, I am open to negotiating a contract to help integrate this software in your CI system. Contact me