Skip to main content

Test Pilot

Format, lint and test only the files that have changed on the current branch.

See also: Test Pilot presentation slides

Test Pilot formats, lints and tests the files that have changed on your current branch compared to main:

$ testpilot
testpilot=> Formatting
testpilot=> Linting
testpilot=> Running unit tests
testpilot=> Printing coverage report
...
Name                                       Stmts   Miss Branch BrPart     Cover   Missing
-----------------------------------------------------------------------------------------
lms/app.py                                    44      0      0      0   100.00%
lms/assets.py                                  8      0      0      0   100.00%
lms/new_file_committed_on_this_branch.py       0      0      0      0   100.00%
lms/new_untracked_file.py                      0      0      0      0   100.00%
tests/unit/lms/app_test.py                    15      0      4      0   100.00%
tests/unit/lms/assets_test.py                  5      0      0      0   100.00%
-----------------------------------------------------------------------------------------
TOTAL                                         72      0      4      0   100.00%
testpilot=> Running functional tests
.

It compares your branch to main and finds:

  • New, modified and deleted files
  • Committed, staged and untracked changes
  • Source, test and functional test files
  • If you've modified a source file (e.g. src/foo/bar.py) Test Pilot will find its corresponding unit test file (e.g. tests/unit/foo/bar_test.py) and will run those tests (as well as formatting and linting the test file). This also works the other way round: if you modify a unit test file then its corresponding source file will be formatted and linted.

This is much faster than running make sure and produces much less output, but it'll still catch 99% of problems on your branch. Just run the full make sure once before sending your pull request.

Only for Hypothesis projects

For now, Test Pilot only works with Hypothesis projects. It uses all sorts of knowledge and assumptions about how the Hypothesis development environment works, what formatting, testing and linting tools we use, how we organize our test files, etc. Test Pilot won't work at all with non-Hypothesis projects unless they happen to do everything exactly as our projects do.

Test Pilot bypasses tox

By default Test Pilot bypasses tox and runs commands directly. For example it runs .tox/tests/bin/pytest directly instead of tox -e tests. This is faster but it means tox won't install or update the dependencies in the virtualenv for you if they aren't already installed and up to date. Other tox things like setting environment variables etc won't happen either. If it doesn't seem to be working try running testpilot with -t / --tox (alias: -s / --slower) and it'll run all the commands through tox instead:

$ testpilot --tox

After doing this once you can usually go back to running testpilot without -t.

Although testpilot without -t won't keep the dependencies in your virtualenv up to date it will detect if your virtualenv doesn't exist at all and call tox to create it. It'll also detect if the project's version of Python isn't installed and call pyenv to install it:

$ testpilot
testpilot=> Installing Python
Downloading Python-3.8.12.tar.xz...
-> https://www.python.org/ftp/python/3.8.12/Python-3.8.12.tar.xz
Installing Python-3.8.12...
...
Installed Python-3.8.12 to /home/seanh/.pyenv/versions/3.8.12

testpilot=> Formatting
testpilot=> It looks like .tox/format/bin/black doesn't exist, running tox to install it
.tox recreate: /home/seanh/Projects/lms/.tox/.tox
...

Debugging

You can use -d / --debug to get Test Pilot to print out exactly what commands it's running and a few other details:

$ testpilot --debug
testpilot=> Running git rev-parse --is-inside-work-tree
testpilot=> Project class: <class 'testpilot.app.StandardProject'>
testpilot=> Installing Python
testpilot=> Running bin/install-python
testpilot=> Running git branch --format %(refname:lstrip=-1) -l main master
testpilot=> Project's main branch: 'master'
testpilot=> Running git diff origin/master --name-only
testpilot=> Running git ls-files --others --exclude-standard
testpilot=> Formatting
testpilot=> Running pyenv exec .tox/format/bin/black --quiet lms/app.py lms/assets.py lms/new_file_committed_on_this_branch.py lms/new_untracked_file.py tests/functional/lti_certification/v13/grading/test_assignment_and_grading.py tests/unit/lms/app_test.py tests/unit/lms/assets_test.py
...

Help

See testpilot --help for the rest of the command line options:

$ testpilot --help

Installing

We recommend using pipx to install Test Pilot. First install pipx then run:

pipx install testpilot

You now have Test Pilot installed! For some help run:

testpilot --help

Upgrading

To upgrade to the latest version run:

pipx upgrade testpilot

To see what version you have run:

testpilot --version

Uninstalling

To uninstall run:

pipx uninstall testpilot

Setting up Your Test Pilot Development Environment

First you'll need to install:

  • Git. On Ubuntu: sudo apt install git, on macOS: brew install git.
  • GNU Make. This is probably already installed, run make --version to check.
  • pyenv. Follow the instructions in pyenv's README to install it. The Homebrew method works best on macOS. The Basic GitHub Checkout method works best on Ubuntu. You don't need to set up pyenv's shell integration ("shims"), you can use pyenv without shims.

Then to set up your development environment:

git clone https://github.com/hypothesis/testpilot.git
cd testpilot
make help

Releasing a New Version of the Project

  1. First, to get PyPI publishing working you need to go to: https://github.com/organizations/hypothesis/settings/secrets/actions/PYPI_TOKEN and add testpilot to the PYPI_TOKEN secret's selected repositories.

  2. Now that the testpilot project has access to the PYPI_TOKEN secret you can release a new version by just creating a new GitHub release. Publishing a new GitHub release will automatically trigger a GitHub Actions workflow that will build the new version of your Python package and upload it to https://pypi.org/project/testpilot/.

Changing the Project's Python Versions

To change what versions of Python the project uses:

  1. Change the Python versions in the cookiecutter.json file. For example:

    "python_versions": "3.10.4, 3.9.12",
    
  2. Re-run the cookiecutter template:

    make template
    
  3. Commit everything to git and send a pull request

Changing the Project's Python Dependencies

To change the production dependencies in the setup.cfg file:

  1. Change the dependencies in the .cookiecutter/includes/setuptools/install_requires file. If this file doesn't exist yet create it and add some dependencies to it. For example:

    pyramid
    sqlalchemy
    celery
    
  2. Re-run the cookiecutter template:

    make template
    
  3. Commit everything to git and send a pull request

To change the project's formatting, linting and test dependencies:

  1. Change the dependencies in the .cookiecutter/includes/tox/deps file. If this file doesn't exist yet create it and add some dependencies to it. Use tox's factor-conditional settings to limit which environment(s) each dependency is used in. For example:

    lint: flake8,
    format: autopep8,
    lint,tests: pytest-faker,
    
  2. Re-run the cookiecutter template:

    make template
    
  3. Commit everything to git and send a pull request

Metadata

Release files for testpilot 0.2.9

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for testpilot 0.2.9
File Size Uploaded Provenance Actions
testpilot-0.2.9.tar.gz 22.9 kB Without provenance Details for testpilot-0.2.9.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for testpilot 0.2.9
File Interpreter ABI Platform Actions
testpilot-0.2.9-py3-none-any.whl Python 3 none any Details for testpilot-0.2.9-py3-none-any.whl

Total release size: 32.8 kB

Release files / testpilot-0.2.9.tar.gz

Download URL testpilot-0.2.9.tar.gz
Size 22.9 kB
Tags Source
SHA-256 checksum
How to use checksums
af8bb46c8436292f2209d44a7c25667384bea8235569d49f3d45fbb22dc54318
BLAKE2b-256 checksum
How to use checksums
ff8acf053363ea594de0a0a51ee3a288017810aabfa34d71ee11d4fc779b0c75
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.0.1 CPython/3.12.8

Release files / testpilot-0.2.9-py3-none-any.whl

Download URL testpilot-0.2.9-py3-none-any.whl
Size 9.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
5d47e770f3a0d18c670552a3f2f6492e56a10d123850d95e22b2256f82a8df1c
BLAKE2b-256 checksum
How to use checksums
c231ae57b3d0cbc6fa640eebd9561d1c2997343ad812336ed1762da407d08931
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.0.1 CPython/3.12.8

Security for testpilot 0.2.9

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page