Invoke Common Tasks

Some common tasks for PyInvoke to bootstrap your code quality and testing workflows.

• Invoke Common Tasks
  • Getting Started
    • Invoke Setup
  • The Tasks
    • build
    • format
    • lint
    • typecheck
    • test (and coverage)
    • ci
    • init-config
  • TODO
  • Roadmap
  • All Together
• Contributing
• Development
• Resources
• Prior Art

Getting Started

pip install invoke-common-tasks
# Or
poetry add -D invoke-common-tasks

# With Extras
pip install invoke-common-tasks[all]
# Or
poetry add --group dev invoke-common-tasks[all]

invoke-common-tasks defines a few extras, where you can also install the tooling to go with each task. By default we do not install the tools that these tasks call, since you could have different pinned versions than what we specify.

However, you can install all of them or distinct subsets:

• format -> black, isort
• lint -> flake8, flake8-docstrings
• typecheck -> mypy
• test -> pytest, pytest-cov, coverage[toml]

So you can specify the following if you only want format and test:

pip install invoke-common-tasks[format,test]

All tasks will still be available but we won't install associated tooling.

Invoke Setup

tasks.py

from invoke_common_tasks import * # noqa

Once your tasks.py is setup like this invoke will have the extra commands:

λ invoke --list
Available tasks:

  build         Build wheel.
  ci            Run linting and test suite for Continuous Integration.
  format        Autoformat code for code style.
  init-config   Setup default configuration for development tooling.
  lint          Linting and style checking.
  test          Run test suite.
  typecheck     Run typechecking tooling.

You can also initialise default configuration for each tool by running the following:

invoke init-config --all

More details in the init-config section.

The Tasks

build

Assuming you are using poetry this will build a wheel (and only a wheel).

format

This will apply code formatting tools black and isort.

These are only triggers for these commands, the specifics of configuration are up to you.

Recommended configuration in your pyproject.toml:

[tool.black]
line-length = 120

[tool.isort]
profile = "black"
multi_line_output = 3
import_heading_stdlib = "Standard Library"
import_heading_firstparty = "Our Libraries"
import_heading_thirdparty = "Third Party"

lint

This will run checks for black, isort and flake8.

Up to you to specify your preferences of plugins for flake8 and its configuration.

Recommended configuration in .flake8:

[flake8]
exclude = 
    venv,
    dist,
    .venv
select = ANN,B,B9,BLK,C,D,DAR,E,F,I,S,W
ignore = E203,E501,W503,D100,D104
per-file-ignores =
    tests/*: D103,S101
max-line-length = 120
max-complexity = 10
import-order-style = google
docstring-convention = google

Recommended flake8 plugins:

• flake8-docstrings

More flake8 plugins:

https://github.com/DmytroLitvinov/awesome-flake8-extensions

typecheck

Simply runs mypy ..

Recommended configuration to add to your pyproject.toml

[tool.mypy]
pretty = true
show_error_codes = true
show_column_numbers = true
show_error_context = true
exclude = [
  'tests/',
  'tasks\.py'
]
follow_imports = 'silent'
ignore_missing_imports = true

# Work your way up to these:
disallow_incomplete_defs = true
# disallow_untyped_defs = true 
# disallow-untyped-calls = true
# strict = true

test (and coverage)

This will simply run python3 -m pytest. This is important to run as a module instead of pytest since it resolves a lot of import issues.

You can simply not import this task if you prefer something else. But all config and plugins are left flexible for your own desires, this simply triggers the entrypoint.

Recommended configuration in pyproject.toml:

[tool.pytest.ini_options]
minversion = "6.0"
addopts = "-s -vvv --color=yes --cov=. --no-cov-on-fail"

[tool.coverage.run]
omit = ["tests/*", "**/__init__.py", "tasks.py"]
branch = true

Assuming you also install pytest-cov and coverage[toml].

Recommended pytest plugins:

• pytest-xdist - Run tests in parallel using maximum cpu cores
• pytest-randomly - Run tests in random order each time to detect tests with unintentional dependencies to each other that should be isolated. Each run prints out the seed if you need to reproduce an exact seeded run.
• pytest-cov - It is recommended to run coverage from the pytest plugin.

List of other pytest plugins:

https://docs.pytest.org/en/latest/reference/plugin_list.html

ci

This is a task with no commands but chains together lint, typecheck and test.

init-config

Experimental: This feature is still in a pre-release state.

Each of the above commands came with some recommended configuration. This command attempts to automate setting up even that part in your pyproject.toml and .flake8 files.

λ invoke init-config --help
Usage: inv[oke] [--core-opts] init-config [--options] [other tasks here ...]

Docstring:
  Setup default configuration for development tooling.

Options:
  -a, --all
  -f, --format
  -l, --lint
  -t, --test
  -y, --typecheck

TODO

• Add task for auto generation of Sphinx Docs.
• Add typechecking support to root tasks.py file now that Invoke v2+ is released.

Roadmap

This project will get marked as a stable v1.0 once the above TODO features are ticked off and this has seen at least 6 months in the wild in production.

All Together

Once all the tasks are imported, you can create a custom task as your default task with runs a few tasks chained together.

from invoke import task
from invoke_common_tasks import *

@task(pre=[format, lint, typecheck, test], default=True)
def all(c):
  """Default development loop."""
  ...

You will notice a few things here:

1. The method has no implementation ...
2. We are chaining a series of @tasks in the pre=[...] argument
3. The default=True on this root tasks means we could run either invoke all or simply invoke.

How cool is that?

Contributing

At all times, you have the power to fork this project, make changes as you see fit and then:

pip install https://github.com/user/repository/archive/branch.zip

Stackoverflow: pip install from github branch

That way you can run from your own custom fork in the interim or even in-house your work and simply use this project as a starting point. That is totally ok.

However if you would like to contribute your changes back, then open a Pull Request "across forks".

Once your changes are merged and published you can revert to the canonical version of pip installing this package.

If you're not sure how to make changes or if you should sink the time and effort, then open an Issue instead and we can have a chat to triage the issue.

Development

git clone https://github.com/neozenith/invoke-common-tasks
cd invoke-common-tasks
poetry shell
# This will not install the dev dependencies which are optional when a consumer uses the project but are actually needed for development of this library
poetry install --all-extras

Resources

• pyinvoke

Prior Art

• https://github.com/Smile-SA/invoke-sphinx
• https://github.com/Dashlane/dbt-invoke
• https://invocations.readthedocs.io/en/latest/index.html