pypants 2.1.1

CLI for working with Python packages and BUILD files in a Pants monorepo

pypants

CLI for working with Python packages and BUILD files in a Pants monorepo.

Features:

• Auto-generate BUILD files based on the package type and import statements
• Generate new Python package folders through an interactive CLI
• Compute a topologically-sorted list of dependencies for a given Python build target

Table of Contents:

• Installation
• Guide
  • Commands
  • Project Configuration
  • Package Configuration
  • Package Types
  • Registering Extra Targets
  • Package Generators
• Development

Installation

pypants requires Python 3.6 or above

pip install pypants

Guide

Commands

pypants process-requirements

Update 3rdparty/python/import-map.json using the entries in 3rdparty/python/requirements.txt. All this does is convert the published package name to an import name. Execute this command when you add a new requirement to requirements.txt.

pypants process-packages

Auto-generate all relevant BUILD files in the project/repo. You should execute this command in a git pre-commit or pre-push hook so your BUILD files are kept up to date. You can also run it on demand after you add a new dependency to an internal package.

pypants generate-package

Starts an interactive CLI that generates a new package folder. This depends on the package generators you registered.

Project Configuration

To configure your project, add a file named .pypants.cfg to the root of your Git repo and paste the example below. You should define the top_dirs option at a minimum.

[project]

; **************
; COMMON OPTIONS
; **************

; REQUIRED: Top-level directories to search for Python packages. These are relative
; to your project/repo root. This is a JSON list of strings.
top_dirs = ["."]

; Prefix to use for names of packages generated by pypants. e.g. foobar_
; python_package_name_prefix =

; Never look for or process files in these directories. This is a JSON list of
; strings. e.g. ["node_modules", "generators"]
; ignore_dirs = []

; ****************
; UNCOMMON OPTIONS
; ****************

; Set of target package names to ignore when collecting build targets. This is a
; JSON list of strings.
; ignore_targets = []

; Path to the location of the import-map.json file relative to the project root
; third_party_import_map_path = 3rdparty/python/import-map.json

; Path to the requirements.txt relative to the project root. The default value is
; the default that Pants uses.
; third_party_requirements_path = 3rdparty/python/requirements.txt

Besides the JSON lists, other options are parsed with Python's built-in ConfigParser.

Package Configuration

pypants currently expects the Python package to be structured like:

mypackage/
├── setup.py
├── .pypants.cfg <---- this is the pypants config file
├── src/
    ├── BUILD <---- pypants will generate this file
    ├── mypackage/
        ├── __init__.py
        ├── ...source code...
├── tests/
    ├── unit/
        ├── BUILD <---- pypants will generate this file
        ├── ...unit tests...
    ├── functional/
        ├── BUILD <---- pypants will generate this file
        ├── ...functional tests...

To configure each package, add a file named .pypants.cfg to the package folder and paste the example below. You should define the type option at a minimum.

[package]

; **************
; COMMON OPTIONS
; **************

; REQUIRED: Package type. See Package Types section for available values.
type = library

; ****************
; UNCOMMON OPTIONS
; ****************

; Extra set of dependencies to include in the python_library target. This is a
; JSON list of strings.
; extra_dependencies = []

; Extra set of tags to include in the Pants build targets. This is a JSON list of
; strings.
; extra_tags = []

; Flag denoting whether to generate a BUILD file.
; generate_build_file = true

; Flag denoting whether to generate a pex_binary target for local.py. This is
; essentially an extra entry point. It's only used for specific package types.
; generate_local_binary = false

; Flag denoting whether to include a pex_binary target for pytest
; generate_pytest_binary = false

Package Types

Each of the package types will result in a different BUILD file.

library

The BUILD file for internal Python libraries has one target defined. For example:

python_library(
    dependencies=[
        "3rdparty/python:arrow",
        "3rdparty/python:isoweek",
        "lib/code/src",
        "lib/logger/src",
    ],
    sources=["my_library/**/*"],
    tags={"code", "lib", "python"},
)

There is no name provided so this target can be referenced just by its containing folder path. In this case it would be "<TOPDIR>/my_library/src".

binary

A binary target can be used for executable scripts (CLIs and servers) and usually depend on internal libraries. The BUILD has a library and binary target defined:

python_library(
    name="lib",
    dependencies=[
        "3rdparty/python:boto3",
        "3rdparty/python:cfn-flip",
        "3rdparty/python:Click",
        "3rdparty/python:jsonschema",
        "lib/logger/src",
    ],
    sources=["cli_deploy/**/*"],
    tags={"apps", "code", "python"},
)
pex_binary(
    name="deploy",
    dependencies=[":lib"],
    source="cli_deploy/cli.py",
    tags={"apps", "code", "python"},
)

• The python_library target is pretty much the same as an internal Python library package
• The pex_binary target defines an explicit name. This is because when we go to build the PEX file, we want to define the filename. In this example, running ./pants binary apps/cli_deploy/src:deploy will result in dist/deploy.pex.
• The only dependency for the binary should be the library. The library will then include all the dependencies.
• source points to the entry point of the binary. This module should handle the if __name__ == "__main__" condition to kick off the script.

test

pypants looks for subfolders named unit, functional, or component within a package's tests/ folder. The BUILD file for test folders have a few targets defined. For example:

python_library(
    name="lib/time_utils/tests/unit",
    dependencies=[
        "3rdparty/python:arrow",
        "lib/python_core/src",
        "lib/time_utils/src"
    ],
    sources=["**/*"],
    tags={"lib", "python", "tests", "unit"},
)
python_tests(
    dependencies=[":lib/time_utils/tests/unit"],
    sources=["**/*.py"],
    tags={"lib", "python", "tests", "unit"},
)
pex_binary(
    name="unittest",
    entry_point="unittest",
    dependencies=[":lib/time_utils/tests/unit"]
)

• The python_library target is mostly here to define the unit tests dependencies in a single place so the other two targets can point to it
• The python_tests target lets us run pytest against the test files that match **/*.py
• The pex_binary target lets us run the unittest module directly. We won't actually package up this target via ./pants binary. Setting the entry_point to "unittest" is essentially the same as running python -m unittest test_something.py from the command line.

lambda_function

The BUILD file for the Lambda handler contains a special-purpose build target: python_awslambda. This target is a wrapper around lambdex. It creates a PEX like the pex_binary target (you can execute it) but it modifies the PEX to work with a Lambda Function. For example:

python_library(
    name="my-lambda-lib",
    sources=["lambda_handler/**/*"],
    dependencies=[
        "3rdparty/python:requests",
        "lib/logger/src",
    ],
)
pex_binary(
    name="my-lambda-bin",
    source="lambda_handler/lambda_handler.py",
    dependencies=[":my-lambda-lib"],
)
python_awslambda(
    name="my-lambda",
    binary=":my-lambda-bin",
    handler="lambda_handler.lambda_handler:lambda_handler",
)

This BUILD file will be placed in the same folder as the .pypants.cfg file.

migration

The BUILD file for an Alembic migration uses the python_app target to include the loose version files:

python_library(
    name="lib",
    dependencies=[
        "3rdparty/python:alembic",
        "3rdparty/python:SQLAlchemy",
        "lib/core/src",
    ],
    sources=["**/*"],
    tags={"code", "db", "migration", "python"},
)
pex_binary(name="alembic", entry_point="alembic.config", dependencies=[":lib"])
python_app(
    name="migrations-my-database-name",
    archive="tar",
    binary=":alembic",
    bundles=[
        bundle(fileset=["alembic.ini"]),
        bundle(fileset=["env.py"]),
        bundle(fileset=["versions/*.py"]),
    ],
    tags={"code", "db", "migration", "python"},
)

This BUILD file will be placed in the same folder as the .pypants.cfg file.

behave

The BUILD file for a behave test package includes a library target with test dependencies and a binary target that wraps behave. For example:

python_library(
    name="lib",
    dependencies=[
        "3rdparty/python:requests",
        "lib/application_config/src",
    ],
    sources=["**/*"],
    tags={"integration", "python", "tests", "tests-integration"},
)
pex_binary(
    source="behave_cli.py",
    dependencies=[":lib"],
    tags={"integration", "python", "tests", "tests-integration"},
)

This BUILD file will be placed in the same folder as the .pypants.cfg file.

The behave_cli.py source references a wrapper script that you should add to the folder:

"""Programmatic entrypoint to running behave from the command line"""
import os
import sys

from behave.__main__ import main as behave_main

if __name__ == "__main__":
    cwd = os.getcwd()
    os.chdir(os.path.dirname(__file__))
    try:
        exit_code = behave_main(sys.argv[1:])
    finally:
        os.chdir(cwd)
        sys.exit(exit_code)

py2sfn_project

py2sfn is a framework that simplifies the creation and deployment of workflows to AWS Step Functions. The BUILD file for a project only includes a generic target with the set of task dependencies:

target(
    dependencies=[
        "stepfunctions/projects/example-project/tasks/lambda_fetchjoke/src:lib",
        "stepfunctions/projects/example-project/tasks/lambda_generatelist/src:lib",
        "stepfunctions/projects/example-project/tasks/lambda_rankcharactersbyjoke/src:lib",
    ],
    tags={"py2sfn-project", "python", "stepfunctions/projects"},
)

This BUILD file will be placed in the same folder as the .pypants.cfg file.

Registering Extra Targets

If your project contains internal packages that don't aren't represented cleanly by the .pypants.cfg file, you can register extra targets programmatically.

1. In your repo, create a new file at .pypants/targets.py
2. Define a top-level function called register_extra_targets. Within that function, instantiate your extra build targets and return a dictionary that maps package name to BuildTarget.

For example, if you have several Alembic database folders:

"""Module that defines extra pypants build targets"""
from typing import Dict

from pypants.config import PROJECT_CONFIG
from pypants.build_targets import AlembicMigrationPackage


def register_extra_targets() -> Dict[str, "pypants.build_targets.base.PythonPackage"]:
    """Register extra targets specific to MyProject"""
    targets = {}

    # Register task targets for Alembic database migration targets
    #
    # * For migrations, this searches db/ looking for eny.py files. If it finds one,
    #   it means we've found an Alembic migration folder and can register a build
    #   target.
    env_py_paths = PROJECT_CONFIG.config_dir_path.joinpath("db").glob("**/env.py")
    for env_py_path in env_py_paths:
        alias = env_py_path.parent.name.replace("_db", "").replace("_", "-")
        package_name = f"migrations-{alias}"
        target = AlembicMigrationPackage(
            target_type="code",
            build_template="migration",
            top_dir_name="db",
            package_dir_name=env_py_path.parent.name,
            package_path=str(env_py_path.parent),
            package_name=package_name,
            build_dir=str(env_py_path.parent),
            extra_tags={"migration"},
        )
        targets[package_name] = target

    return targets

Package Generators

The generate-package command can be used to create a new package on disk. It sources package "generators" (folders that define the package boilerplate) from the .pypants/generators folder in your repo. To create a new package generator, copy one of the folders from examples/generators/ to <your repo>/.pypants/generators/<name> and modify it as needed. The generators use a tool called cookiecutter to rendere templates.

Development

If you're working on pypants locally and want to test out how your changes impact your target project:

1. Activate the virtualenv for your target project
2. Install flit: pip install flit
3. Keeping your virtualenv activated, install pypants via cd /path/to/pypants/repo && flit install -s