Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

An opinionated CLI tool for Python monorepo MGMT.

Project description

Scream

Still very much a WIP. Lots of work to be done; please open issues or pull requests.

Python 2.7 Python 3.7 License: MIT

What you do when you see a bunch of python... packages.

or...

An opinionated CLI tool for Python monorepo MGMT.


It's objective is to ease the creation, testing, and deploying of multiple python packages in a single repository. To ensure non-overlapping names with PYPI, this tool forces you to use namespace packages. Namespaces are defined according to python's pkgutil-style.

Why use Scream?

  • Scream holds all packages to the same standards

    • Enforces consistent styling.
    • Enforces consistent testing.
    • Enforces consistent documentation.
  • Uses tox to setup virtualenvs for isolated testing across python versions.

  • Pre-commit hooks to help prevent those gosh darn mistakes.

Problems with most monorepos

  1. Testing & CI/CD pipeline can become slow with many packages.

    The other available solutions are:

    These tools are extremely powerful, but sometimes are overkill, and introduce a fair amount of overhead to manage.

  2. Installing intra-monorepo package dependencies is hard with a private repositories.

    The available solutions are:

    • Have all your packages distributed publically.
    • Host a private PYPI index.
    • pip install using dependency links.

How scream works

Scream aims to cause as little overhead as possible for managing your monorepo. No fancy third party configuration or private PYPI repositories.

Scream uses the existing python packaging requirements to resolve intra-monorepo dependencies, and git to detect what's changed since the parent branch.

For example:

cat setup.cfg.

[metadata]
name = company_packageA
version = 0.0.1
description = Your package description!

[options]
packages = find:
install_requires =
    company_packageB

If you make a change to company_packageA then run tests...

> scream test --dry-run

The following packages have changes compared since branch: `master`:
        company_packagea

Packages that require testing:
        company_packagea
        company_packageb

Commands

  • scream init - Run this first. Initiates a monorepo in an empty directory.
  • scream new <package-name> - Creates new template package.
  • scream test [--dry-run][--all][--parallel] - Tests packages and package dependents that have changed.
  • scream install <package-name>[--test] - Installs a package. --test installs testing dependencies.
  • deploy <package-name> - Runs deploy.py in your package directory.
  • scream build - Builds a python wheel and bundles it with all it's dependencies as wheels. (TODO)
  • -v will enable verbose logs.

Quickstart

By default packages are tested against python 3.7.x, which means you have it available on your PATH. If you about different versions, please see the configuration options.

> mkdir mymonorepo
> cd mymonorepo

> scream init
Done!
Create a new package with `scream new <namespace>.<package_name>`

> scream new com.packagea
Created project `com.packagea`

> scream new com.packageb
Created project `com.packageb`

> scream test --all --parallel
Running tests... (Using all cores)

> coverage report

Using your monorepo packages

The two common ways you would install packages from this monorepo are:

  1. Clone your repository and run scream build or scream install in your CI tool to build packages and ship them to your machines.
  2. Standard pip install individual packages to any machine from private or public github repos.

If your repository is public, you can simply install a subpackage anywhere using:

pip install 'git+ssh://git@github.com/ORG/REPO.git@master#subdirectory=examplea'

If your repository is private, you need a few extra steps to make sure packages that depend on other packages in this monorepo can be installed.

  1. Specify dependency_links in the setup.cfg for each 'local' dependency:
dependency_links =
    git+ssh://git@github.com/ORG/REPO.git@master#egg=examplea-0#subdirectory=subpackages/examplea
  1. pip install as before, but specifiy the flag --process-dependency-links
pip install 'git+ssh://git@github.com/ORG/REPO.git@master#subdirectory=examplea' --process-dependency-links

Configuration

  1. In your packages setup.cfg the variable python_requires determines which versions of python your package will be tested against.
  • Ex. python_requires = 2.7, 3.6, 3.7

    Note: the python versions you intend to test must be available on your path.

Upcoming Features

  • [ ] Auto version handling.
  • [ ] Pre commit hook to flake8.
  • [ ] Pre commit hook to avoid committing a large file.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for scream, version 0.0.34
Filename, size File type Python version Upload date Hashes
Filename, size scream-0.0.34-py2.py3-none-any.whl (25.0 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes
Filename, size scream-0.0.34.tar.gz (18.5 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page