pipgrip 0.0.2

Composable command line interface toolkit

pipgrip

pipgrip is a lightweight pip dependency resolver with deptree preview functionality based on the PubGrub algorithm, which is also used by poetry. For one or more PEP 508 dependency specifications, pipgrip recursively fetches/builds the Python wheels necessary for version solving, and optionally discovers and renders the full resulting dependency tree.

Installation

This pure-Python, OS independent package is available on PyPI:

pip install pipgrip

Usage

This package can be used to:

• Alleviate Python dependency hell by resolving the latest viable combination of required packages
• Render an exhaustive dependency tree for any given pip-compatible package(s) with --tree
• Detect version conflicts for given constraints and give human readable feedback about it
• Find dependency conflicts in local projects:
  • pipgrip --tree .
• Install complex packages without worries using:
  • pip install -U --no-deps `pipgrip --pipe aiobotocore[awscli]`
• Generate a lockfile with a complete working set of dependencies (see known caveats)
  • pipgrip aiobotocore[awscli] | pip install -U --no-deps -r /dev/stdin
  • pipgrip --lock -tree aiobotocore[awscli] && pip install -U --no-deps -r ./pipgrip.lock

$ pipgrip --help

Usage: pipgrip [OPTIONS] [DEPENDENCIES]...

Options:
  --lock                  Write out pins to './pipgrip.lock'.
  --pipe                  Output space-separated pins instead of newline-
                          separated pins.
  --json                  Output pins as json dict instead of newline-
                          separated pins.
  --tree                  Output human readable dependency tree (top-down).
  --reversed-tree         Output human readable dependency tree (bottom-up).
  --max-depth INTEGER     Maximum tree rendering depth (defaults to -1).
  --cache-dir PATH        Use a custom cache dir.
  --no-cache-dir          Disable pip cache for the wheels downloaded by
                          pipper. Overrides --cache-dir.
  --index-url TEXT        Base URL of the Python Package Index (default
                          https://pypi.org/simple).
  --extra_index-url TEXT  Extra URLs of package indexes to use in addition to
                          --index-url.
  --pre                   Include pre-release and development versions. By
                          default, pip only finds stable versions.
  -v, --verbose           Control verbosity: -v will print cyclic dependencies
                          (WARNING), -vv will show solving decisions (INFO),
                          -vvv for development (DEBUG).
  --help                  Show this message and exit.

Dependency trees

Exhaustive dependency trees without the need to install any packages (at most build some wheels).

$ pipgrip --tree pipgrip

pipgrip (0.0.1)
├── anytree (2.7.3)
│   └── six (1.13.0)
├── click (7.0)
├── packaging>=17 (20.0)
│   ├── pyparsing>=2.0.2 (2.4.6)
│   └── six (1.13.0)
├── pkginfo>=1.4.2 (1.5.0.1)
├── setuptools>=38.3 (44.0.0)
└── wheel (0.33.6)

Lockfile generation

Using the --lock option, resolved (pinned) dependencies are additionally written to ./pipgrip.lock.

$ pipgrip --lock boto3

boto3==1.10.46
botocore==1.13.46
docutils==0.15.2
jmespath==0.9.4
python-dateutil==2.8.1
six==1.13.0
urllib3==1.25.7
s3transfer==0.2.1

Version conflicts

If version conflicts exist for the given (ranges of) package version(s), a verbose explanation is raised.

$ pipgrip auto-sklearn~=0.6 dragnet==2.0.4

Error: Because dragnet (2.0.4) depends on scikit-learn (>=0.15.2,<0.21.0)
 and auto-sklearn (0.6.0) depends on scikit-learn (<0.22,>=0.21.0), dragnet (2.0.4) is incompatible with auto-sklearn (0.6.0).
And because no versions of auto-sklearn match >0.6.0,<1.0, dragnet (2.0.4) is incompatible with auto-sklearn (>=0.6.0,<1.0).
So, because root depends on both auto-sklearn (~=0.6) and dragnet (==2.0.4), version solving failed.

NOTE: If older versions of auto-sklearn are allowed, PubGrub will try all acceptable versions of auto-sklearn. In this case, auto-sklearn==0.5.2 requires scikit-learn (<0.20,>=0.19), making it compatible with dragnet==2.0.4.

Cyclic dependencies

If cyclic dependencies are found, it is noted in the resulting tree.

$ pipgrip --tree keras==2.2.2

keras==2.2.2 (2.2.2)
├── h5py (2.10.0)
│   ├── numpy>=1.9.1 (1.18.0)
│   └── six>=1.9.0 (1.13.0)
├── keras-applications==1.0.4 (1.0.4)
│   ├── h5py (2.10.0)
│   │   ├── numpy>=1.9.1 (1.18.0)
│   │   └── six>=1.9.0 (1.13.0)
│   ├── keras==2.2.2 (2.2.2, cyclic)
│   └── numpy>=1.9.1 (1.18.0)
├── keras-preprocessing==1.0.2 (1.0.2)
│   ├── keras==2.2.2 (2.2.2, cyclic)
│   ├── numpy>=1.9.1 (1.18.0)
│   ├── scipy>=0.14 (1.4.1)
│   │   └── numpy>=1.9.1 (1.18.0)
│   └── six>=1.9.0 (1.13.0)
├── numpy>=1.9.1 (1.18.0)
├── pyyaml (5.2)
├── scipy>=0.14 (1.4.1)
│   └── numpy>=1.9.1 (1.18.0)
└── six>=1.9.0 (1.13.0)

Known caveats

• pip install -U `pipgrip package` without --no-deps is unsafe while pip doesn't yet have a built-in dependency resolver, and leaves room for interpretation by pip
• Package names are canonicalised in wheel metadata, resulting in e.g. path.py -> path-py and keras_preprocessing -> keras-preprocessing in output
• VCS Support isn't implemented yet
• --reversed-tree isn't implemented yet
• Since pip install -r does not accept . as requirement, --pipe format must be used when installing local projects
• Installing packages using pipgrip is not very intuitive, so maybe pipgrip needs a stable --install flag

Development

Create a virtual environment and get ready to develop:

make install

This make-command is equivalent to the following steps:

Install pre-commit and other continous integration dependencies in order to make commits and run tests.

pip install -r requirements/ci.txt
pre-commit install

With requirements installed, make lint and make test can now be run. There is also make clean, and make all which runs all three.

To import the package in the python environment, install the package (-e for editable installation, upon import, python will read directly from the repository).

pip install -e .

See also

• PubGrub spec
• pip needs a dependency resolver
• pipdeptree
• mixology
• poetry-semver
• johnnydep

---

BSD 3-Clause License

Copyright (c) 2020, ddelange
All rights reserved.