pipxl

Additional tools for pip - compile, sync, deptree & license

Installation

pip install pipxl

Features

pipxl is a wrapper around pip, the default Python package manager.

pipxl adds the following to pip:

• pipxl compile: from a list of top-level dependencies, generate a pip-compliant requirements file with all dependencies and dependencies-of-dependencies pinned
• pipxl sync: sync current environment with the requirements file
• pipxl deptree: show for a set of packages the dependencies and dependencies-of-dependencies including all version specifiers
• pipxl deptreerev: show for a set of packages the dependency tree in reverse
• pipxl license: list the licensing of the dependencies, including sub-dependencies, in an easy format

compile

Compile can take in a pyproject.toml file, one or more requirements file and direct package specifications, including combinations of those:

# pyproject.toml
$ pipxl compile .

# pyproject.toml, with custom output_file. Defaults to requirements_lock.txt
$ pipxl compile . --o my_requirements_lock.txt 

# pyproject.toml, including optional dependencies defined under `all`
$ pipxl compile .[all]

# single requirements file
$ pipxl compile --r requirements.in

# multiple requirements files
$ pipxl compile --r requirements.in --r requirements-dev.in

# direct package specifications
$ pipxl compile pandas==1.4.3

The generated file includes all packages, with version, the correct hash, and detailed version information for why the dependency was included:

pandas==1.4.3 \
    --hash=sha256:6f803320c9da732cc79210d7e8cc5c8019aad512589c910c66529eb1b1818230
numpy==1.23.3 \
    --hash=sha256:004f0efcb2fe1c0bd6ae1fcfc69cc8b6bf2407e0f18be308612007a0762b4089
    # via pandas [numpy (>=1.21.0) ; python_version >= "3.10"]
python-dateutil==2.8.2 \
    --hash=sha256:961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9
    # via pandas [python-dateutil (>=2.8.1)]
pytz==2022.2.1 \
    --hash=sha256:220f481bdafa09c3955dfbdddb7b57780e9a94f5127e35456a48589b9e0c0197
    # via pandas [pytz (>=2020.1)]
six==1.16.0 \
    --hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254
    # via python-dateutil [six (>=1.5)]

sync

Sync updates the active (virtual) environment:

$ pipxl sync --r requirements_lock.txt

deptree

The deptree command shows the dependency tree, listing for each requested package all dependencies, dependencies-of-dependencies, and so on.

# pyproject.toml
$ pipxl deptree .
...

# direct package specification
$ pipxl deptree pandas==1.4.3
pandas==1.4.3
        python-dateutil==2.8.2 [python-dateutil (>=2.8.1)]
                six==1.16.0 [six (>=1.5)]
        pytz==2022.2.1 [pytz (>=2020.1)]
        numpy==1.23.3 [numpy (>=1.21.0) ; python_version >= "3.10"]

deptreerev

The output of deptree works top-down: for every requested package, the dependencies are listed. deptreerev works the opposite way: for each package, it lists the packages that depend on it. This is useful to figure out why a particular dependency has been installed.

$ pipxl deptreerev pandas==1.4.3 numpy==1.21.1
pandas==1.4.3
numpy==1.21.1
        pandas==1.4.3 [numpy (>=1.21.0) ; python_version >= "3.10"]
python-dateutil==2.8.2
        pandas==1.4.3 [python-dateutil (>=2.8.1)]
pytz==2022.2.1
        pandas==1.4.3 [pytz (>=2020.1)]
six==1.16.0
        python-dateutil==2.8.2 [six (>=1.5)]
                pandas==1.4.3 [python-dateutil (>=2.8.1)]

This shows that for instance six is a dependency of python-dateutil, which in turn is a dependency of pandas. In square brackets, you can see how the dependency has been defined. For example, python-dateutil requires six >= 1.5; the pip resolver has selected the most recent version 1.16.0.

license

Not all packages on PyPi are licensed for usage in all environments, particularly not in corporate environments. It is thus useful to check under which license the package comes. The problem is complicated by dependencies of the dependencies; there are typically a large number of those and they may change over time. The license commands collects all dependencies and dependencies-of-dependencies, and groups the output by license. Do note that some packages may not have uploaded the license information (properly) to PyPi. For those, *UNKNOWN* will be displayed. pipxl does not attempt to extract it from Github, the wheel, or other sources, as it may result in the wrong license being assigned.

# pyproject.toml
$ pipxl license .
...

$ pipxl license pandas==1.4.3
BSD                 : numpy
BSD-3-Clause        : pandas
Dual License        : python-dateutil
MIT                 : pytz, six

Why pipxl?

pipxl is a set of additional tools to pip that I wish would be part of pip, as they are useful in nearly every Python project I work on.

Initial inspiration for creating pipxl comes from the release of pip 22.2, which adds the following to pip install

• --dry-run flag to not actually install packages
• a reports flag providing detailed JSON output of the resolution process.

In combination, the two allow us to pass a set of requirements to pip, and get the results of the resolver without hooking into the pip api. Alternatives, most with a narrower scope, such as pip-tools do hook into the api, but it is not official and not recommended, although for lack of alternatives (until pip 22.2!) various packages do use it. By using the CLI, the risk of breakage on pip version upgrades should be limited (although the JSON report may change).

What is still missing is being able to provide a list of packages to be considered installed, as to make use of the --upgrade-strategy=only-if-needed option. Right now, pipxl will eagerly upgrade all packages not requested (i.e. dependencies-of-dependencies), as it cannot simulate this behavior currently. In theory, it could set up a temporary virtual environment, actually install current packages, dry-run pip install with the only-if-needed flag, and record the results.

Non-goals

• Dependency resolution: pipxl uses the pip resolver, and does not, and will not, try to aid pip. pipxl is geared towards explaining what pip, and in particular the pip resolver, does, not replacing it.

Alternatives

• pip-tools: offers pip-compile and pip-sync, similar to pipxl lock and pipxl sync. Relies on pip internals. See this issue for using the pip CLI interface instead.
• pipgrip: pip dependency resolver using the PubGrub algorithm, useful for viewing dependency trees
• pipdeptree: offers functionality similar to pipxl deptree and pipxl deptreerev
• dep_license: offers functionality similar to pipxl license
• poetry: alternative to pip
• pdm: alternative to pip

License

pipxl is distributed under the terms of the MIT license.