tox-constraints

Reproducible tests, with minimal configuration, by default

Benefits

This plugin is helpful because it

makes using pinned versions the default,
tells virtualenv to use the pinned versions,
facilitates gathering of dependencies, and
alerts you of the above pitfalls so that, hopefully, you do not have to learn it the hard way.

Motivation

The best way to improve reproducibility of tox without this plugin is to set PIP_CONSTRAINT using either

setenv, or
passenv.

This is good, it ensures that all packages installed with pip will use the constraints file¹.

Setting PIP_CONSTRAINT does however not ensure that packages installed with other tools will use the constraints file. Notably tox creates environments using virtualenv, which seeds the environment with some version of pip, setuptools, and wheel. By default, a version that was bundled with the virtualenv is used, so while the behavior may be surprising it should be reproducible. The versions installed can be controlled² using setenv or passenv to set

VIRTUALENV_PIP,
VIRTUALENV_SETUPTOOLS, and
VIRTUALENV_WHEEL.

Setting PIP_CONSTRAINT also does not ensure that all packages that will be installed are listed in the constraints file³. pip-compile is a great tool to help both resolve all transient dependencies and assign a consistent set of versions. But it cannot pick up deps from tox.ini file or build-system.requires from pyproject.toml.

Limitations

Known limitations and problems include

deps from environments not on the envlist will not be gathered.
-l should be set when gathering dependencies to avoid actually running the environments.
The build-system.requires section from pyproject.toml must be manually reproduced in a text file to make it available to pip-compile. There is an open issue in pip-tools that, if implemented, would resolve this.

Using the -c flag on the other hand does not ensure that build dependencies are pinned, see pip#8439. ↩
Except when it does not. Something about it caching and upgrading packages locally causes it to occasionally ignore the specified versions. It can be hard to realize that this is happening and when it does the best course of action seems to be removing the cache at ~/.local/share/virtualenv/. ↩
One could enable hash checking mode in which case pip would refuse to install any package for which it has not been given a hash. However, this creates new problems such as the package under test not having a hash. This package previously attempted to solve this use case but stopped since hash checking mode has been mostly broken in pip since the new resolver. 😮‍💨 ↩

Metadata

Release files for tox-constraints 0.14.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tox-constraints 0.14.1
File	Size	Uploaded
tox-constraints-0.14.1.tar.gz	12.5 kB	May 23, 2022	Details

Built distribution (wheel)

Table of built distributions (wheels) for tox-constraints 0.14.1
File	Interpreter	ABI	Platform	Reset
tox_constraints-0.14.1-py3-none-any.whl 7.5 kB May 23, 2022	Python 3	none	any	Details

Total release size: 20.0 kB

Release files / tox-constraints-0.14.1.tar.gz

Download URL	tox-constraints-0.14.1.tar.gz
Size	12.5 kB
Tags	Source
SHA-256 checksum How to use checksums	`716def98703d1029debbb670908a8566ce60aa26cd04dcf39231608d23a347b6`
BLAKE2b-256 checksum How to use checksums	`45f8e19cd592bc84ab56b36f986a9ac6c864541ff4c07bd260502c1842e0e572`
Upload date	May 23, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.0 CPython/3.9.13`

Release files / tox_constraints-0.14.1-py3-none-any.whl

Download URL	tox_constraints-0.14.1-py3-none-any.whl
Size	7.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`65563a92aa74762c8b48dab0a75c77a5a5bd19c6d989d55abd6532277bcbdf49`
BLAKE2b-256 checksum How to use checksums	`3db9daf9fa7beff4dae7cce18d83633c9b6888ec726593385cdf7ac474cb915b`
Upload date	May 23, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/4.0.0 CPython/3.9.13`