Skip to main content

Use layered Python virtual environment stacks to share large dependencies

Project description

Virtual Environment Stacks for Python

Machine learning and AI libraries for Python are big. Really big. Nobody wants to download and install multiple copies of PyTorch or CUDA if they can reasonably avoid it.

venvstacks allows you to package Python applications and all their dependencies into a portable, deterministic format, without needing to include copies of these large Python frameworks in every application archive.

It achieves this by using Python's sitecustomize.py environment setup feature to chain together three layers of Python virtual environments:

  • "Runtime" layers: environment containing the desired version of a specific Python runtime
  • "Framework" layers: environments containing desired versions of key Python frameworks
  • "Application" layers: environments containing components to be launched directly

Application layer environments may include additional unpackaged Python launch modules or packages for invocation with python's -m switch.

While the layers are archived and published separately, their dependency locking is integrated, allowing the application layers to share dependencies installed in the framework layers, and the framework layers to share dependencies installed in the runtime layers.

Refer to the Project Overview for an example of specifying, locking, building, and publishing a set of environment stacks.

Installing

venvstacks is available from the Python Package Index, and can be installed with pipx:

$ pipx install venvstacks

Alternatively, it can be installed as a user level package (although this may make future Python version upgrades more irritating):

$ pip install --user venvstacks

Interactions with other packaging tools

The base runtime environment layers are installed with pdm (with the installed runtimes coming from the python-build-standalone project). pdm is also used to manage the development of the venvstacks project itself.

The layered framework and app environments are created with the standard library's venv module.

The Python packages in each layer are currently being installed directly with pip, but are expected to eventually move to being installed with uv to reduce environment setup times during builds.

Platform-specific environment locking for each layer is performed using uv pip compile. Refer to pyproject.toml for the specific issues preventing the adoption of uv for additional purposes.

venvstacks expects precompiled wheel archives to be available for all included Python distribution packages. When this is not the case, other projects like wagon or fromager may be useful in generating the required input archives.

Caveats and Limitations

  • This project does NOT support combining arbitrary virtual environments with each other. Instead, it allows larger integrated applications to split up their Python dependencies into distinct layers, without needing to download and install multiple copies of large dependencies (such as the PyTorch ML/AI framework). The environment stack specification and build process helps ensure that shared dependencies are kept consistent across layers, while unshared dependencies are free to vary across the application components that need them.
  • The venvstacks Python API is not yet stable. Any interface not specifically declared as stable in the documentation may be renamed or relocated without a deprecation period. API stabilisation (mostly splitting up the overly large venvstacks.stacks namespace) will be the trigger for the 1.0 milestone release.
  • While the venvstacks CLI is broadly stable, there are still some specific areas where changes may occur (such as in the handling of relative paths).
  • Dynamic library dependencies across layers currently only work on Windows. There is a proposal in place for resolving that limitation, but it has not yet been implemented.
  • Local exports to filesystems which do not support symlinks (such as VFAT and FAT32) are nominally supported (with symlinks being replaced by the files they refer to), but this support is not currently tested.
  • To avoid relying on the Python ecosystem's still limited support for cross-platform component installation, the stack build processes need to be executed on the target platform (for example, by using an OS matrix in GitHub Actions).

Project History

The initial (and ongoing) development of the venvstacks project is being funded by LM Studio, where it serves as the foundation of LM Studio's support for local execution of Python AI frameworks such as Apple's MLX.

The use of "🐸" (frog) and "🦎" (newts are often mistaken for lizards and vice-versa!) as the Unicode support test characters is a reference to the internal LM Studio project that initially built and continues to maintain venvstacks: Project Amphibian.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

venvstacks-0.2.1.tar.gz (93.5 kB view details)

Uploaded Source

Built Distribution

venvstacks-0.2.1-py3-none-any.whl (45.5 kB view details)

Uploaded Python 3

File details

Details for the file venvstacks-0.2.1.tar.gz.

File metadata

  • Download URL: venvstacks-0.2.1.tar.gz
  • Upload date:
  • Size: 93.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: pdm/2.21.0 CPython/3.12.7 Linux/6.5.0-1025-azure

File hashes

Hashes for venvstacks-0.2.1.tar.gz
Algorithm Hash digest
SHA256 1b2a490363b62951405e5ae55ec7424e7f93138eedf99ed2d216d12d76490a48
MD5 dc8809c103653b5975122598f2bb7ac4
BLAKE2b-256 a8fbf9aae83aa6581e29f88468a98857f29e0504683bac993f9db1a8dde8f685

See more details on using hashes here.

File details

Details for the file venvstacks-0.2.1-py3-none-any.whl.

File metadata

  • Download URL: venvstacks-0.2.1-py3-none-any.whl
  • Upload date:
  • Size: 45.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: pdm/2.21.0 CPython/3.12.7 Linux/6.5.0-1025-azure

File hashes

Hashes for venvstacks-0.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 72d256cb6ac4b7550d29e9627cfda56a72905bce56e705925f5902c7042ac4fa
MD5 1df8e69d7015f0b0be00e544e79234aa
BLAKE2b-256 e2e12a2bb2a7d688d8f4712dbfee52f662c7d13fc20f2b85681960b1a377f23e

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page