venvctl 1.3.8

A package to create portable virtual environments programmatically.

venvctl

venvctl is a CLI tool allowing the creation of fully portable Python virtual environments.

Synopsis

venvctl helps building fully portable Python virtual environments, in bulk, or single mode, keeping the state in config files. Each virtual environment comes with a detailed markdown report, allowing to control the integrity of its state, broken dependencies, errors, and warnings that occurred in the build process. Eventually, the folders ready for distributions get packed in tarballs, as shown in the example below:

...
│   ...
│   └── pyvenv.cfg
├── builds
│   ├── ansible_2_9.tar.gz
│   └── base.tar.gz
└── reports
    ├── ansible_2_9.md
    └── base.md

Limitations

It is evident that the portability is limited to systems sharing the same kernel, do not expect to build a virtual environment on a Debian based system, and deploy it on a RHEL host, to mention one of the many cases;

It is possible shipping a virtual environment with a different version of python other than the one installed on the destination system, yet again, do not assume to run python3 based venvs on a system solely configured with python 2.x.

Requirements

venvctl relies on a few packages to explicate its core functionalities:

piphyperd==1.9.6

piphyperd, a wrapper around pip to manage installations and audits.

markd==0.1.20

markd, a Python package that facilitates the generation of markdown flavored files.

virtualenv==20.0.15  # Virtual Python Environment builder.
click8==8.0.1  # Composable command line interface toolkit.
binaryornot==0.4.4  # Ultra-lightweight pure Python package to check if a file is binary or text.

Installation

venvctl is currently distributed only through PyPi.org

pip install --user venvctl

Visit the project page for further information about the package status and releases.

Documentation

For the detailed instructions and a full API walkthroug, refer to the Official Documenation.

You can leverage venvctl both programmatically, or calling the CLI, as shown in the examples below:

"""Python"""
from venvctl import VenvCtl

# Build virtualenvs in batch
VenvCtl(config_file=/path/to/config.json,
        python_binary=/usr/bin/python{version},
        output_dir=/my/output/dir).run()

#!/bin/bash

# Build virtualenvs in batch
venvctl generate \
    --config ~/path/to/your/config/venvs.json \
    --out ./venvs

"""Python"""
from venvctl import VenvCtl

name = "test-venv"
packages = [
    "Click==7.0",
    "binaryornot==0.4.4"
]

# Build a single virtual env;
# It will generate the config file for you.
VenvCtl.create_venv(name=name, packages=packages_list,
                    output_dir=/my/output/dir)

#!/bin/bash

# Build a single virtual env;
# It will generate the config file for you.
venvctl create --name my_venv --packages '["tox", "docker"]' --out /my/output/dir

A config file follows the json structure:

[
  {
      "name": "base",
      "packages": [
          "asn1crypto==1.3.0",
          "dnspython==1.16.0",
          "enum34==1.1.10",
          "ipaddress==1.0.23",
          "jmespath==0.9.5",
          "lxml==4.5.0",
          "paramiko==2.7.1",
          "psutil==5.7.0",
          "pycrypto==2.6.1",
          "pyopenssl==19.1.0",
          "python-ldap==3.2.0",
          "python-memcached==1.59"
      ]
  },
  {
    "name": "base_networking",
    "parent": "base",
    "packages": [
      "f5-sdk==3.0.21",
      "bigsuds==1.0.6",
      "netaddr==0.7.19",
      "cloudshell-networking-cisco-iosxr==4.0.6"
    ]
  },
  {
    "name": "ansible_2_9",
    "parent": "base",
    "packages": [
        "ansible==2.9.6"
    ]
  },
  {
    "name": "ansible_2_9_networking",
    "parent": "base_networking",
    "packages": [
        "ansible==2.9.6"
    ]
  }
]

The build process follows an inheritance pattern, in the example above, the environment named base is the core for the rest; ansible_2_9 inherits its packages; ansible_2_9_tox adds modules on the top of its parent, ansible_2_9.

With this logic, the build process in bulk can be quite fast, even when deploying complex virtual environments.

Run it with Containers

You can also take advantage of a container image, built to ship venvctl and the whole toolchain to build virutal environments leveraging both Python 2 and Python 3. Here to follow, two examples leveraging Docker.

Build virtual environments in batch, shipping Python 3.6.8 shipped with RHEL based systems:

docker run -it --rm -v $(pwd)/conf.json:/opt/conf.json \
  -v $(pwd)/venvs:/opt/venvs:rw \
  eu.gcr.io/hyperd-containers/venv-builder:latest venvctl generate \
  --config /opt/conf.json \
  --out /opt/venvs \

Build virtual environments in batch, shipping Python 2.7.16:

docker run -it --rm -v $(pwd)/conf.json:/opt/conf.json \
  -v $(pwd)/venvs:/opt/venvs:rw \
  eu.gcr.io/hyperd-containers/venv-builder:latest venvctl generate \
  --config /opt/conf.json \
  --out /opt/venvs \
  --python /usr/bin/python2

Containers limitations

Currently, the only image available is based on CentOS 8. It fits the purpose of any RHEL based deployments, however it won't be useful in other scenarios. More kernels will be added, stay tuned, or feel free to build your own and poke us.

License

GNU General Public License v3 (GPLv3)

About the author

Francesco Cosentino

I'm a surfer, a crypto trader, and a DevSecOps Engineer with 15 years of experience designing highly-available distributed production environments and developing cloud-native apps in public and private clouds.