Skip to main content

Python Compose, a tool for spinning up many long-running Python services simultaneously.

Project description

Python Compose

License PyPI - Python Version Supported Platforms PyPI - Downloads GitHub Repo stars

Python Compose is a command line tool and library for spinning up many long-running python applications in isolated Python environments. Inspired by Docker Compose, Python Compose is meant to provide similar functionality without having to relying on Docker containers, networking etc.

Currently, Python Compose supports the following environments:

In the future, we wish to support:

Installation

Use the package manager pip to install python-compose.

pip install python-compose

Currently, Python Compose supports venv, pyenv-virtualenv and conda environments. The pyenv-virtualenv and conda managers are not installed with Python Compose. To install them, follow instructions from the pyenv-virtualenv and conda documentation.

Concepts

Unit

A unit is an object which represents how to create, run and clean up a service for a specific type of Python environment handler. Examples can be found in the python_compose/unit directory.

YAML Specification

Python Compile can currently only be instantiated from a YAML file with specification derived from the internal spec.py Python file. A Python Compose YAML file expects a top-level key, units which maps to a list of units i.e. Python scripts to run. Each unit's environment is described by the key unit_type. A unit will have different arguments. Some required, some are optional. Again, these definitions are in spec.py

Alternatives Comparison

Python Compose isn't the only tool in the world for this task. It happens to be a convenient one if you're working in a Python only environment and want to minimize external tools and dependencies. Below is a chart comparing Python Compose versus other solutions.

Tool Operating Systems Virtual Networking Ability to Run Non-Python Programs Multiple Python Version Support Non-Python Dependencies Required
Python Compose Linux/MacOS With Conda or subprocess
Python Multiprocessing Linux/MacOS/Windows With subprocess
Docker Compose Linux/MacOS/Windows
Systemd Unit Files Linux
Bazel Linux/MacOS/Windows

Example

We provide many examples of how to use Python Compose as both a library and as a command line tool in the examples directory. Below we provide what we will assume is the most Docker Compose-like use case, starting many backend services from a single config file.

In a directory, create a Python script, httpd.py that starts an HTTP service:

import argparse
from http.server import BaseHTTPRequestHandler, HTTPServer

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("port", type=int)
    args = parser.parse_args()

    print(args)
    server_address = ("", args.port)
    httpd = HTTPServer(server_address, BaseHTTPRequestHandler)
    httpd.serve_forever()

Then, create a yaml file, config.yaml to start this backend service in multiple environments:

units:
- unit_type: "venv"
  name: "httpd_0"
  env_dir: "./.envs"
  script_path: "./httpd.py"
  script_args: ["8080"]
- unit_type: "venv"
  name: "httpd_1"
  env_dir: "./.envs"
  script_path: "./httpd.py"
  script_args: ["8081"]
- unit_type: "pyenv-virtualenv"
  name: "httpd_2"
  py_version: "3.10"
  script_path: "./httpd.py"
  script_args: ["8082"]
- unit_type: "conda"
  name: httpd_3
  script: ["python3", "../conda/httpd.py", "8083"]

To spin up all containers, run

python3 -m python-compose config.yaml

To exit, press <Ctrl-C>.

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

Setting up Your Development Environment.

Developers should first install all needed development dependencies:

pip3 install -e '.[dev]'

Then make all desired changes

Adding a new Unit

To add a new unit, you will want to first add the unit to python_compose/unit. Then, add an example to the examples directory and finally update spec.py.

Testing

To test Python Compile, run pytest after installation of all dev dependencies.

License

MIT

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

python_compose-1.2.0.tar.gz (12.4 kB view details)

Uploaded Source

Built Distribution

python_compose-1.2.0-py3-none-any.whl (12.4 kB view details)

Uploaded Python 3

File details

Details for the file python_compose-1.2.0.tar.gz.

File metadata

  • Download URL: python_compose-1.2.0.tar.gz
  • Upload date:
  • Size: 12.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: python-httpx/0.24.0

File hashes

Hashes for python_compose-1.2.0.tar.gz
Algorithm Hash digest
SHA256 fb7668b55f0f242c570b0280e0e54923b54b6518a33ded53dbe455a7b708426e
MD5 d8e9ff776b6f397fed9129cc6f1182e9
BLAKE2b-256 a0199067335c67d27af9a3aa3597b1940a62fd9e11b465e6931744969d695d55

See more details on using hashes here.

Provenance

File details

Details for the file python_compose-1.2.0-py3-none-any.whl.

File metadata

File hashes

Hashes for python_compose-1.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 9865e22770711c9fe285baab45c7211a70d20cd6c1af0c69e6784b59fede74fb
MD5 54fab7a22aa1944c358d56170a263676
BLAKE2b-256 52e1fac1f54cdf039b30ffc4ac12330d3032f97a153f4a9d3c318681d1db58ba

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page