Skip to main content

Distributed execution utility.

Project description

Distributed Executor

GitHub License Python Version

A cross-platform reference implementation for processing shell commands over a distributed, asynchronous queue. dist_exe is a single producer (server) many consumer (client) system. It functions similar to GNU Parallel but scales out to many servers and even across operating systems.

This tool uses only standard library methods so there are no dependencies. It has been tested on Linux, macOS, and Windows 10 in Python 3.7 environments. The server and clients don't even need to be using the same platform.

Installation

pip install dist_exe [--upgrade] [--user]

Usage

The server role reads from standard input and publishes to a queue. If you don't have something actively creating work a good approach is to create a MANIFEST file and use tail -f to wait for work to arrive.

user@host-01: ~ $ touch MANIFEST
user@host-01: ~ $ tail -f MANIFEST | dist_exe server -H 0.0.0.0 -p 5050 -k SECRET

The client role connects to the remote queue and pulls items off, executing them using the local shell. You can specify a template for the command line invocation, otherwise the items are take as being the entire invocation.

user@host-02: ~ $ dist_exe client -H host-01 -p 5050 -k SECRET -t 'echo {}'

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dist_exe-0.0.6.tar.gz (5.3 kB view details)

Uploaded Source

File details

Details for the file dist_exe-0.0.6.tar.gz.

File metadata

  • Download URL: dist_exe-0.0.6.tar.gz
  • Upload date:
  • Size: 5.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.0.1 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3

File hashes

Hashes for dist_exe-0.0.6.tar.gz
Algorithm Hash digest
SHA256 9c12d43c89a60c359ccc3d8b3851cb302242d01d638fa32d64d15333bf8e3558
MD5 b7f46db5ee58ee4e4b529fd528471387
BLAKE2b-256 cad3c155aacb02eb4e4d3a174487a58963e6eaedb06ae085c5d0d92ad0d62747

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page