Skip to main content

Distributed execution utility.

Project description

Distributed Executor

GitHub License Python Version

A cross-platform reference implementation for processing shell commands over a distributed, asynchronous queue. dist_exe is a single producer (server) many consumer (client) system. It functions similar to GNU Parallel but scales out to many servers and even across operating systems.

This tool uses only standard library methods so there are no dependencies. It has been tested on Linux, macOS, and Windows 10 in Python 3.7 environments. The server and clients don't even need to be using the same platform.


pip install dist_exe [--upgrade] [--user]


The server role reads from standard input and publishes to a queue. If you don't have something actively creating work a good approach is to create a MANIFEST file and use tail -f to wait for work to arrive.

user@host-01: ~ $ touch MANIFEST
user@host-01: ~ $ tail -f MANIFEST | dist_exe server -H -p 5050 -k SECRET

The client role connects to the remote queue and pulls items off, executing them using the local shell. You can specify a template for the command line invocation, otherwise the items are take as being the entire invocation.

user@host-02: ~ $ dist_exe client -H host-01 -p 5050 -k SECRET -t 'echo {}'

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for dist-exe, version 0.0.6
Filename, size File type Python version Upload date Hashes
Filename, size dist_exe-0.0.6.tar.gz (5.3 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page