Skip to main content

Establish a connection to the errand-boy deamon to execute commands without the memory overhead incurred by os.fork().

Project description

What does it do?

Uses Python multiprocessing to maintain a pool of worker processes used to execute arbitrary terminal commands.

Why not use subprocess.Popen()?

Under the hood subprocess.Popen() uses os.fork(), which copies the currently running process’ memory before launching the command you want to run. If your process uses a lot of memory, such as a Celery worker using an eventlet pool, this can cause a “Cannot allocate memory” error.

errand-boy still uses subprocess.Popen(), but tries to keep a low memory footprint. Your celery greenthread workers can communicate with it via asynchronous network calls.

Further reading:

# http://stackoverflow.com/a/13329386/241955 # http://stackoverflow.com/a/14942111/241955

Setup

Install:

pip install errand-boy

# optional pip install setproctitle

Usage

Run tests:

cd errand-boy/
python -m unittest discover

Run server:

python -m errand_boy.run

Run client (useful for testing/debugging):

python -m errand_boy.run 'ls -al'

Use the client in your code:

from errand_boy.transports.unixsocket import UNIXSocketTransport

transport = UNIXSocketTransport()

process, stdout, stderr = transport.run_cmd('ls -al')

print process.returncode
print stdout
print stderr

Use a subprocess.Popen-like interface:

from errand_boy.transports.unixsocket import UNIXSocketTransport

transport = UNIXSocketTransport()

process = transport.Popen('ls -al')

stdout, stderr = process.communicate(input='foo')

print process.returncode
print stdout
print stderr

Run load tests:

python -m errand_boy.run --max-accepts=0

pip install Fabric locustio
cd errand-boy/
fab locust_local

Does it work in other languages?

It shouldn’t be too diffcult to write client libraries in other languages. You just need to:

  1. Establish a connection to the server’s socket.
  2. Send the command you wish to execute as a string followed by \r\n\r\n. Then send your input or an empty string followed by \r\n\r\n.
  3. Receive data back from the connection until the server stops sending back data. The server will close the connection when it’s done.

Project details


Release history Release notifications

History Node

0.3.9

History Node

0.3.8

History Node

0.3.7

History Node

0.3.6

History Node

0.3.5

History Node

0.3.4

History Node

0.3.3

History Node

0.3.2

History Node

0.3.1

This version
History Node

0.3.0

History Node

0.1.1

History Node

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
errand-boy-0.3.0.tar.gz (9.0 kB) Copy SHA256 hash SHA256 Source None Aug 12, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page