Skip to main content

Run multiple subprocesses asynchronous/in parallel with streamed output/non-blocking reading. Also various tools to replace shell scripts.

Project description

shelljob

A simple way to manage several parallel subprocesses. This provides for asynchronous processes and non-blocking reading of their output.

Parallel Subprocesses

Using the Job system is the quickest approach to just run processes and log their output (by default in files named ‘/tmp/job_ID.log’)

from shelljob import job

jm = job.FileMonitor()
jm.run([
    [ 'ls', '-alR', '/usr/local' ],
    'my_prog',
    'build output input',
])

An array will passed directly to subprocess.Popen, a string is first parsed with shlex.split.

The lower level Group class provides a simple container for more manual job management.

from shelljob import proc

g = proc.Group()
p1 = g.run( [ 'ls', '-al', '/usr/local' ] )
p2 = g.run( [ 'program', 'arg1', 'arg2' ] )

while g.is_pending():
    lines = g.readlines()
    for proc, line in lines:
        sys.stdout.write( "{}:{}".format( proc.pid, line ) )

Simple Subprocess calls

A simplified call function allows timeouts on subprocesses and easy acces to their output.

from shelljob import proc

# capture the output
output = proc.call( 'ls /tmp' )
# this raises a proc.Timeout exception
proc.call( 'sleep 10', timeout = 0.1 )

Find

The ‘find’ funtion is a multi-faceted approach to generating listings of files.

from shelljob import fs

files = fs.find( '/usr/local', name_regex = '.*\\.so' )
print( "\n".join(files) )

Refer to the API docs for all parameters. Just let me know if there is some additional option you need.

Issues

You can use my Launchpad project to submit issues.

Project details


Release history Release notifications

History Node

0.5.6

History Node

0.5.5

History Node

0.4.4

History Node

0.4.3

History Node

0.4.2

History Node

0.4.1

History Node

0.4.0

This version
History Node

0.3.5

History Node

0.3.4

History Node

0.3.2

History Node

0.3.1

History Node

0.3.0

History Node

0.2.0

History Node

0.1.1

History Node

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
shelljob-0.3.5.tar.gz (49.5 kB) Copy SHA256 hash SHA256 Source None Dec 10, 2013

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page