Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

Lightweight tool for piping subprocess processes to each other

Project description

Vorlauf

What?

Vorlauf is a very minimal tool that helps you to create process pipelines (in the shell sense). It also helps separate out the definition of a Process from the running of a that process with a given stdin, stdout, and stderr.

Why?

Because the subprocess api for chaining processes together is cumbersome and not very well documented.

How?

This library does basically nothing - it’s implemented in fewer than 100 lines. There are two classes available, Process and Pipeline.

Process

The Process class is passed args, cwd, and env, and is executed by calling Process.run with optional stdin, stdout, and stderr parameters.

By removing the stdout, stderr, and stdin from the creation of the Process class, we can create reusable Process definitions:

critical_grepper = Process('grep', 'CRITICAL')

syslog = open('/var/log/syslog', 'r')
apachelog = open('/var/log/httpd/error.log', 'r')

filtered = open('critical.log', 'w')

for logfile in (syslog, apachelog):
  critical_grepper.run(stdin=logfile, stdout=filtered)

Pipeline

The Pipeline class stores a list of Process classes which, when run with Pipeline.run with optional stdin and stdout, pipes the processes together. If present, stdin is passed to the first process, and if present, stdout is passed to the last process.

Example

from vorlauf import Pipeline, Process

pipeline = Pipeline()
pipeline.add(Process('cat', 'foo.txt'))
pipeline.add(Process('grep', 'something'))
pipeline.add(Process('uniq'))

with open('new.txt', 'wb') as fd:
    pipeline.run(stdout=fd)

And because of operator overloading built into the Process and Pipeline classes, this can be simplified as:

from vorlauf import Process as P

pipeline = P('cat', 'foo.txt') | P('grep', 'something') | P('uniq')
with open('new.txt', 'wb') as fd:
    pipeline.run(stdout=fd)

Finally, you could use the Process class to create reusable components:

from vorlauf import Process

class GPG(Process):

    def __init__(self, passphrase):
        super(GPG, self).__init__('gpg', '-c', '--passphrase', passphrase, '-')


class MySQLDump(Process):

    def __init__(self, password, dbname, **kwargs):
        super(MySQLDump, self).__init__(
            'mysqldump', '-u', 'root', '-p{}'.format(password), dbname
        )


with open('mysql.dump', 'wb') as fd:
    pipeline = MySQLDump('loldongs', 'foo') | GPG('supersekrit')
    pipeline.run(stdout=fd)

Tests

Run:

virtualenv venv
venv/bin/pip install -e .
venv/bin/python test.py

Changelog

1.0.0 (2015-09-22)

  • Initial release

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for vorlauf, version 1.0.0
Filename, size File type Python version Upload date Hashes
Filename, size vorlauf-1.0.0.tar.gz (3.2 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page