iterpipes 0.3

A library for running shell pipelines using shell-like syntax

This is a beta release. The library may be changed significantly. Comments are welcome.

Facts

• Enables piping infinite streams through shell pipelines in Python

• Represents a shell command as an ordinary function Iterable -> Iterable

• Allows mixing shell commands and Python functions in a single pipeline

• Uses standard interprocessing modules subprocess, threading

• Allows doing things marked as red warning boxes at the subprocess help page

• 0.2 KLOC, tests included

Documentation

• iterpipes Manual (incomplete)

Basic Usage

Get the iterable of files in /path/to/dir:

>>> from iterpipes import linecmd, run, strip
>>> files = run(linecmd('ls {}', '/path/to/dir') | strip('\n'))
>>> list(files)[:3]
[u'bin', u'boot', u'dev']

Pipe 100 000 lines through wc -l, join the resulting iterable into a single string and convert it to an int:

>>> from iterpipes import cmd, join
>>> wc = cmd('wc -l') | strip() | join | int
>>> numbers = ('%d\n' % i for i in xrange(100000))
>>> wc(numbers)     # or run(wc, numbers)
100000

Delete /path/to/dir and all the files under it, get the return code or check for exceptions:

>>> from iterpipes import call, check_call
>>> call(cmd('rm -fr {}', '/path/to/dir'))
0
>>> check_call(cmd('rm -fr {}', '/path/to/dir'))

Total lines in *.py files under /path/to/dir, use safe shell parameters formatting:

>>> total = cmd(
...     'find {} -name {} -print0 | xargs -0 wc -l | tail -1 | awk {}',
...     '/path/to/dir', '\*.py', '{print $1}')
>>> run(total | strip() | join | int)
315

Load an Atom feed of the iterpipes source code repository using curl:

>>> from iterpipes import bincmd
>>> from xml.etree import ElementTree as etree
>>> url = 'http://bitbucket.org/vlasovskikh/iterpipes/atom/'
>>> e = run(bincmd('curl -s {}', url) | join | etree.fromstring)
>>> e.tag
'{http://www.w3.org/2005/Atom}feed'

Reviews and Discussions

• With iterpipes, python is ready to replace bash for scripting. Really

Download

Clone the iterpipes Mercurial repository if you are interested in following the library. There is also a Git clone at GitHub.