Skip to main content

A library for running shell pipelines using shell-like syntax

Project description

This is a beta release. The library may be changed significantly. Comments are welcome.


  • Enables piping infinite streams through shell pipelines in Python
  • Represents a shell command as an ordinary function Iterable -> Iterable
  • Allows mixing shell commands and Python functions in a single pipeline
  • Uses standard interprocessing modules subprocess, threading
  • Allows doing things marked as red warning boxes at the subprocess help page
  • 0.2 KLOC, tests included


Basic Usage

Get the iterable of files in /path/to/dir:

>>> from iterpipes import linecmd, run, strip
>>> files = run(linecmd('ls {}', '/path/to/dir') | strip('\n'))
>>> list(files)[:3]
[u'bin', u'boot', u'dev']

Pipe 100 000 lines through wc -l, join the resulting iterable into a single string and convert it to an int:

>>> from iterpipes import cmd, join
>>> wc = cmd('wc -l') | strip() | join | int
>>> numbers = ('%d\n' % i for i in xrange(100000))
>>> wc(numbers)     # or run(wc, numbers)

Delete /path/to/dir and all the files under it, get the return code or check for exceptions:

>>> from iterpipes import call, check_call
>>> call(cmd('rm -fr {}', '/path/to/dir'))
>>> check_call(cmd('rm -fr {}', '/path/to/dir'))

Total lines in *.py files under /path/to/dir, use safe shell parameters formatting:

>>> total = cmd(
...     'find {} -name {} -print0 | xargs -0 wc -l | tail -1 | awk {}',
...     '/path/to/dir', '\*.py', '{print $1}')
>>> run(total | strip() | join | int)

Load an Atom feed of the iterpipes source code repository using curl:

>>> from iterpipes import bincmd
>>> from xml.etree import ElementTree as etree
>>> url = ''
>>> e = run(bincmd('curl -s {}', url) | join | etree.fromstring)
>>> e.tag


Clone the iterpipes Mercurial repository if you are interested in following the library. There is also a Git clone at GitHub.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for iterpipes, version 0.3
Filename, size File type Python version Upload date Hashes
Filename, size iterpipes-0.3.tar.gz (5.0 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page