An async-compatible library for composing Python shell pipelines
This is a little module to help you chain together subprocess pipelines, a bit more nicely than python natively lets you. The key element here is the support for asyncio.
This will only work on versions of python that support async/await syntax I've only tested this on Linux. I don't see any reason why it wouldn't work on OSX/BSD, but it's possible. I doubt this works on Windows at all.
This requires 'async' and 'await' keywords, so Python 3.5+ only.
When you use a
PIPE for stdout in the native asyncio.subprocess module, you
don't get back a value that can be passed to another Popen like you do in the
subprocess module. Instead you get a wrapper. Sometimes you can resolve that by
just using the 'shell' execution model, but sometimes that's not a great fit.
The solution is to do an error-prone set of operations with os.pipe and linking up various subprocesses stdin/stdout. This gets especially ugly when you've got a chain of 3+ shell commands.
The synchronous form is pretty straightforward:
import asyncpipe pipe = asyncpipe.PipeBuilder('ls') # you can chain it pipe.chain('grep', '^S') # or use the 'or' operator pipe | 'wc -l' results = pipe.call() matches = int(results[-1].stdout.strip())
All of those return the underlying object, and the starting arguments are optional, so this is equivalent:
import asyncpipe pipe = asyncpipe.PipeBuilder().chain('ls') | 'grep ^s' | ['wc' '-l'] results = pipe.call() matches = int(results[-1].stdout.strip())
Async code is similar, except you need to pass the event loop as the 'loop'
value to the PipeBuilder, or set it any time before calling
import asyncpipe import asyncio loop = asyncio.new_event_loop() # if you don't do this, you will get errors from asyncio asyncio.set_event_loop(loop) pipe = asyncpipe.PipeBuilder(loop=loop) pipe.chain('ls') | 'grep ^s' | ['wc' '-l'] results = loop.run_until_complete(pipe.call_async()) matches = int(results[-1].stdout.strip())
If you were doing more in your event loop, you would of course just