An async-compatible library for composing Python shell pipelines
Project description
asyncpipe
This is a little module to help you chain together subprocess pipelines, a bit more nicely than python natively lets you. The key element here is the support for asyncio.
OS Compatibility
This will only work on versions of python that support async/await syntax I've only tested this on Linux. I don't see any reason why it wouldn't work on OSX/BSD, but it's possible. I doubt this works on Windows at all.
Python compatibility
This requires 'async' and 'await' keywords, so Python 3.5+ only.
Motivation
When you use a PIPE
for stdout in the native asyncio.subprocess module, you
don't get back a value that can be passed to another Popen like you do in the
subprocess module. Instead you get a wrapper. Sometimes you can resolve that by
just using the 'shell' execution model, but sometimes that's not a great fit.
The solution is to do an error-prone set of operations with os.pipe and linking up various subprocesses stdin/stdout. This gets especially ugly when you've got a chain of 3+ shell commands.
Use
Synchronous
The synchronous form is pretty straightforward:
import asyncpipe
pipe = asyncpipe.PipeBuilder('ls')
# you can chain it
pipe.chain('grep', '^S')
# or use the 'or' operator
pipe | 'wc -l'
results = pipe.call()
matches = int(results[-1].stdout.strip())
All of those return the underlying object, and the starting arguments are optional, so this is equivalent:
import asyncpipe
pipe = asyncpipe.PipeBuilder().chain('ls') | 'grep ^s' | ['wc' '-l']
results = pipe.call()
matches = int(results[-1].stdout.strip())
Asyncronous
Async code is similar, except you need to pass the event loop as the 'loop'
value to the PipeBuilder, or set it any time before calling call_async()
:
import asyncpipe
import asyncio
loop = asyncio.new_event_loop()
# if you don't do this, you will get errors from asyncio
asyncio.set_event_loop(loop)
pipe = asyncpipe.PipeBuilder(loop=loop)
pipe.chain('ls') | 'grep ^s' | ['wc' '-l']
results = loop.run_until_complete(pipe.call_async())
matches = int(results[-1].stdout.strip())
If you were doing more in your event loop, you would of course just
await pipe.call_async()
!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for asyncpipe-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ae5a3fd1b644cc05c84447d91a80687ff177cafece31de1749246927602a44c7 |
|
MD5 | e3f95c2e3cfe7e958995345f305f406d |
|
BLAKE2b-256 | 55cfb4311680fe02eaf79caa02b58bc0254169e9761c146148bc1cbbe84f4eae |