Pipetools
pipetools enables function composition similar to using Unix pipes.
It allows forward-composition and piping of arbitrary functions - no need to decorate them or do anything extra.
It also packs a bunch of utils that make common operations more convenient and readable.
Source is on github.
Why?
Piping and function composition are some of the most natural operations there are for plenty of programming tasks. Yet Python doesn’t have a built-in way of performing them. That forces you to either deep nesting of function calls or adding extra glue code.
Example
Say you want to create a list of python files in a given directory, ordered by filename length, as a string, each file on one line and also with line numbers:
>>> print(pyfiles_by_length('../pipetools'))
1. ds_builder.py
2. __init__.py
3. compat.py
4. utils.py
5. main.pyAll the ingredients are already there, you just have to glue them together. You might write it like this:
def pyfiles_by_length(directory):
all_files = os.listdir(directory)
py_files = [f for f in all_files if f.endswith('.py')]
sorted_files = sorted(py_files, key=len, reverse=True)
numbered = enumerate(py_files, 1)
rows = ("{0}. {1}".format(i, f) for i, f in numbered)
return '\n'.join(rows)Or perhaps like this:
def pyfiles_by_length(directory):
return '\n'.join('{0}. {1}'.format(*x) for x in enumerate(reversed(sorted(
[f for f in os.listdir(directory) if f.endswith('.py')], key=len)), 1))Or, if you’re a mad scientist, you would probably do it like this:
pyfiles_by_length = lambda d: (reduce('{0}\n{1}'.format,
map(lambda x: '%d. %s' % x, enumerate(reversed(sorted(
filter(lambda f: f.endswith('.py'), os.listdir(d)), key=len))))))But there should be one – and preferably only one – obvious way to do it.
So which one is it? Well, to redeem the situation, pipetools give you yet another possibility!
pyfiles_by_length = (pipe
| os.listdir
| where(X.endswith('.py'))
| sort_by(len).descending
| (enumerate, X, 1)
| foreach("{0}. {1}")
| '\n'.join)Why would I do that, you ask? Comparing to the native Python code, it’s
Easier to read – minimal extra clutter
Easier to understand – one-way data flow from one step to the next, nothing else to keep track of
Easier to change – want more processing? just add a step to the pipeline
Removes some bug opportunities – did you spot the bug in the first example?
Of course it won’t solve all your problems, but a great deal of code can be expressed as a pipeline, giving you the above benefits. Read on to see how it works!
Installation
$ pip install pipetoolsUsage
The pipe
The pipe object can be used to pipe functions together to form new functions, and it works like this:
from pipetools import pipe
f = pipe | a | b | c
# is the same as:
def f(x):
return c(b(a(x)))A real example, sum of odd numbers from 0 to x:
from functools import partial
from pipetools import pipe
odd_sum = pipe | range | partial(filter, lambda x: x % 2) | sum
odd_sum(10) # -> 25Note that the chain up to the sum is lazy.
Automatic partial application in the pipe
As partial application is often useful when piping things together, it is done automatically when the pipe encounters a tuple, so this produces the same result as the previous example:
odd_sum = pipe | range | (filter, lambda x: x % 2) | sumAs of 0.1.9, this is even more powerful, see X-partial.
Built-in tools
Pipetools contain a set of pipe-utils that solve some common tasks. For example there is a shortcut for the filter class from our example, called where():
from pipetools import pipe, where
odd_sum = pipe | range | where(lambda x: x % 2) | sumWell that might be a bit more readable, but not really a huge improvement, but wait!
If a pipe-util is used as first or second item in the pipe (which happens quite often) the pipe at the beginning can be omitted:
odd_sum = range | where(lambda x: x % 2) | sumOK, but what about the ugly lambda?
where(), but also foreach(), sort_by() and other pipe-utils can be quite useful, but require a function as an argument, which can either be a named function – which is OK if it does something complicated – but often it’s something simple, so it’s appropriate to use a lambda. Except Python’s lambdas are quite verbose for simple tasks and the code gets cluttered…
X object to the rescue!
from pipetools import where, X
odd_sum = range | where(X % 2) | sumHow ‘bout that.
Automatic string formatting
Since it doesn’t make sense to compose functions with strings, when a pipe (or a pipe-util) encounters a string, it attempts to use it for (advanced) formatting:
>>> countdown = pipe | (range, 1) | reversed | foreach('{}...') | ' '.join | '{} boom'
>>> countdown(5)
'4... 3... 2... 1... boom'Feeding the pipe
Sometimes it’s useful to create a one-off pipe and immediately run some input through it. And since this is somewhat awkward (and not very readable, especially when the pipe spans multiple lines):
result = (pipe | foo | bar | boo)(some_input)It can also be done using the > operator:
result = some_input > pipe | foo | bar | booBut wait, there is more
Checkout the Maybe pipe, partial application on steroids or automatic data structure creation in the full documentation.
1 maintainer
Metadata
1 maintainer
Release files for pipetools 1.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pipetools-1.1.0.tar.gz | 16.0 kB | Details |
Release files / pipetools-1.1.0.tar.gz
| Download URL | pipetools-1.1.0.tar.gz |
|---|---|
| Size | 16.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
8811d8871a1d555df93f3cd965edf7e8f3de96a4cecb88545664a063f7e5d220
|
|
BLAKE2b-256 checksum How to use checksums |
190488ecb4bdb72760929ebcb48d0ab5016f123d7cfa73564d83841bcc0114b6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/4.6.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.9.5
|
