Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Iterator pipeline wrapper in the spirit of Martin Fowler's Collection Pipeline pattern

Project description

IterPipe

Introduction

This is a wrapper for the iterator functions in the Python Standard Library to make it easier and more readable to chain them together into a pipeline.

It is a way to implement Martin Fowler's Collection Pipeline pattern (https://martinfowler.com/articles/collection-pipeline/) with Python's standard iterator functions.

To illustrate the concept, we will perform the following sequence of steps in each example:

  1. Start with a range iterator,
  2. Then filter to pass only the values larger or equal to 6
  3. Then square each number from step 2, using the map function
  4. Then sum the squares from step 3

Here is the code, using intermediate variables, without chaining.

def filter_func(x):
    # Simple function used in "filter" examples below
    return x >= 6


def squared(x):
    # Simple function to square it's input.  Used in "map" below.
    return x * x

input = range(10)

intermediate_1 = filter(filter_func, input)

# Apply squared function to each value returned by intermediate_1 iterator
intermediate_2 = map(squared, intermediate_1)

output = sum(intermediate_2)

print("Output without chaining: {output}".format(output=output))

Here is the same code, using regular Python syntax chaining. Note that execution happens inside-out, with the last operation (sum) appearing first.

def filter_func(x):
    return x >= 6


def squared(x):
    # Simple function to square it's input.  Used in "map" below.
    return x * x

input = range(10)

output = sum(map(squared, filter(filter_func, input)))

print("Output with regular Python chaining: {output}".format(output=output))

And here is the same code again, using the IterPipe wrapper. Note that it reads in the same order as execution happens.

from IterPipe import IterPipe

def filter_func(x):
    return x >= 6


def squared(x):
    # Simple function to square it's input.  Used in "map" below.
    return x * x

input = range(10)

output = (IterPipe(input) 
          .filter(filter_func)
          .map(squared)
          .sum()
          )

print("Output with IterPipe chaining: {output}".format(output=output))

The IterPipe wrapper supports the following functions that operate on iterators from builtins, itertools and functools.

  • accumulate
  • all
  • any
  • chain
  • combinations
  • combinations_with_replacement
  • compress
  • cycle
  • dict
  • dropwhile
  • enumerate
  • filter
  • filterfalse
  • frozenset
  • groupby
  • islice
  • iterator
  • list
  • map
  • max
  • min
  • next
  • permutations
  • product
  • reduce
  • set
  • sorted
  • starmap
  • sum
  • takewhile
  • tee
  • tuple
  • zip
  • zip_longest

Installation

Works with Python 3.4 or later.

pip install -U IterPipe

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for IterPipe, version 0.0.3
Filename, size File type Python version Upload date Hashes
Filename, size IterPipe-0.0.3-py3-none-any.whl (17.0 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size IterPipe-0.0.3.tar.gz (4.4 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page