Skip to main content

Streaming operations on NumPy arrays

Project description


Documentation Build Status PyPI Version Conda-forge Version

npstreams is an open-source Python package for streaming NumPy array operations. The goal is to provide tested routines that operate on streams (or generators) of arrays instead of dense arrays.

Streaming reduction operations (sums, averages, etc.) can be implemented in constant memory, which in turns allows for easy parallelization.

This approach has been a huge boon when working with lots of images; the images are read one-by-one from disk and combined/processed in a streaming fashion.

This package is developed in conjunction with other software projects in the Siwick research group.

Motivating Example

Consider the following snippet to combine 50 images from an iterable source:

import numpy as np

images = np.empty( shape = (2048, 2048, 50) )
for index, im in enumerate(source):
    images[:,:,index] = im

avg = np.average(images, axis = 2)

If the source iterable provided 1000 images, the above routine would not work on most machines. Moreover, what if we want to transform the images one by one before averaging them? What about looking at the average while it is being computed? Let's look at an example:

import numpy as np
from npstreams import iaverage
from scipy.misc import imread

stream = map(imread, list_of_filenames)
averaged = iaverage(stream)

At this point, the generators map and iaverage are 'wired' but will not compute anything until it is requested. We can look at the average evolve:

import matplotlib.pyplot as plt
for avg in average:

We can also use last to get at the final average:

from npstreams import last

total = last(averaged) # average of the entire stream

Streaming Functions

npstreams comes with some streaming functions built-in. Some examples:

  • Numerics : isum, iprod, isub, etc.
  • Statistics : iaverage (weighted mean), ivar (single-pass variance), etc.

More importantly, npstreams gives you all the tools required to build your own streaming function. All routines are documented in the API Reference on


npstreams provides a function for benchmarking common use cases.

To run the benchmark with default parameters, from the interpreter:

from npstreams import benchmark

From a command-line terminal:

python -c 'import npstreams; npstreams.benchmark()'

The results will be printed to the screen.

Future Work

Some of the features I want to implement in this package in the near future:

  • Optimize the CUDA-enabled routines
  • More functions : more streaming functions borrowed from NumPy and SciPy.

API Reference

The API Reference on provides API-level documentation, as well as tutorials.


The only requirement is NumPy. To have access to CUDA-enabled routines, PyCUDA must also be installed. npstreams is available on PyPI; it can be installed with pip.:

python -m pip install npstreams

npstreams can also be installed with the conda package manager, from the conda-forge channel:

conda config --add channels conda-forge
conda install npstreams

To install the latest development version from Github:

python -m pip install git+git://

Tests can be run using the pytest package.


If you find this software useful, please consider citing the following publication:

L. P. René de Cotret, M. R. Otto, M. J. Stern. and B. J. Siwick, An open-source software ecosystem for the interactive exploration of ultrafast electron scattering data, Advanced Structural and Chemical Imaging 4:11 (2018) DOI: 10.1186/s40679-018-0060-y.

Support / Report Issues

All support requests and issue reports should be filed on Github as an issue.


npstreams is made available under the BSD License, same as NumPy. For more details, see LICENSE.txt.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for npstreams, version 1.6.5
Filename, size File type Python version Upload date Hashes
Filename, size npstreams-1.6.5-py3-none-any.whl (40.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size npstreams-1.6.5.tar.gz (35.0 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page