Fast simple 1D and 2D histograms
Project description
About
Sometimes you just want to compute simple 1D or 2D histograms with regular bins. Fast. No nonsense. Numpy’s histogram functions are versatile, and can handle for example nonregular binning, but this versatility comes at the expense of performance.
The fasthistogram minipackage aims to provide simple and fast histogram functions for regular bins that don’t compromise on performance. It doesn’t do anything complicated  it just implements a simple histogram algorithm in C and keeps it simple. The aim is to have functions that are fast but also robust and reliable. The result is a 1D histogram function here that is 715x faster than numpy.histogram, and a 2D histogram function that is 2025x faster than numpy.histogram2d.
To install:
pip install fasthistogram
or if you use conda you can instead do:
conda install c condaforge fasthistogram
The fast_histogram module then provides two functions: histogram1d and histogram2d:
from fast_histogram import histogram1d, histogram2d
Example
Here’s an example of binning 10 million points into a regular 2D histogram:
In [1]: import numpy as np
In [2]: x = np.random.random(10_000_000)
In [3]: y = np.random.random(10_000_000)
In [4]: %timeit _ = np.histogram2d(x, y, range=[[1, 2], [2, 4]], bins=30)
935 ms ± 58.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [5]: from fast_histogram import histogram2d
In [6]: %timeit _ = histogram2d(x, y, range=[[1, 2], [2, 4]], bins=30)
40.2 ms ± 624 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
(note that 10_000_000 is possible in Python 3.6 syntax, use 10000000 instead in previous versions)
The version here is over 20 times faster! The following plot shows the speedup as a function of array size for the bin parameters shown above:
as well as results for the 1D case, also with 30 bins. The speedup for the 2D case is consistently between 2025x, and for the 1D case goes from 15x for small arrays to around 7x for large arrays.
Q&A
Why don’t the histogram functions return the edges?
Computing and returning the edges may seem trivial but it can slow things down by a factor of a few when computing histograms of 10^5 or fewer elements, so not returning the edges is a deliberate decision related to performance. You can easily compute the edges yourself if needed though, using numpy.linspace.
Doesn’t package X already do this, but better?
This may very well be the case! If this duplicates another package, or if it is possible to use Numpy in a smarter way to get the same performance gains, please open an issue and I’ll consider deprecating this package :)
One package that does include fast histogram functions (including in ndimensions) and can compute other statistics is vaex, so take a look there if you need more advanced functionality!
Are the 2D histograms not transposed compared to what they should be?
There is technically no ‘right’ and ‘wrong’ orientation  here we adopt the convention which gives results consistent with Numpy, so:
numpy.histogram2d(x, y, range=[[xmin, xmax], [ymin, ymax]], bins=[nx, ny])
should give the same result as:
fast_histogram.histogram2d(x, y, range=[[xmin, xmax], [ymin, ymax]], bins=[nx, ny])
Why not contribute this to Numpy directly?
As mentioned above, the Numpy functions are much more versatile, so they could not be replaced by the ones here. One option would be to check in Numpy’s functions for cases that are simple and dispatch to functions such as the ones here, or add dedicated functions for regular binning. I hope we can get this in Numpy in some form or another eventually, but for now, the aim is to have this available to packages that need to support a range of Numpy versions.
Why not use Cython?
I originally implemented this in Cython, but found that I could get a 50% performance improvement by going straight to a C extension.
What about using Numba?
I specifically want to keep this package as easy as possible to install, and while Numba is a great package, it is not trivial to install outside of Anaconda.
Could this be parallelized?
This may benefit from parallelization under certain circumstances. The easiest solution might be to use OpenMP, but this won’t work on all platforms, so it would need to be made optional.
Couldn’t you make it faster by using the GPU?
Almost certainly, though the aim here is to have an easily installable and portable package, and introducing GPUs is going to affect both of these.
Why make a package specifically for this? This is a tiny amount of functionality
Packages that need this could simply bundle their own C extension or Cython code to do this, but the main motivation for releasing this as a minipackage is to avoid making purePython packages into packages that require compilation just because of the need to compute fast histograms.
Can I contribute?
Yes please! This is not meant to be a finished package, and I welcome pull request to improve things.
Project details
Release history Release notifications  RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for fast_histogram0.12pp39pypy39_pp73manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  544e91031e9ddcbe6a802b82fab5f9c51488b36012ae172e343f1a64c1babc39 

MD5  fde73f7bda9a55482bab18fca3503d8d 

BLAKE2b256  54032d4edd2028e596b244093f551b098ffa8b32f8a9ddd364e2795da87aca2f 
Hashes for fast_histogram0.12pp38pypy38_pp73manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  a2418b285685d0b272f1f6844d812bdcc5cb980e201cab8b4f25f650c6da58fb 

MD5  7611fa8abe43c3e7cd839b94aed86aa4 

BLAKE2b256  92d6fd58ca3d877626741f00f81b5120cf82479204e09bc8aab4f363278cd32b 
Hashes for fast_histogram0.12cp38abi3win_amd64.whl
Algorithm  Hash digest  

SHA256  d82e92e95f05b279a3969152a3ec7f14c9103ac2034b86610d84ed1f970ee950 

MD5  7e6f559012d4bb575152fdafa117e580 

BLAKE2b256  f6b9cf606fd0d759d40a8379ca257f2057f6f32a5d1b7f978a80f167b87748e9 
Hashes for fast_histogram0.12cp38abi3win32.whl
Algorithm  Hash digest  

SHA256  d634ac4f7b82bb63d8c00ddfbd14873a7ef7a1ee9a9111022713b4ac45542c29 

MD5  db436375702eda91e5a71d6611ce137e 

BLAKE2b256  2fa57e73fae946ba0e245931289ea126f9da4c7fc5cfc670830227edf1b601cf 
Hashes for fast_histogram0.12cp38abi3musllinux_1_1_x86_64.whl
Algorithm  Hash digest  

SHA256  36381627ad8a7ecb44aa99aa621e615d8a19e5edd6da0fb4e9c6b4f80f6c8b6c 

MD5  20373f73aa44bc495220e816512bc365 

BLAKE2b256  718e20e19619570868d369f183a8b0c2f63765588808050af181fcb3307bac9e 
Hashes for fast_histogram0.12cp38abi3manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm  Hash digest  

SHA256  9323569ab79cb1b27139a06599dc7502906c3e0dd0b9ab09228609248386ab62 

MD5  f0fbe0315f6b34c679db9cdadbada6dc 

BLAKE2b256  95b8784799e21c51143bd08aec33e83f0aa9731ba4430d84facccfe2039aa583 
Hashes for fast_histogram0.12cp38abi3manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm  Hash digest  

SHA256  ada2e6a594ba79076d1f1581d31623caecf4d199577209c8cc78f89628cc3645 

MD5  90a41e46d04f861a097fb104168125eb 

BLAKE2b256  d161e4e7009c2c5f79cd5182353b55c9a16b4b38e808181b698d24b8fe21b03c 
Hashes for fast_histogram0.12cp38abi3macosx_11_0_arm64.whl
Algorithm  Hash digest  

SHA256  98e1d33d2a0db9196ae4561ff456b1f5884ca7934abb446f907a0594c49c3e4d 

MD5  ec279b09eb0087c7f1eaeaaa70867fec 

BLAKE2b256  5cf4f8375f7d68e03ccc4df19a8e235ea0c92ff9d0198eaaa960249b13e9d0e3 
Hashes for fast_histogram0.12cp38abi3macosx_10_9_x86_64.whl
Algorithm  Hash digest  

SHA256  650a6472805761bca7c89fa50076c8cf1cdfed2a898473a05c2b9240b61ae5e5 

MD5  7db6eaa23394a51c81f4409256e94084 

BLAKE2b256  67290cabf05f877c057b98010ac84f1e6d747a89ca0ef072677ceb4d1a1c54d3 