flox
This project explores strategies for fast GroupBy reductions with dask.array. It used to be called dask_groupby
It was motivated by
- Dask Dataframe GroupBy blogpost
- numpy_groupies in Xarray issue
(See a presentation about this package, from the Pangeo Showcase).
Acknowledgements
This work was funded in part by
- NASA-ACCESS 80NSSC18M0156 "Community tools for analysis of NASA Earth Observing System Data in the Cloud" (PI J. Hamman, NCAR),
- NASA-OSTFL 80NSSC22K0345 "Enhancing analysis of NASA data with the open-source Python Xarray Library" (PIs Scott Henderson, University of Washington; Deepak Cherian, NCAR; Jessica Scheick, University of New Hampshire), and
- NCAR's Earth System Data Science Initiative.
It was motivated by very very many discussions in the Pangeo community.
API
There are two main functions
flox.groupby_reduce(dask_array, by_dask_array, "mean")"pure" dask array interfaceflox.xarray.xarray_reduce(xarray_object, by_dataarray, "mean")"pure" xarray interface; though work is ongoing to integrate this package in xarray.
Implementation
See the documentation for details on the implementation.
Custom reductions
flox implements all common reductions provided by numpy_groupies in aggregations.py.
It also allows you to specify a custom Aggregation (again inspired by dask.dataframe),
though this might not be fully functional at the moment. See aggregations.py for examples.
mean = Aggregation(
# name used for dask tasks
name="mean",
# operation to use for pure-numpy inputs
numpy="mean",
# blockwise reduction
chunk=("sum", "count"),
# combine intermediate results: sum the sums, sum the counts
combine=("sum", "sum"),
# generate final result as sum / count
finalize=lambda sum_, count: sum_ / count,
# Used when "reindexing" at combine-time
fill_value=0,
# Used when any member of `expected_groups` is not found
final_fill_value=np.nan,
)
1 maintainer
Release files for flox 0.11.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| flox-0.11.2.tar.gz | 956.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| flox-0.11.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size:1.0 MB
Release files / flox-0.11.2.tar.gz
| Download URL | flox-0.11.2.tar.gz |
|---|---|
| Size | 956.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ca48d391e4a06cdf1c24368bf73114191851149611a5b318d82436fa713efdf0
|
|
BLAKE2b-256 checksum How to use checksums |
61fd94b6e6dba647c062434f21b40efe800ca3fed2292856f82c816a8a626b97
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.10.6 {"installer":{"name":"uv","version":"0.10.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / flox-0.11.2-py3-none-any.whl
| Download URL | flox-0.11.2-py3-none-any.whl |
|---|---|
| Size | 89.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
d3932bfe9e26729dd7e471c4d448abeaee7a62272621f584de0e704b3c0665cb
|
|
BLAKE2b-256 checksum How to use checksums |
de7b4ce04d96cb4538ab3eab6bbcc06e624bbbc3661011bfab0b536f4fb026ab
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.10.6 {"installer":{"name":"uv","version":"0.10.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
