Skip to main content

GroupBy operations for dask.array

Project description

GitHub Workflow CI Status pre-commit.ci status image Documentation Status

PyPI Conda-forge

NASA-80NSSC18M0156 NASA-80NSSC22K0345

flox

This project explores strategies for fast GroupBy reductions with dask.array. It used to be called dask_groupby It was motivated by

  1. Dask Dataframe GroupBy blogpost
  2. numpy_groupies in Xarray issue

(See a presentation about this package, from the Pangeo Showcase).

Acknowledgements

This work was funded in part by

  1. NASA-ACCESS 80NSSC18M0156 "Community tools for analysis of NASA Earth Observing System Data in the Cloud" (PI J. Hamman, NCAR),
  2. NASA-OSTFL 80NSSC22K0345 "Enhancing analysis of NASA data with the open-source Python Xarray Library" (PIs Scott Henderson, University of Washington; Deepak Cherian, NCAR; Jessica Scheick, University of New Hampshire), and
  3. NCAR's Earth System Data Science Initiative.

It was motivated by very very many discussions in the Pangeo community.

API

There are two main functions

  1. flox.groupby_reduce(dask_array, by_dask_array, "mean") "pure" dask array interface
  2. flox.xarray.xarray_reduce(xarray_object, by_dataarray, "mean") "pure" xarray interface; though work is ongoing to integrate this package in xarray.

Implementation

See the documentation for details on the implementation.

Custom reductions

flox implements all common reductions provided by numpy_groupies in aggregations.py. It also allows you to specify a custom Aggregation (again inspired by dask.dataframe), though this might not be fully functional at the moment. See aggregations.py for examples.

    mean = Aggregation(
        # name used for dask tasks
        name="mean",
        # operation to use for pure-numpy inputs
        numpy="mean",
        # blockwise reduction
        chunk=("sum", "count"),
        # combine intermediate results: sum the sums, sum the counts
        combine=("sum", "sum"),
        # generate final result as sum / count
        finalize=lambda sum_, count: sum_ / count,
        # Used when "reindexing" at combine-time
        fill_value=0,
        # Used when any member of `expected_groups` is not found
        final_fill_value=np.nan,
    )

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

flox-0.6.0.tar.gz (378.7 kB view details)

Uploaded Source

Built Distribution

flox-0.6.0-py3-none-any.whl (59.4 kB view details)

Uploaded Python 3

File details

Details for the file flox-0.6.0.tar.gz.

File metadata

  • Download URL: flox-0.6.0.tar.gz
  • Upload date:
  • Size: 378.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.7

File hashes

Hashes for flox-0.6.0.tar.gz
Algorithm Hash digest
SHA256 8dcbda2fb81b4e97c0120769616d5136af88023459fa9e74c567923d35d4b29f
MD5 1e3206c5c5968cca508b5664086c0da3
BLAKE2b-256 d8f89d34436519320fcfe486a252b727c1d61b069fc72d72e180dc0fc88831ae

See more details on using hashes here.

File details

Details for the file flox-0.6.0-py3-none-any.whl.

File metadata

  • Download URL: flox-0.6.0-py3-none-any.whl
  • Upload date:
  • Size: 59.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.7

File hashes

Hashes for flox-0.6.0-py3-none-any.whl
Algorithm Hash digest
SHA256 916b12d775e5420afff92a726b0f8df07a475dce25307070eac18e80feaa7d18
MD5 16262aa6ce5e3d0446225311a09383ce
BLAKE2b-256 041fb1c8d48d4fae61a9b7507d5d8bf04149d5b65d846540e5abcad0d8306065

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page