Skip to main content

dask-array

Dask array with query optimization

Motivation

Dask Array is powerful, but requires expertise to drive effectively.

The dask-array project reimplements dask.array, but represents your calculation at a level where it can be optimized intelligently before executed. This allows the project to reorder and replace calculations to provide the same result but with a more efficient path.

Installation

pip install dask-array

Usage

This project looks and feels like Dask Array

import dask_array as da

x = da.ones((1000, 1000), chunks=(100, 100))

y = x + x.T
result = y[:100, :100]

But when you go to compute, your calculation gets rewritten to be more efficient. This is apparent if you look at the query structure of the underlying array.

>>> result.pprint()
Operation             Shape    Bytes   Chunks
  Getitem          (100, 100)   78 kiB  100×100
   Add          (1000, 1000)  7.6 MiB  100×100
     Ones       (1000, 1000)  7.6 MiB  100×100
     Transpose  (1000, 1000)  7.6 MiB  100×100
       Ones     (1000, 1000)  7.6 MiB  100×100

This calculation starts from a large array expression, then takes a small slice. It is more efficient to push that slice into the expression before building the task graph.

The optimize function rewrites things automatically.

>>> result.optimize().pprint()
Operation            Shape   Bytes   Chunks
  FusedBlockwise  (100, 100)  78 kiB  100×100
   Ones          (100, 100)  78 kiB  100×100

You don't need to call optimize though. Dask compute/persist machinery will do this for you. You just need to change your import.

# import dask.array as da
import dask_array as da

Native Frisky acceleration

The normal Dask scheduler path is pure Python and works without a Rust toolchain. On platforms with a native wheel, dask-array also includes a Rust accelerator that Frisky can use to submit compact task records instead of materializing large Python task graphs. This is automatic when computing with a Frisky scheduler; otherwise computations stay on the standard Dask path.

Xarray

Dask-array can replace the default dask.array by registering itself as an Xarray chunk manager.

from dask_array.xarray import register
register()

After this all xarray calculations will benefit from query optimization.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_array-0.7.0.tar.gz (688.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dask_array-0.7.0-py3-none-any.whl (443.2 kB view details)

Uploaded Python 3

dask_array-0.7.0-cp310-abi3-win_amd64.whl (1.0 MB view details)

Uploaded CPython 3.10+Windows x86-64

dask_array-0.7.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

dask_array-0.7.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

dask_array-0.7.0-cp310-abi3-macosx_11_0_arm64.whl (1.1 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file dask_array-0.7.0.tar.gz.

File metadata

  • Download URL: dask_array-0.7.0.tar.gz
  • Upload date:
  • Size: 688.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dask_array-0.7.0.tar.gz
Algorithm Hash digest
SHA256 35560a9321a8180bd70dfcb422754d236b4875be4caac3c70652629d99f45eb0
MD5 bdccf9d5a29f518c9a111af5f8edc4ef
BLAKE2b-256 802c80e9fb53edd8a49df0fd20f5a55ef8c0732b2540b56ecf074a215ff8508a

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.0.tar.gz:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.7.0-py3-none-any.whl.

File metadata

  • Download URL: dask_array-0.7.0-py3-none-any.whl
  • Upload date:
  • Size: 443.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dask_array-0.7.0-py3-none-any.whl
Algorithm Hash digest
SHA256 c225ae563ffff2f194e9851bfdc58010095e637c1ff58a966abe9dabb52d9b5f
MD5 a3461d23ba4d01b18c66551ea8e212df
BLAKE2b-256 3fc72421030af6d91854359dcd6a99ae4e11d86283fb075394ebdd8ec695323b

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.0-py3-none-any.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.7.0-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: dask_array-0.7.0-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dask_array-0.7.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 5ecb2dad4f248aab0ece5475462805180e34dd9cde51cf06dc977fbe18815c59
MD5 7498d66ca637a02bac3d9718482189ab
BLAKE2b-256 ef5fdcbde5327daa2a7a4d28dba6262f7c8ac50dde37c90335daa95cb23192ba

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.0-cp310-abi3-win_amd64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.7.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for dask_array-0.7.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 908223ded300026acf66214fe15613f9b71b8b792495f1f3c7d5955f2eaf5234
MD5 914efd4699ee284850f448a3d9572d4d
BLAKE2b-256 9b29db8f313ff5be99b929e99fe927612fa4c4c79a180b3cb6c6048060a8c2be

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.7.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for dask_array-0.7.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 2701b02deac9781c45c73d035c8261a1910bfdcf0a04f6839aec13cc2ab84110
MD5 14927b362c934ab4812e5eb6e3eb18fa
BLAKE2b-256 98a03abf4a082f1feaefb841eabced3845e4cb0d04ad64daa94f8140c03a0e35

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.7.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dask_array-0.7.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 0ac8e5ac79304af8a0207a2e9e2874744db579c191f8291ae960eb78e1b84da7
MD5 cd06f27d43450de9d21f075a9feb51d0
BLAKE2b-256 ce4a556665612ba43f69d81415405c185cfb19e895715d9982e08eff05f157a5

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page