Skip to main content

dask-array

Dask array with query optimization

Motivation

Dask Array is powerful, but requires expertise to drive effectively.

The dask-array project reimplements dask.array, but represents your calculation at a level where it can be optimized intelligently before executed. This allows the project to reorder and replace calculations to provide the same result but with a more efficient path.

Installation

pip install dask-array

Usage

This project looks and feels like Dask Array

import dask_array as da

x = da.ones((1000, 1000), chunks=(100, 100))

y = x + x.T
result = y[:100, :100]

But when you go to compute, your calculation gets rewritten to be more efficient. This is apparent if you look at the query structure of the underlying array.

>>> result.pprint()
Operation             Shape    Bytes   Chunks
  Getitem          (100, 100)   78 kiB  100×100
   Add          (1000, 1000)  7.6 MiB  100×100
     Ones       (1000, 1000)  7.6 MiB  100×100
     Transpose  (1000, 1000)  7.6 MiB  100×100
       Ones     (1000, 1000)  7.6 MiB  100×100

This calculation starts from a large array expression, then takes a small slice. It is more efficient to push that slice into the expression before building the task graph.

The optimize function rewrites things automatically.

>>> result.optimize().pprint()
Operation            Shape   Bytes   Chunks
  FusedBlockwise  (100, 100)  78 kiB  100×100
   Ones          (100, 100)  78 kiB  100×100

You don't need to call optimize though. Dask compute/persist machinery will do this for you. You just need to change your import.

# import dask.array as da
import dask_array as da

Native Frisky acceleration

The normal Dask scheduler path is pure Python and works without a Rust toolchain. On platforms with a native wheel, dask-array also includes a Rust accelerator that Frisky can use to submit compact task records instead of materializing large Python task graphs. This is automatic when computing with a Frisky scheduler; otherwise computations stay on the standard Dask path.

Xarray

Dask-array can replace the default dask.array by registering itself as an Xarray chunk manager.

from dask_array.xarray import register
register()

After this all xarray calculations will benefit from query optimization.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_array-0.4.0.tar.gz (512.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dask_array-0.4.0-py3-none-any.whl (366.3 kB view details)

Uploaded Python 3

dask_array-0.4.0-cp310-abi3-win_amd64.whl (817.7 kB view details)

Uploaded CPython 3.10+Windows x86-64

dask_array-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (961.1 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

dask_array-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (948.9 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

dask_array-0.4.0-cp310-abi3-macosx_11_0_arm64.whl (911.1 kB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file dask_array-0.4.0.tar.gz.

File metadata

  • Download URL: dask_array-0.4.0.tar.gz
  • Upload date:
  • Size: 512.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.0.tar.gz
Algorithm Hash digest
SHA256 396f372fe68f4a5f572e263b45835dbbc6fcfaa8578a12cf63d209e865be4e60
MD5 2c55d8b6becd24f56c2fe0fdcf5995e0
BLAKE2b-256 1dafefa2e9d7226c7c41be7c4785cd03df48045c80aa46344eca888353b1308d

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.0.tar.gz:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.0-py3-none-any.whl.

File metadata

  • Download URL: dask_array-0.4.0-py3-none-any.whl
  • Upload date:
  • Size: 366.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.0-py3-none-any.whl
Algorithm Hash digest
SHA256 9f67c496c1170f694faf7eea368b30d0986c573b2a7b362141b1154345e5a01b
MD5 6351bdfd58e61e58910ed52a63b37a31
BLAKE2b-256 a5a23f4a6f7eae462307e775905dcd0fa9a50cc41287c12920c2f00a7db0789f

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.0-py3-none-any.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.0-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: dask_array-0.4.0-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 817.7 kB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 97551af94528d1316b637e895f27454f3287dc6f2c3cc124047bc353b10d66fd
MD5 b81c84a25bf05d25d1e546880d5fbaa1
BLAKE2b-256 7e9e15c0a8e047788ec88b5be11260a500fdffb59a9bcb63c47baed82224d490

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.0-cp310-abi3-win_amd64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 48257b4028243ec82581b9638e84dc5c548918c657e7081b9fe366158ad28f0a
MD5 951e622609a62d4600df8d75bd0b06e4
BLAKE2b-256 6887d2834b43f64523efcb9b7837b00e4d13c1acb016f58b8e7e9b02473122bb

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 737b7b6ffe23b5a93fffc4fc557d1e7fbe4c5c593461e0a532837dc116d1a9eb
MD5 78a7e727ddc8e89ba5c0506aa763cbe7
BLAKE2b-256 37eb9e01b7b8df21d734f99a9bbe39744fa276318f5e782e547c85815fc891a0

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 181ec696c9e3edae06b7f79c43bb3f5a0e714143f90d771a308344b06ecf0afb
MD5 caf6aa07b09240f745548063b3b859c5
BLAKE2b-256 58fd277eef0517344195acc346cd1829bb2b4459838b8d81c0a259e96a8c5391

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page