Skip to main content

dask-array

Dask array with query optimization

Motivation

Dask Array is powerful, but requires expertise to drive effectively.

The dask-array project reimplements dask.array, but represents your calculation at a level where it can be optimized intelligently before executed. This allows the project to reorder and replace calculations to provide the same result but with a more efficient path.

Installation

pip install dask-array

Usage

This project looks and feels like Dask Array

import dask_array as da

x = da.ones((1000, 1000), chunks=(100, 100))

y = x + x.T
result = y[:100, :100]

But when you go to compute, your calculation gets rewritten to be more efficient. This is apparent if you look at the query structure of the underlying array.

>>> result.pprint()
Operation             Shape    Bytes   Chunks
  Getitem          (100, 100)   78 kiB  100×100
   Add          (1000, 1000)  7.6 MiB  100×100
     Ones       (1000, 1000)  7.6 MiB  100×100
     Transpose  (1000, 1000)  7.6 MiB  100×100
       Ones     (1000, 1000)  7.6 MiB  100×100

This calculation starts from a large array expression, then takes a small slice. It is more efficient to push that slice into the expression before building the task graph.

The optimize function rewrites things automatically.

>>> result.optimize().pprint()
Operation            Shape   Bytes   Chunks
  FusedBlockwise  (100, 100)  78 kiB  100×100
   Ones          (100, 100)  78 kiB  100×100

You don't need to call optimize though. Dask compute/persist machinery will do this for you. You just need to change your import.

# import dask.array as da
import dask_array as da

Native Frisky acceleration

The normal Dask scheduler path is pure Python and works without a Rust toolchain. On platforms with a native wheel, dask-array also includes a Rust accelerator that Frisky can use to submit compact task records instead of materializing large Python task graphs. This is automatic when computing with a Frisky scheduler; otherwise computations stay on the standard Dask path.

Xarray

Dask-array can replace the default dask.array by registering itself as an Xarray chunk manager.

from dask_array.xarray import register
register()

After this all xarray calculations will benefit from query optimization.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_array-0.3.0.tar.gz (498.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dask_array-0.3.0-py3-none-any.whl (359.2 kB view details)

Uploaded Python 3

dask_array-0.3.0-cp310-abi3-win_amd64.whl (763.6 kB view details)

Uploaded CPython 3.10+Windows x86-64

dask_array-0.3.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (907.8 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

dask_array-0.3.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (901.0 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

dask_array-0.3.0-cp310-abi3-macosx_11_0_arm64.whl (861.9 kB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file dask_array-0.3.0.tar.gz.

File metadata

  • Download URL: dask_array-0.3.0.tar.gz
  • Upload date:
  • Size: 498.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.3.0.tar.gz
Algorithm Hash digest
SHA256 3835e34e81575a09dfcb1cfec9d33f56e29a1ea0431aed23849212eb711d095c
MD5 013fb221eafb0bf951808c1d01dbc95e
BLAKE2b-256 ee333d19a32c47513a63a9d8a8e8e84458f8f57ea85667b5b4babbe97597eeac

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.3.0.tar.gz:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.3.0-py3-none-any.whl.

File metadata

  • Download URL: dask_array-0.3.0-py3-none-any.whl
  • Upload date:
  • Size: 359.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 4acd8df03576ce2255476b1a80494328acf29630783cf8daa05f4860fa3d4c67
MD5 405aa68492829d8d6c2e548a633b2f80
BLAKE2b-256 4658986909ac2760f89134f5035461fd5805e2e872b439f02e00ec760f75e5d2

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.3.0-py3-none-any.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.3.0-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: dask_array-0.3.0-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 763.6 kB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.3.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 43578c49bb61dcf10f1978de0bdf546e0dce57bb59c7409354245d005416f68f
MD5 9e88fdc5415f9084a4434ca05ca66bff
BLAKE2b-256 4428c849824604105a2d5118c81dc5ef1a38273c1f31321c3430b82ff09d1764

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.3.0-cp310-abi3-win_amd64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.3.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for dask_array-0.3.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e127e4e396b54c59c31e9166aa56bb570289e03835eae3bc8eaf0ab567ccfd88
MD5 07a982e5baaea0798457269b3d5c2a2b
BLAKE2b-256 e7e8d57f278f245420fbab372c25abe703a818aca98a322e9eee3c35bb5daccb

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.3.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.3.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for dask_array-0.3.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 4135d6457e09473a3a6b1f53a4821b84dd0a68c3d6235145245a7b45ab52ee0f
MD5 18e0425ebaccabb08942647887f2864b
BLAKE2b-256 0eca41574c739f130aee3796e33fea249d3011df7cae3f0580b102313243c5d6

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.3.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.3.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dask_array-0.3.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 10e7a309bbb494d80eea23dc927ace295df8f1b5199ed6ed2715da1ccc41501c
MD5 d4389d4d06a3cd051c6409b1f6846fda
BLAKE2b-256 0b544ed1180c7c9e1016da1ae2d36d5a5637372df1588072a51a7a6dacad55aa

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.3.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page