Skip to main content

dask-array

Dask array with query optimization

Motivation

Dask Array is powerful, but requires expertise to drive effectively.

The dask-array project reimplements dask.array, but represents your calculation at a level where it can be optimized intelligently before executed. This allows the project to reorder and replace calculations to provide the same result but with a more efficient path.

Installation

pip install dask-array

Usage

This project looks and feels like Dask Array

import dask_array as da

x = da.ones((1000, 1000), chunks=(100, 100))

y = x + x.T
result = y[:100, :100]

But when you go to compute, your calculation gets rewritten to be more efficient. This is apparent if you look at the query structure of the underlying array.

>>> result.pprint()
Operation             Shape    Bytes   Chunks
  Getitem          (100, 100)   78 kiB  100×100
   Add          (1000, 1000)  7.6 MiB  100×100
     Ones       (1000, 1000)  7.6 MiB  100×100
     Transpose  (1000, 1000)  7.6 MiB  100×100
       Ones     (1000, 1000)  7.6 MiB  100×100

This calculation starts from a large array expression, then takes a small slice. It is more efficient to push that slice into the expression before building the task graph.

The optimize function rewrites things automatically.

>>> result.optimize().pprint()
Operation            Shape   Bytes   Chunks
  FusedBlockwise  (100, 100)  78 kiB  100×100
   Ones          (100, 100)  78 kiB  100×100

You don't need to call optimize though. Dask compute/persist machinery will do this for you. You just need to change your import.

# import dask.array as da
import dask_array as da

Native Frisky acceleration

The normal Dask scheduler path is pure Python and works without a Rust toolchain. On platforms with a native wheel, dask-array also includes a Rust accelerator that Frisky can use to submit compact task records instead of materializing large Python task graphs. This is automatic when computing with a Frisky scheduler; otherwise computations stay on the standard Dask path.

Xarray

Dask-array can replace the default dask.array by registering itself as an Xarray chunk manager.

from dask_array.xarray import register
register()

After this all xarray calculations will benefit from query optimization.

Call register() before you create any chunked arrays. It is the only thing that turns this on: installing or importing dask-array leaves xarray alone, so that adding this package to an environment cannot change how other libraries that use xarray and dask behave.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_array-0.7.1.tar.gz (689.2 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dask_array-0.7.1-py3-none-any.whl (442.8 kB view details)

Uploaded Python 3

dask_array-0.7.1-cp310-abi3-win_amd64.whl (1.0 MB view details)

Uploaded CPython 3.10+Windows x86-64

dask_array-0.7.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

dask_array-0.7.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

dask_array-0.7.1-cp310-abi3-macosx_11_0_arm64.whl (1.1 MB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file dask_array-0.7.1.tar.gz.

File metadata

  • Download URL: dask_array-0.7.1.tar.gz
  • Upload date:
  • Size: 689.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dask_array-0.7.1.tar.gz
Algorithm Hash digest
SHA256 d4b3c1eb97e4b3630f2385922977534ff2193e627b511ae04f08f60798c88959
MD5 1f2a76341b45425a832ee45dac564a2f
BLAKE2b-256 18d84befdcb3840edbd7900ca3234a2b7699d0d001181e746b9f45f090638a1c

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.1.tar.gz:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.7.1-py3-none-any.whl.

File metadata

  • Download URL: dask_array-0.7.1-py3-none-any.whl
  • Upload date:
  • Size: 442.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dask_array-0.7.1-py3-none-any.whl
Algorithm Hash digest
SHA256 6b6be01d3783bca71941458e5a213485bd61d0a95f229dfbc53c3fa262496034
MD5 9af44dc078d131bf888f0e32db768251
BLAKE2b-256 00ee92e8170d26a5594b3d4fc388955b9c2e401cf7ebbd8e8582b0e0bbabed89

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.1-py3-none-any.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.7.1-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: dask_array-0.7.1-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 1.0 MB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for dask_array-0.7.1-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 dc3d79caeb443d5cf1935409003225e63d5caafdb41aa5f61a28b85a867a7d68
MD5 f40da6176fa377b5bf4d454aab41af58
BLAKE2b-256 4fde04d2280fe297374f1da236c69790a2471555a4ac6b9dc364c032f6d8d79a

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.1-cp310-abi3-win_amd64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.7.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for dask_array-0.7.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 127a66b47689b9ae870c61831f2d993b0a2fc28b3980164f8e2a0fb47fffde72
MD5 265e7d7c7d14c23ded4a0c4c23de815d
BLAKE2b-256 01ff94041375ca7f283e021b63dfe50c2d2580b425dc31a13db5c147a6fb5483

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.7.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for dask_array-0.7.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 17ad5502e9e656c4326486846a181940854162c21dfbd2771399ab6c6280e7b6
MD5 42a534ce1f3839d1a1e3cfb9b308ef79
BLAKE2b-256 ac418e660bd98950fc38c82ddbb1c47f08d05ce79bd59a1406db96bf9dcecfbf

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.7.1-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dask_array-0.7.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 118fb5d92fa82e808dabf00ad01fb3196fabd3bd90de6546282787a989aca5c4
MD5 744a8384650c8691a9f88585c1958481
BLAKE2b-256 377803746fbfabef5ae8695de6d5af9a5823ecde59006c52ac15a45788bdc911

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.7.1-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page