Skip to main content

dask-array

Dask array with query optimization

Motivation

Dask Array is powerful, but requires expertise to drive effectively.

The dask-array project reimplements dask.array, but represents your calculation at a level where it can be optimized intelligently before executed. This allows the project to reorder and replace calculations to provide the same result but with a more efficient path.

Installation

pip install dask-array

Usage

This project looks and feels like Dask Array

import dask_array as da

x = da.ones((1000, 1000), chunks=(100, 100))

y = x + x.T
result = y[:100, :100]

But when you go to compute, your calculation gets rewritten to be more efficient. This is apparent if you look at the query structure of the underlying array.

>>> result.pprint()
Operation             Shape    Bytes   Chunks
  Getitem          (100, 100)   78 kiB  100×100
   Add          (1000, 1000)  7.6 MiB  100×100
     Ones       (1000, 1000)  7.6 MiB  100×100
     Transpose  (1000, 1000)  7.6 MiB  100×100
       Ones     (1000, 1000)  7.6 MiB  100×100

This calculation starts from a large array expression, then takes a small slice. It is more efficient to push that slice into the expression before building the task graph.

The optimize function rewrites things automatically.

>>> result.optimize().pprint()
Operation            Shape   Bytes   Chunks
  FusedBlockwise  (100, 100)  78 kiB  100×100
   Ones          (100, 100)  78 kiB  100×100

You don't need to call optimize though. Dask compute/persist machinery will do this for you. You just need to change your import.

# import dask.array as da
import dask_array as da

Native Frisky acceleration

The normal Dask scheduler path is pure Python and works without a Rust toolchain. On platforms with a native wheel, dask-array also includes a Rust accelerator that Frisky can use to submit compact task records instead of materializing large Python task graphs. This is automatic when computing with a Frisky scheduler; otherwise computations stay on the standard Dask path.

Xarray

Dask-array can replace the default dask.array by registering itself as an Xarray chunk manager.

from dask_array.xarray import register
register()

After this all xarray calculations will benefit from query optimization.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_array-0.5.0.tar.gz (557.1 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dask_array-0.5.0-py3-none-any.whl (391.6 kB view details)

Uploaded Python 3

dask_array-0.5.0-cp310-abi3-win_amd64.whl (849.5 kB view details)

Uploaded CPython 3.10+Windows x86-64

dask_array-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (999.0 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

dask_array-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (989.8 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

dask_array-0.5.0-cp310-abi3-macosx_11_0_arm64.whl (953.7 kB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file dask_array-0.5.0.tar.gz.

File metadata

  • Download URL: dask_array-0.5.0.tar.gz
  • Upload date:
  • Size: 557.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.5.0.tar.gz
Algorithm Hash digest
SHA256 127caa856afd765b6d5320ae896c058d7141f02c43939e03adcaea40bff23ac3
MD5 c1ab8ea394836d7ca296258c804494e4
BLAKE2b-256 1f547469206aebd2208c621e5db604050a67ffb322a941e47e9d293475f55541

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.5.0.tar.gz:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.5.0-py3-none-any.whl.

File metadata

  • Download URL: dask_array-0.5.0-py3-none-any.whl
  • Upload date:
  • Size: 391.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.5.0-py3-none-any.whl
Algorithm Hash digest
SHA256 3c4afab937bac825ec4e41faa89aded7d42e43d593eae0658f72e8dd9bd9a422
MD5 d83d856f583e193f10b3f290ae59d2f4
BLAKE2b-256 5592a52b9f61aead81803272ebb023f66817653ce9c9e42e20d1f323d73c12d1

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.5.0-py3-none-any.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.5.0-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: dask_array-0.5.0-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 849.5 kB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.5.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 7c0c98ba4187d9dc25293e535a9fe549807678e24899c90ce826fbc6fb166d4b
MD5 4fa93391a02d54b06de440c81146e871
BLAKE2b-256 7a330ced594611926e7e62118fd4166e89145e3808eab12f4344fb1ffe85a022

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.5.0-cp310-abi3-win_amd64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for dask_array-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 c3d40d32edf3a57979fd43343a5c06c81dcd6d6d44a6bf32a1493e3c06118537
MD5 631932b5022b75d9682f62346451c1ff
BLAKE2b-256 ce1adf2f4e7118ac1419f7bb6136fe0b60ce9080f6ff6031b8f399fa95b5c183

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for dask_array-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 778acafd20991f720d27941a613a1284eb271eeb3b72fb95e8bfe537b86406c5
MD5 33696846b2592a24853f5880602f3a81
BLAKE2b-256 3efcf79dabb84f343543502cc6bea389698eb0158acf4e47a53781a666458556

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.5.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dask_array-0.5.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 8e54efc9e2fd824747dc9c9ffd5e693cfff9134b304647adb442303a8a90da4e
MD5 1c0f6c8dfa1a2f5470df1ee131ce6862
BLAKE2b-256 fe49aef6388fe54cb7a4549283ce6646b9b713102acfbdf9d89e142e118efa0c

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.5.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page