Skip to main content

dask-array

Dask array with query optimization

Motivation

Dask Array is powerful, but requires expertise to drive effectively.

The dask-array project reimplements dask.array, but represents your calculation at a level where it can be optimized intelligently before executed. This allows the project to reorder and replace calculations to provide the same result but with a more efficient path.

Installation

pip install dask-array

Usage

This project looks and feels like Dask Array

import dask_array as da

x = da.ones((1000, 1000), chunks=(100, 100))

y = x + x.T
result = y[:100, :100]

But when you go to compute, your calculation gets rewritten to be more efficient. This is apparent if you look at the query structure of the underlying array.

>>> result.pprint()
Operation             Shape    Bytes   Chunks
  Getitem          (100, 100)   78 kiB  100×100
   Add          (1000, 1000)  7.6 MiB  100×100
     Ones       (1000, 1000)  7.6 MiB  100×100
     Transpose  (1000, 1000)  7.6 MiB  100×100
       Ones     (1000, 1000)  7.6 MiB  100×100

This calculation starts from a large array expression, then takes a small slice. It is more efficient to push that slice into the expression before building the task graph.

The optimize function rewrites things automatically.

>>> result.optimize().pprint()
Operation            Shape   Bytes   Chunks
  FusedBlockwise  (100, 100)  78 kiB  100×100
   Ones          (100, 100)  78 kiB  100×100

You don't need to call optimize though. Dask compute/persist machinery will do this for you. You just need to change your import.

# import dask.array as da
import dask_array as da

Native Frisky acceleration

The normal Dask scheduler path is pure Python and works without a Rust toolchain. On platforms with a native wheel, dask-array also includes a Rust accelerator that Frisky can use to submit compact task records instead of materializing large Python task graphs. This is automatic when computing with a Frisky scheduler; otherwise computations stay on the standard Dask path.

Xarray

Dask-array can replace the default dask.array by registering itself as an Xarray chunk manager.

from dask_array.xarray import register
register()

After this all xarray calculations will benefit from query optimization.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_array-0.4.2.tar.gz (518.9 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dask_array-0.4.2-py3-none-any.whl (369.6 kB view details)

Uploaded Python 3

dask_array-0.4.2-cp310-abi3-win_amd64.whl (824.5 kB view details)

Uploaded CPython 3.10+Windows x86-64

dask_array-0.4.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (967.9 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

dask_array-0.4.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (955.6 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

dask_array-0.4.2-cp310-abi3-macosx_11_0_arm64.whl (917.8 kB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file dask_array-0.4.2.tar.gz.

File metadata

  • Download URL: dask_array-0.4.2.tar.gz
  • Upload date:
  • Size: 518.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.2.tar.gz
Algorithm Hash digest
SHA256 cb574331c01ed3c2991cda0c158c8941123836c146b22fbeeecc879b902fdfc9
MD5 3b52bbafebcdfe74f01fca5971a3b3b2
BLAKE2b-256 aaed808b878be87a33bb6063f4c131b6adef755dee64cf8b877042de11a3d06b

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.2.tar.gz:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.2-py3-none-any.whl.

File metadata

  • Download URL: dask_array-0.4.2-py3-none-any.whl
  • Upload date:
  • Size: 369.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.2-py3-none-any.whl
Algorithm Hash digest
SHA256 2b9f4a6ec17adbaa656a0f5e1c1b755a26edef9be696dbdc8b14026719d7b06f
MD5 476ac7a687c94df013db408e0778631f
BLAKE2b-256 13163aeee8a8e835776c0baa082f867014ee02d9357de520173c5a5dfb4cb4c9

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.2-py3-none-any.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.2-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: dask_array-0.4.2-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 824.5 kB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.2-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 649e93e4e6a310693b2e9cce582a8b050fc83bb1b79bf1121ec8fb33c0f43341
MD5 460d0bf632f125aa59c4e39bfc754dd3
BLAKE2b-256 7e771378bc973a1cef53db9614b74c38add130b0bd91b19edefb01ac80f83aad

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.2-cp310-abi3-win_amd64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 bf11e1ff393900a950a56db6995c50808986620123d3f3c24fe76efd4fc970c2
MD5 44405b6dfeb0083982dde2aa8e894e30
BLAKE2b-256 1ea95044b536096b4a55de98d20bcc3d6cda186a2cef7a805dac5642f01920cd

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.2-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 b9d7b4ae6c0ad839fc5e666260c2075659d14c4a88e07ee61b4fb3b2f8653dd0
MD5 2d25c895d4a41bc264b22340a5cf6cf9
BLAKE2b-256 454b1a1c67a65015f41ff09e3b2403e8e11618b84950896b175603553f65929f

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.2-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.2-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.2-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 69b169bf9f5d6294f158f59691c6747773a88cc623edc333dd5858bde3346bbb
MD5 32d0ecdbbe67ca2fe6b41aa7a3f3ecde
BLAKE2b-256 65bac721f61365d5d2b409ade580a96c6fda531bb8e9aeb2f1ad2757d9650e38

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.2-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page