Skip to main content

dask-array

Dask array with query optimization

Motivation

Dask Array is powerful, but requires expertise to drive effectively.

The dask-array project reimplements dask.array, but represents your calculation at a level where it can be optimized intelligently before executed. This allows the project to reorder and replace calculations to provide the same result but with a more efficient path.

Installation

pip install dask-array

Usage

This project looks and feels like Dask Array

import dask_array as da

x = da.ones((1000, 1000), chunks=(100, 100))

y = x + x.T
result = y[:100, :100]

But when you go to compute, your calculation gets rewritten to be more efficient. This is apparent if you look at the query structure of the underlying array.

>>> result.pprint()
Operation             Shape    Bytes   Chunks
  Getitem          (100, 100)   78 kiB  100×100
   Add          (1000, 1000)  7.6 MiB  100×100
     Ones       (1000, 1000)  7.6 MiB  100×100
     Transpose  (1000, 1000)  7.6 MiB  100×100
       Ones     (1000, 1000)  7.6 MiB  100×100

This calculation starts from a large array expression, then takes a small slice. It is more efficient to push that slice into the expression before building the task graph.

The optimize function rewrites things automatically.

>>> result.optimize().pprint()
Operation            Shape   Bytes   Chunks
  FusedBlockwise  (100, 100)  78 kiB  100×100
   Ones          (100, 100)  78 kiB  100×100

You don't need to call optimize though. Dask compute/persist machinery will do this for you. You just need to change your import.

# import dask.array as da
import dask_array as da

Native Frisky acceleration

The normal Dask scheduler path is pure Python and works without a Rust toolchain. On platforms with a native wheel, dask-array also includes a Rust accelerator that Frisky can use to submit compact task records instead of materializing large Python task graphs. This is automatic when computing with a Frisky scheduler; otherwise computations stay on the standard Dask path.

Xarray

Dask-array can replace the default dask.array by registering itself as an Xarray chunk manager.

from dask_array.xarray import register
register()

After this all xarray calculations will benefit from query optimization.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_array-0.6.0.tar.gz (583.1 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dask_array-0.6.0-py3-none-any.whl (399.9 kB view details)

Uploaded Python 3

dask_array-0.6.0-cp310-abi3-win_amd64.whl (869.6 kB view details)

Uploaded CPython 3.10+Windows x86-64

dask_array-0.6.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

dask_array-0.6.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.0 MB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

dask_array-0.6.0-cp310-abi3-macosx_11_0_arm64.whl (975.8 kB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file dask_array-0.6.0.tar.gz.

File metadata

  • Download URL: dask_array-0.6.0.tar.gz
  • Upload date:
  • Size: 583.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.6.0.tar.gz
Algorithm Hash digest
SHA256 238dc6774fa8550be506c37b09c5965da816737c8e184d5b1365a281d7df8a38
MD5 5a6645dba10ee9921176e2e37080299c
BLAKE2b-256 d4754e2d3fe0e0e174c2afcaaec71c7a1dd0f0d483d99e3d0f23accf020423bd

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.6.0.tar.gz:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.6.0-py3-none-any.whl.

File metadata

  • Download URL: dask_array-0.6.0-py3-none-any.whl
  • Upload date:
  • Size: 399.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.6.0-py3-none-any.whl
Algorithm Hash digest
SHA256 6c2236f2327597eb2467c4de6e7e55ec4f749ff2de134855553aab25d891d262
MD5 91c7faf51af643c8aac5c782a1604017
BLAKE2b-256 38c56ff938c530dfa25a7973449321ee9935467e7c6aebda232e44a7dc6a8eed

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.6.0-py3-none-any.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.6.0-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: dask_array-0.6.0-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 869.6 kB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.6.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 55fe4898318f2a0ac3dd073a632110f01033d13db8f7692ed779c1c0c3494597
MD5 8d1ed7565cc0053621a893e4590ff5b2
BLAKE2b-256 3ebba93ef1547e635b021feded3e38e62257fcd83cdeb2f406fafe26a48e10bb

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.6.0-cp310-abi3-win_amd64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.6.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for dask_array-0.6.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 82d2b46a8bab37b4d9cc18ef0d128a0bd0ef0d96cc1dcca2076441ba35a3fc5b
MD5 35b32d7e84ae9deb38644c2001bbd81f
BLAKE2b-256 7f7789c237897fb0be1ad16a81c5ebd3426683c139c4babe1aa0d280951eaf0f

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.6.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.6.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for dask_array-0.6.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 d35eb36b19ab2d98ef0fee405122bc3538cb18f0ea0423ebe987f0f84ecc5596
MD5 7f0e6a525e2d440e20654dad44b8b2c9
BLAKE2b-256 b939b1568e6e0a0dad50aff17fdd6c81f654d9be5bf9a025f7963bd56eb2d5e7

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.6.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.6.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dask_array-0.6.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 a8d89f66c4bb3df228d66c5a2866db2893b2cab64d237a9902b5f2ffd80125c0
MD5 c9dac13134218699f61e3742c4744278
BLAKE2b-256 e68b2ab91ef445142be6ff4ccd5bc44820878472c42392b8bc91b2c760bfdedf

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.6.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page