Skip to main content

dask-array

Dask array with query optimization

Motivation

Dask Array is powerful, but requires expertise to drive effectively.

The dask-array project reimplements dask.array, but represents your calculation at a level where it can be optimized intelligently before executed. This allows the project to reorder and replace calculations to provide the same result but with a more efficient path.

Installation

pip install dask-array

Usage

This project looks and feels like Dask Array

import dask_array as da

x = da.ones((1000, 1000), chunks=(100, 100))

y = x + x.T
result = y[:100, :100]

But when you go to compute, your calculation gets rewritten to be more efficient. This is apparent if you look at the query structure of the underlying array.

>>> result.pprint()
Operation             Shape    Bytes   Chunks
  Getitem          (100, 100)   78 kiB  100×100
   Add          (1000, 1000)  7.6 MiB  100×100
     Ones       (1000, 1000)  7.6 MiB  100×100
     Transpose  (1000, 1000)  7.6 MiB  100×100
       Ones     (1000, 1000)  7.6 MiB  100×100

This calculation starts from a large array expression, then takes a small slice. It is more efficient to push that slice into the expression before building the task graph.

The optimize function rewrites things automatically.

>>> result.optimize().pprint()
Operation            Shape   Bytes   Chunks
  FusedBlockwise  (100, 100)  78 kiB  100×100
   Ones          (100, 100)  78 kiB  100×100

You don't need to call optimize though. Dask compute/persist machinery will do this for you. You just need to change your import.

# import dask.array as da
import dask_array as da

Native Frisky acceleration

The normal Dask scheduler path is pure Python and works without a Rust toolchain. On platforms with a native wheel, dask-array also includes a Rust accelerator that Frisky can use to submit compact task records instead of materializing large Python task graphs. This is automatic when computing with a Frisky scheduler; otherwise computations stay on the standard Dask path.

Xarray

Dask-array can replace the default dask.array by registering itself as an Xarray chunk manager.

from dask_array.xarray import register
register()

After this all xarray calculations will benefit from query optimization.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_array-0.4.1.tar.gz (514.4 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dask_array-0.4.1-py3-none-any.whl (366.8 kB view details)

Uploaded Python 3

dask_array-0.4.1-cp310-abi3-win_amd64.whl (819.6 kB view details)

Uploaded CPython 3.10+Windows x86-64

dask_array-0.4.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (963.1 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

dask_array-0.4.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (950.8 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

dask_array-0.4.1-cp310-abi3-macosx_11_0_arm64.whl (913.0 kB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file dask_array-0.4.1.tar.gz.

File metadata

  • Download URL: dask_array-0.4.1.tar.gz
  • Upload date:
  • Size: 514.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.1.tar.gz
Algorithm Hash digest
SHA256 a1bf3e835c5dbfdd91ff28208cd582550e6c18deb36b0278f924b6203ca277ff
MD5 e53aed29618c679d5dce902badd34252
BLAKE2b-256 d1711613885894ac3f8ab8709aeed92b824e38f3073b7120ed23b18704738f5e

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.1.tar.gz:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.1-py3-none-any.whl.

File metadata

  • Download URL: dask_array-0.4.1-py3-none-any.whl
  • Upload date:
  • Size: 366.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.1-py3-none-any.whl
Algorithm Hash digest
SHA256 3aefd8247473a66b96c9376db06689da1387ba79a50daae165b0db4f2ea2a3c8
MD5 63658cab2e815afae337cced60cdfc25
BLAKE2b-256 8d343472ddf71970ff9867e0cf15179bb3eea3d9de12237889861cc3168e1bb9

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.1-py3-none-any.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.1-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: dask_array-0.4.1-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 819.6 kB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.1-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 17b89fa41367dbec681a0ffb8a8dcfe5f00a099603ba5fd4900fba36c22252a6
MD5 39b959f1aacee764950ec7040aaeed8e
BLAKE2b-256 216274a41b70546e4e5887507e10354eb9035a0667325d8910a0108ad944deca

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.1-cp310-abi3-win_amd64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 34ffa95834b751267a5a23067d7d967f6418ed598693964bbf9c0f19ef36c6f7
MD5 28b8fda96782a45347b349692a550de7
BLAKE2b-256 515a513baf7b140d349d0053fe605d830e3b4ff830faed6095d077c617ad202a

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.1-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 42919984fe7e24df22cfe73ef7cef654e7ca377bdf5c8e786c5f4b7ca69a3687
MD5 a09d6d4aea8109206188cf168737564c
BLAKE2b-256 85e7688849dc3d33e53b4fb9fadabbf32770370ac06973552c5aa9cf537ebc4b

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.1-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.1-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.1-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 dec9d635eebe7edc5a357fdf323106a2468956b666283a9071354ad319ce8d2e
MD5 e5b48c8d0141abccc399fafb7648d142
BLAKE2b-256 93a20b5248eb0391d1fa59f94677989b32c09f341e0f2c7be26edeee1d2c51a6

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.1-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page