Skip to main content

dask-array

Dask array with query optimization

Motivation

Dask Array is powerful, but requires expertise to drive effectively.

The dask-array project reimplements dask.array, but represents your calculation at a level where it can be optimized intelligently before executed. This allows the project to reorder and replace calculations to provide the same result but with a more efficient path.

Installation

pip install dask-array

Usage

This project looks and feels like Dask Array

import dask_array as da

x = da.ones((1000, 1000), chunks=(100, 100))

y = x + x.T
result = y[:100, :100]

But when you go to compute, your calculation gets rewritten to be more efficient. This is apparent if you look at the query structure of the underlying array.

>>> result.pprint()
Operation             Shape    Bytes   Chunks
  Getitem          (100, 100)   78 kiB  100×100
   Add          (1000, 1000)  7.6 MiB  100×100
     Ones       (1000, 1000)  7.6 MiB  100×100
     Transpose  (1000, 1000)  7.6 MiB  100×100
       Ones     (1000, 1000)  7.6 MiB  100×100

This calculation starts from a large array expression, then takes a small slice. It is more efficient to push that slice into the expression before building the task graph.

The optimize function rewrites things automatically.

>>> result.optimize().pprint()
Operation            Shape   Bytes   Chunks
  FusedBlockwise  (100, 100)  78 kiB  100×100
   Ones          (100, 100)  78 kiB  100×100

You don't need to call optimize though. Dask compute/persist machinery will do this for you. You just need to change your import.

# import dask.array as da
import dask_array as da

Native Frisky acceleration

The normal Dask scheduler path is pure Python and works without a Rust toolchain. On platforms with a native wheel, dask-array also includes a Rust accelerator that Frisky can use to submit compact task records instead of materializing large Python task graphs. This is automatic when computing with a Frisky scheduler; otherwise computations stay on the standard Dask path.

Xarray

Dask-array can replace the default dask.array by registering itself as an Xarray chunk manager.

from dask_array.xarray import register
register()

After this all xarray calculations will benefit from query optimization.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_array-0.4.3.tar.gz (520.6 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

dask_array-0.4.3-py3-none-any.whl (370.0 kB view details)

Uploaded Python 3

dask_array-0.4.3-cp310-abi3-win_amd64.whl (826.1 kB view details)

Uploaded CPython 3.10+Windows x86-64

dask_array-0.4.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (970.0 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

dask_array-0.4.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (957.8 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

dask_array-0.4.3-cp310-abi3-macosx_11_0_arm64.whl (919.5 kB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

File details

Details for the file dask_array-0.4.3.tar.gz.

File metadata

  • Download URL: dask_array-0.4.3.tar.gz
  • Upload date:
  • Size: 520.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.3.tar.gz
Algorithm Hash digest
SHA256 51749f7641fe5c81396a7b5357de99eefa4f2a47dffc62eeb35900d561d358e8
MD5 2e4d58c34c34671b79fee545f0879285
BLAKE2b-256 4bd9a58ca02906b6fa43cef01d52c8a64249a533e3342f35f35a4c862a88bfb8

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.3.tar.gz:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.3-py3-none-any.whl.

File metadata

  • Download URL: dask_array-0.4.3-py3-none-any.whl
  • Upload date:
  • Size: 370.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.3-py3-none-any.whl
Algorithm Hash digest
SHA256 1ded6753aaa3480be149ec9e5b578da670f9badf7012c9cb66db9a7233de817a
MD5 e361e06ca500be49b2566a721a6fe1e6
BLAKE2b-256 ac996b99b221f80d7f2f3f8613ce5b1f98bae94a01a3428e3ecad761a476bc5f

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.3-py3-none-any.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.3-cp310-abi3-win_amd64.whl.

File metadata

  • Download URL: dask_array-0.4.3-cp310-abi3-win_amd64.whl
  • Upload date:
  • Size: 826.1 kB
  • Tags: CPython 3.10+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for dask_array-0.4.3-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 619e850e672d8468f0db2997efd12cca0558616c2cfaa6355df1c390f6eaf716
MD5 b3927b0b7a23810eba71fbe453eba570
BLAKE2b-256 1ed3f22d428fc9cb73128a7fab603c81819193f40b282b47b1f9d85c41769fae

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.3-cp310-abi3-win_amd64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 8c3dcefa07815995ca8bcba7b09193d68ccce7bc1c4e3e2183c272969c504f62
MD5 bb1d78be4b313812bfac95cdb1b3ae59
BLAKE2b-256 0962db521195faccce69f1983a106f1404829e5c70ef95905d1453064dcc3697

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.3-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 cb41c5203f9655c32a8c17ef1097173018ca9b3a20dcda2e5c0b1cba52a79bea
MD5 b760d4d79e7c2e8f386bab9ca5b36154
BLAKE2b-256 831836992346a0f81d88b41357605ee018f8241526cc58a9990040055eea4816

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.3-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file dask_array-0.4.3-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for dask_array-0.4.3-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 95fc228b9750887abea7c8a54a7c171f24fd1be0a0049fc25e4e294193354092
MD5 80cf41d5ca5b70224f820c7116eb029a
BLAKE2b-256 1b7d936164ac838f709e38567e9fe5aa34a1c9a0567e7091480331564b2dab95

See more details on using hashes here.

Provenance

The following attestation bundles were made for dask_array-0.4.3-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on mrocklin/dask-array

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page