Skip to main content

Customizable caching of Dask-delayed.

Project description

Dask-checkpoint

Dask-checkpoint is a Python package that adds a customizable caching capabilities to dask. It builds on top of dask.delayed, adding load and save instructions to the dask graph.

from dask_checkpoint import Storage, task

storage = Storage.from_fsspec("my_directory")


@task(save=True)
def add_one(x):
    return x + 1


x0 = add_one(1).compute()  # computed
with storage():
    x1 = add_one(1).compute()  # computed and saved to storage
    x2 = add_one(1).compute()  # loaded from storage
x3 = add_one(1).compute()  # recomputed, not loaded from storage

assert x0 == x1 == x2 == x3

Installation

Dask-checkpoint can be installed from PyPI:

pip install dask-checkpoint

Getting started

Check out the tutorial to see Dask-checkpoint in action.

Development

To set up a development environment in a new conda environment, run the following commands:

git clone https://github.com/maurosilber/dask-checkpoint
cd dask-checkpoint
conda env create -f environment-dev.yml
pre-commit install

Run tests locally with tox:

tox

or, if you have mamba installed:

CONDA_EXE=mamba tox

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_checkpoint-0.2.1.tar.gz (15.2 kB view details)

Uploaded Source

Built Distribution

dask_checkpoint-0.2.1-py2.py3-none-any.whl (12.6 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file dask_checkpoint-0.2.1.tar.gz.

File metadata

  • Download URL: dask_checkpoint-0.2.1.tar.gz
  • Upload date:
  • Size: 15.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.1 CPython/3.10.6

File hashes

Hashes for dask_checkpoint-0.2.1.tar.gz
Algorithm Hash digest
SHA256 334c52953796e2ab749a655b590e1057cf146ca319d7d07a562f7dccf5e85924
MD5 8950665e1bc8413e440b136269e25727
BLAKE2b-256 6cd4e6ff55d6d298d0ffb9641732c007e9542f74a342b68215a7dbe0a2dee50d

See more details on using hashes here.

File details

Details for the file dask_checkpoint-0.2.1-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for dask_checkpoint-0.2.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 83842ed0bc8e71b04c4a7d1eeaeeb9cbfc6c4c625baf760af9881501a1515f3f
MD5 16eba2cd340a26e85572ca2f69fa2c5a
BLAKE2b-256 c4e93cf9679d3a908d31ee43e58f1d284c084f9b3a975a05a3379867f412e957

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page