Skip to main content

Customizable caching of Dask-delayed.

Project description

Dask-checkpoint

Dask-checkpoint is a Python package that adds a customizable caching capabilities to dask. It builds on top of dask.delayed, adding load and save instructions to the dask graph.

from dask_checkpoint import Storage, task

storage = Storage.from_fsspec("my_directory")


@task(save=True)
def add_one(x):
    return x + 1


x0 = add_one(1).compute()  # computed
with storage():
    x1 = add_one(1).compute()  # computed and saved to storage
    x2 = add_one(1).compute()  # loaded from storage
x3 = add_one(1).compute()  # recomputed, not loaded from storage

assert x0 == x1 == x2 == x3

Installation

Dask-checkpoint can be installed from PyPI:

pip install dask-checkpoint

Getting started

Check out the tutorial to see Dask-checkpoint in action.

Development

To set up a development environment in a new conda environment, run the following commands:

git clone https://github.com/maurosilber/dask-checkpoint
cd dask-checkpoint
conda env create -f environment-dev.yml
pre-commit install

Run tests locally with tox:

tox

or, if you have mamba installed:

CONDA_EXE=mamba tox

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

dask_checkpoint-0.2.1.tar.gz (15.2 kB view hashes)

Uploaded Source

Built Distribution

dask_checkpoint-0.2.1-py2.py3-none-any.whl (12.6 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page