Skip to main content

Kedro-Accelerator speeds up pipelines by parallelizing I/O in the background.

Project description


Kedro pipelines consist of nodes, where an output from one node A can be an input to another node B. The Data Catalog defines where and how Kedro loads and saves these inputs and outputs, respectively. By default, a sequential Kedro pipeline:

  1. runs node A
  2. persists the output of A, often to remote storage like Amazon S3
  3. potentially runs other nodes
  4. fetches the output of A, loading it back into memory
  5. runs node B

Persisting intermediate data sets enables partial pipeline runs (e.g. running node B without rerunning node A) and analysis/debugging of these data sets. However, the I/O in steps 2 and 4 above was not necessary to run node B, given the requisite data was already in memory after step 1. Kedro-Accelerator speeds up pipelines by parallelizing this I/O in the background.

How do I install Kedro-Accelerator?

Kedro-Accelerator is a Python plugin. To install it:

pip install kedro-accelerator

How do I use Kedro-Accelerator?

As of Kedro 0.16.4, TeePlugin—the core of Kedro-Accelerator—will be auto-discovered upon installation. In older versions, hook implementations should be registered with Kedro through the ProjectContext. Hooks were introduced in Kedro 0.16.0.


The following conditions must be true for Kedro-Accelerator to speed up your pipeline:

  • Your project must use either SequentialRunner or ParallelRunner.


The Kedro-Accelerator repository includes the Iris data set example pipeline generated using Kedro 0.16.1. Intermediate data sets have been replaced with custom SlowDataSet instances to simulate a slow filesystem. You can try different load and save delays by modifying catalog.yml.

To get started, create and activate a new virtual environment. Then, clone the repository and pip install requirements:

git clone
cd kedro-accelerator
KEDRO_VERSION=0.17.4 pip install -r src/requirements.txt  # Specify your desired Kedro version.

You can compare pipeline execution times with and without TeePlugin. Kedro-Accelerator also provides CachePlugin so that you can test performance using CachedDataSet in asynchronous mode. Assuming parametrized load and save delays of 10 seconds for intermediate datasets, you should see the following results:

Strategy Command Total time Log
Baseline (i.e. no caching/plugins) kedro run 2 minutes Log
TeePlugin kedro run --hooks kedro_accelerator.plugins.TeePlugin 10 seconds (saving all outputs) Log
CachePlugin (i.e. CachedDataSet) with is_async=True kedro run --async --hooks kedro_accelerator.plugins.CachePlugin 30 seconds (saving split_data, train_model, and predict node outputs) Log

Prior to Kedro version 0.17.0, prefix extra hooks passed to kedro run with src. (e.g. src.kedro_accelerator.plugins.TeePlugin).

For a more complete discussion of the above benchmarks, see quantumblacklabs/kedro#420 (comment).

What license do you use?

Kedro-Accelerator is licensed under the MIT License.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kedro-accelerator-0.3.0.tar.gz (5.6 kB view hashes)

Uploaded source

Built Distribution

kedro_accelerator-0.3.0-py3-none-any.whl (5.1 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page