Skip to main content

protection against stepping on teammates toes in a kedro project

Project description

Steel Toes

a kedro hook to protect against breaking changes to data

steel-toes is a kedro hook designed to prevent stepping on your teammates toes. It will branch your data automatically based on your git branch, or manually by passing the branch name into the hook.


kedro is a ✨ fantastic project that allows for super-fast prototyping of data pipelines, while yielding production-ready pipelines. kedro promotes collaborative projects by giving each team member access to the exact same data. Team members will often make their own branch of the project and begin work. Sometimes these changes will break existing functionality. Sometimes we make mistakes as we develop, and fix them before merging in. Either case can be detrimental to a teammate working downstream of your changes if not careful.

steel-toes hooks into your catalog to prevent changing downstream data on your teammates while developing in parallel.

on_catalog_created and before_pipeline_run

When your project creates a catalog steel-toes will look to see if branched data exists, if it does it will swap the filepath to the branched path. So you will be able to load the latest data from the perspective of any branch simulaneusly.


After your node is ran, before saving, steel-toes will check if your filepath was swapped, if not it will swap it to the branched filepath before saving.


steel-toes is deployed to pypi and can easily be pip installed.

pip install steel-toes

Python Usage

since steel-toes requires access to the project_context to work you will need to use a bit of an unconventional method to initialize your hooks.

Example Setup

from steel_toes import SteelToes

class ProjectContext(KedroContext):
   project_name = "kedro0160"
   project_version = "0.16.1"
   package_name = "kedro0160"
def hooks(self):
   self._hooks = [ SteelToes(self), ]
   return self._hooks

CLI Usage

The CLI provides a handy interface to clean up your branched datasets.

$ steel-toes --help
Usage: steel-toes [OPTIONS] COMMAND [ARGS]...


  -V, --version  Prints version and exits
  --help         Show this message and exit.

  clean-branch  finds branch datasets and removes them

steel-toes also registers itself as a kedro global cli plugin. You can run kedro clean-branch to clean your branched data.

$ kedro clean-branch --help
Usage: kedro clean-branch [OPTIONS]

  finds branch datasets and removes them

  --dryrun                   Displays the files that would be deleted using
                             the specified command without actually deleting

  -b, --branch TEXT          git branch to clean files from
  -d, --directory DIRECTORY  Path to save the static site to
  -h, --help                 Show this message and exit.


You're Awesome for considering a contribution! Contributions are welcome, please check out the Contributing Guide for more information. Please be a positive member of the community and embrace feedback


We use SemVer for versioning. For the versions available, see the tags on this repository.


Waylon Walker - Waylon Walker - Original Author


This project is licensed under the MIT License - see the LICENSE. file for details

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

steel-toes-0.0.1.tar.gz (11.8 kB view hashes)

Uploaded Source

Built Distribution

steel_toes-0.0.1-py3-none-any.whl (14.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page