protection against stepping on teammates toes in a kedro project
Project description
Steel Toes
a kedro hook to protect against breaking changes to data
steel-toes
is a kedro hook designed to prevent stepping on your teammates toes. It will branch your data automatically based on your git branch, or manually by passing the branch name into the hook.
Motivation
kedro
is a ✨ fantastic project that allows for super-fast prototyping of data pipelines, while yielding production-ready pipelines. kedro
promotes collaborative projects by giving each team member access to the exact same data. Team members will often make their own branch of the project and begin work. Sometimes these changes will break existing functionality. Sometimes we make mistakes as we develop, and fix them before merging in. Either case can be detrimental to a teammate working downstream of your changes if not careful.
steel-toes
hooks into your catalog to prevent changing downstream data on your teammates while developing in parallel.
on_catalog_created and before_pipeline_run
When your project creates a catalog steel-toes
will look to see if branched data exists, if it does it will swap the filepath to the branched path. So you will be able to load the latest data from the perspective of any branch simulaneusly.
after_node_run
After your node is ran, before saving, steel-toes
will check if your filepath
was swapped, if not it will swap it to the branched filepath
before saving.
Installation
steel-toes
is deployed to pypi and can easily be pip
installed.
pip install steel-toes
Python Usage
since steel-toes
requires access to the project_context to work you will
need to use a bit of an unconventional method to initialize your hooks.
Example Setup
from steel_toes import SteelToes
class ProjectContext(KedroContext):
project_name = "kedro0160"
project_version = "0.16.1"
package_name = "kedro0160"
@property
def hooks(self):
self._hooks = [ SteelToes(self), ]
return self._hooks
CLI Usage
The CLI provides a handy interface to clean up your branched datasets.
$ steel-toes --help
Usage: steel-toes [OPTIONS] COMMAND [ARGS]...
help
Options:
-V, --version Prints version and exits
--help Show this message and exit.
Commands:
clean-branch finds branch datasets and removes them
steel-toes
also registers itself as a kedro
global cli plugin. You can run kedro clean-branch
to clean your branched data.
$ kedro clean-branch --help
Usage: kedro clean-branch [OPTIONS]
finds branch datasets and removes them
Options:
--dryrun Displays the files that would be deleted using
the specified command without actually deleting
them.
-b, --branch TEXT git branch to clean files from
-d, --directory DIRECTORY Path to save the static site to
-h, --help Show this message and exit.
Contributing
You're Awesome for considering a contribution! Contributions are welcome, please check out the Contributing Guide for more information. Please be a positive member of the community and embrace feedback
Versioning
We use SemVer for versioning. For the versions available, see the tags on this repository.
Author
- Waylon Walker - Original Author
License
This project is licensed under the MIT License - see the LICENSE. file for details
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for steel_toes-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5636042f3b45dcc0f0f585cba2f7987fca105da82c412217264099452fa977e2 |
|
MD5 | a50cc0df091d3b7f4fdfc34be7ca6620 |
|
BLAKE2b-256 | 12f2d9fe1b3437157b909f9cd5b8673adfd59a3d8a66f645cb39912d837f1510 |