Kedro helps you build production-ready data and analytics pipelines
What is Kedro?
"The centre of your data pipeline."
Kedro is a workflow development tool that helps you build data pipelines that are robust, scalable, deployable, reproducible and versioned. We provide a standard approach so that you can:
- spend more time building your data pipeline,
- worry less about how to write production-ready code,
- standardise the way that your team collaborates across your project,
- work more efficiently.
This work was later turned into a product thanks to the following contributors: Ivan Danov, Dmitrii Deriabin, Gordon Wrigley, Yetunde Dada, Nasef Khan, Kiyohito Kunii, Nikolaos Kaltsas, Meisam Emamjome, Peteris Erins, Lorena Balan, Richard Westenra and Anton Kirilenko.
How do I install Kedro?
kedro is a Python package. To install it, simply run:
pip install kedro
For more detailed installation instructions, including how to setup Python virtual environments, please visit our installation guide.
What are the main features of Kedro?
1. Project template and coding standards
- A standard and easy-to-use project template
- Configuration for credentials, logging, data loading and Jupyter Notebooks / Lab
- Test-driven development using
- Sphinx integration to produce well-documented code
2. Data abstraction and versioning
- Separation of the compute layer from the data handling layer, including support for different data formats and storage options
- Versioning for your data sets and machine learning models
3. Modularity and pipeline abstraction
- Support for pure Python functions,
nodes, to break large chunks of code into small independent sections
- Automatic resolution of dependencies between
- Visualise your data pipeline with Kedro-Viz, a tool that shows the pipeline structure of Kedro projects
Note: Read our FAQs to learn how we differ from workflow managers like Airflow and Luigi.
A pipeline visualisation generated using Kedro-Viz
4. Feature extensibility
- A plugin system that injects commands into the Kedro command line interface (CLI)
- List of officially supported plugins:
- Kedro can be deployed locally, on-premise and cloud (AWS, Azure and GCP) servers, or clusters (EMR, Azure HDinsight, GCP and Databricks)
What are the main Kedro building blocks?
You can find the overview of Kedro architecture here.
How do I use Kedro?
Our documentation explains:
- A typical Kedro workflow
- How to set up the project configuration
- Building your first pipeline
- How to use the CLI offered by
kedro run, ...)
Note: The CLI is a convenient tool for being able to run
kedrocommands but you can also invoke the Kedro CLI as a Python module with
python -m kedro
How do I find Kedro documentation?
This CLI command will open the documentation for your current version of Kedro in a browser:
Documentation for the latest stable release can be found here. Check these out first:
Can I contribute?
Yes! Want to help build Kedro? Check out our guide to contributing.
How do I upgrade Kedro?
Once Kedro is installed, you can check your version as follows:
To later upgrade Kedro to a different version, simply run:
pip install kedro -U
What licence do you use?
Kedro is licensed under the Apache 2.0 License.
Do you want to be part of the team that builds Kedro and other great products at QuantumBlack? If so, you're in luck! QuantumBlack is currently hiring Software Engineers who love using data to drive their decisions. Take a look at our open positions and see if you're a fit.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.