kedro-databricks
Kedro plugin to develop Kedro pipelines for Databricks. This plugin strives to provide an excellent developer experience when using Kedro on Databricks.
Key Features
- Initialization: Transform your local Kedro project into a Databricks Asset Bundle.
- Generation: Generate Asset Bundle resource definitions from your Kedro pipelines.
- Deployment: Deploy your Kedro pipelines to Databricks as Jobs.
- Execution: Run your Kedro pipelines on Databricks straight from the command line.
- Cleanup: Remove all Databricks resources created by the plugin.
Additional Capabilities
- Generate resources per node or per pipeline (
--resource-generator). - Extend generation with custom resource generator classes.
- Apply default, targeted, and regex-based overrides in
conf/<env>/databricks.yml. - Include non-job Databricks resources (for example volumes) in generated bundle resources.
- Forward raw Databricks CLI args via
-- ...for advanced workflows. - Automatically upload local
data/during deploy when target catalog_file_pathis configured.
Documentation & Contributing
To learn more about the plugin, please refer to the documentation.
Interested in contributing? Check out our contribution guidelines to get started!
Breaking Changes
Version 0.14.0
To accommodate using Databricks Free Edition, we had to change the structure of overrides defined in conf/<env>/databricks.yml.
Before:
default:
environments:
- environment_key: default
spec:
environment_version: '4'
dependencies:
- ../dist/*.whl
tasks:
- task_key: default
environment_key: default
After:
resources:
jobs:
default:
environments:
- environment_key: default
spec:
environment_version: '4'
dependencies:
- ../dist/*.whl
tasks:
- task_key: default
environment_key: default
This was done so that we could default to creating a volume in a newly initialized kedro-databricks project.
While this requires users to migrate their databricks configuration, it also extends the ability of kedro-databricks beyond that of applying overrides to specific jobs. Now, you can add any type of resource in your conf/<env>/databricks.yml and those will be generated as well.
NOTE: Merges are only applied for
jobscurrently, so any other defined will be generated as defined in the configuration.
In addition to the changes to the structure of conf/<env>/databricks.yml, we now also tag the generated resources with their resource type and target environment, meaning that newly generated resources will be named like target.<env>.<resource-type>.<resouce-name>.yml.
2 maintainers
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file kedro_databricks-0.18.1.tar.gz.
File metadata
- Download URL: kedro_databricks-0.18.1.tar.gz
- Upload date:
- Size: 1.8 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.11.6 {"installer":{"name":"uv","version":"0.11.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
358165930031f1db269cbde6f72c08dbf9930e1b88cfc85dff0e1057ee2526e1
|
|
| MD5 |
87fe7a25118615ce15c30d5a9af30a2b
|
|
| BLAKE2b-256 |
6d072c139fac81a30c1d995e41a9002f48361b956aa8927e3b96924ba37c52ed
|
File details
Details for the file kedro_databricks-0.18.1-py3-none-any.whl.
File metadata
- Download URL: kedro_databricks-0.18.1-py3-none-any.whl
- Upload date:
- Size: 37.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.11.6 {"installer":{"name":"uv","version":"0.11.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b7c8579a04ea8a9b1c2ef5ee370783100ccacb4ade5353a245bbe5422c7cf331
|
|
| MD5 |
cb6ab6e20ea28ecbf34b0bf1e6ae6e7e
|
|
| BLAKE2b-256 |
eaf3b39388e1953a4031cc048fd24e63d7fba6548e084f405306979b38446f13
|
