kedro-databricks

Kedro plugin to develop Kedro pipelines for Databricks. This plugin strives to provide an excellent developer experience when using Kedro on Databricks.

Key Features

Initialization: Transform your local Kedro project into a Databricks Asset Bundle.
Generation: Generate Asset Bundle resource definitions from your Kedro pipelines.
Deployment: Deploy your Kedro pipelines to Databricks as Jobs.
Execution: Run your Kedro pipelines on Databricks straight from the command line.
Cleanup: Remove all Databricks resources created by the plugin.

Additional Capabilities

Generate resources per node or per pipeline (--resource-generator).
Extend generation with custom resource generator classes.
Apply default, targeted, and regex-based overrides in conf/<env>/databricks.yml.
Include non-job Databricks resources (for example volumes) in generated bundle resources.
Forward raw Databricks CLI args via -- ... for advanced workflows.
Automatically upload local data/ during deploy when target catalog _file_path is configured.

Documentation & Contributing

To learn more about the plugin, please refer to the documentation.

Interested in contributing? Check out our contribution guidelines to get started!

Breaking Changes

Version `0.14.0`

To accommodate using Databricks Free Edition, we had to change the structure of overrides defined in conf/<env>/databricks.yml.

Before:

default:
    environments:
        - environment_key: default
    spec:
        environment_version: '4'
        dependencies:
            - ../dist/*.whl
    tasks:
        - task_key: default
          environment_key: default

After:

resources:
    jobs:
        default:
            environments:
                - environment_key: default
            spec:
                environment_version: '4'
                dependencies:
                    - ../dist/*.whl
            tasks:
                - task_key: default
                environment_key: default

This was done so that we could default to creating a volume in a newly initialized kedro-databricks project.

While this requires users to migrate their databricks configuration, it also extends the ability of kedro-databricks beyond that of applying overrides to specific jobs. Now, you can add any type of resource in your conf/<env>/databricks.yml and those will be generated as well.

NOTE: Merges are only applied for jobs currently, so any other defined will be generated as defined in the configuration.

In addition to the changes to the structure of conf/<env>/databricks.yml, we now also tag the generated resources with their resource type and target environment, meaning that newly generated resources will be named like target.<env>.<resource-type>.<resouce-name>.yml.

Metadata

Release files for kedro-databricks 0.18.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for kedro-databricks 0.18.2
File	Size	Uploaded
kedro_databricks-0.18.2.tar.gz	1.8 MB	Sep 1, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for kedro-databricks 0.18.2
File	Interpreter	ABI	Platform	Reset
kedro_databricks-0.18.2-py3-none-any.whl 37.5 kB Sep 1, 2026	Python 3	none	any	Details

Total release size: 1.9 MB

Release files / kedro_databricks-0.18.2.tar.gz

Download URL	kedro_databricks-0.18.2.tar.gz
Size	1.8 MB
Tags	Source
SHA-256 checksum How to use checksums	`f6e4efb528cdea095924953d089dd87e0f8c03f90f83ae5841a65308df861f16`
BLAKE2b-256 checksum How to use checksums	`c0e05fc11cddfbbb25f8849b053958ff6a1f96157fdd110cbf3191e53c44661f`
Upload date	Sep 1, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.11.6 {"installer":{"name":"uv","version":"0.11.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`

Release files / kedro_databricks-0.18.2-py3-none-any.whl

Download URL	kedro_databricks-0.18.2-py3-none-any.whl
Size	37.5 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`8de93c4b5f07aaca57ec3bdaa65f3f6e3e7f7596a2713f8f2179f9df90694b26`
BLAKE2b-256 checksum How to use checksums	`f90312eecc351c953633070d23f09d8440e9d71fa73e826f2aa4b54421c8357f`
Upload date	Sep 1, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.11.6 {"installer":{"name":"uv","version":"0.11.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`