kedro-databricks

Kedro plugin to develop Kedro pipelines for Databricks. This plugin strives to provide an excellent developer experience when using Kedro on Databricks.

Key Features

Initialization: Transform your local Kedro project into a Databricks Asset Bundle.
Generation: Generate Asset Bundle resource definitions from your Kedro pipelines.
Deployment: Deploy your Kedro pipelines to Databricks as Jobs.
Execution: Run your Kedro pipelines on Databricks straight from the command line.
Cleanup: Remove all Databricks resources created by the plugin.

Additional Capabilities

Generate resources per node or per pipeline (--resource-generator).
Extend generation with custom resource generator classes.
Apply default, targeted, and regex-based overrides in conf/<env>/databricks.yml.
Include non-job Databricks resources (for example volumes) in generated bundle resources.
Forward raw Databricks CLI args via -- ... for advanced workflows.
Automatically upload local data/ during deploy when target catalog _file_path is configured.

Documentation & Contributing

To learn more about the plugin, please refer to the documentation.

Interested in contributing? Check out our contribution guidelines to get started!

Breaking Changes

Version `0.14.0`

To accommodate using Databricks Free Edition, we had to change the structure of overrides defined in conf/<env>/databricks.yml.

Before:

default:
    environments:
        - environment_key: default
    spec:
        environment_version: '4'
        dependencies:
            - ../dist/*.whl
    tasks:
        - task_key: default
          environment_key: default

After:

resources:
    jobs:
        default:
            environments:
                - environment_key: default
            spec:
                environment_version: '4'
                dependencies:
                    - ../dist/*.whl
            tasks:
                - task_key: default
                environment_key: default

This was done so that we could default to creating a volume in a newly initialized kedro-databricks project.

While this requires users to migrate their databricks configuration, it also extends the ability of kedro-databricks beyond that of applying overrides to specific jobs. Now, you can add any type of resource in your conf/<env>/databricks.yml and those will be generated as well.

NOTE: Merges are only applied for jobs currently, so any other defined will be generated as defined in the configuration.

In addition to the changes to the structure of conf/<env>/databricks.yml, we now also tag the generated resources with their resource type and target environment, meaning that newly generated resources will be named like target.<env>.<resource-type>.<resouce-name>.yml.

Metadata

Release files for kedro-databricks 0.18.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for kedro-databricks 0.18.3
File	Size	Uploaded
kedro_databricks-0.18.3.tar.gz	1.8 MB	Sep 1, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for kedro-databricks 0.18.3
File	Interpreter	ABI	Platform	Reset
kedro_databricks-0.18.3-py3-none-any.whl 37.6 kB Sep 1, 2026	Python 3	none	any	Details

Total release size: 1.9 MB

Release files / kedro_databricks-0.18.3.tar.gz

Download URL	kedro_databricks-0.18.3.tar.gz
Size	1.8 MB
Tags	Source
SHA-256 checksum How to use checksums	`a2f8d437f557dfddfd9ef58d27a5b633c2adc7759fef97bc59931d45967dcd69`
BLAKE2b-256 checksum How to use checksums	`a1ff49b18a0c7034f63df76da8db4d0398632f37023015bc73e22dee6ced0afe`
Upload date	Sep 1, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.11.6 {"installer":{"name":"uv","version":"0.11.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`

Release files / kedro_databricks-0.18.3-py3-none-any.whl

Download URL	kedro_databricks-0.18.3-py3-none-any.whl
Size	37.6 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`0ec82d77730b19933f55bf6b99ba4e93d150a6b025536aecd6b9f133ed37b68e`
BLAKE2b-256 checksum How to use checksums	`c9dc5a1278637d1305724011174fdb8eeb323322b146ce7f0af6a09862b6849a`
Upload date	Sep 1, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.11.6 {"installer":{"name":"uv","version":"0.11.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`