Keep your local python scripts installed and in sync with a databricks notebook. Shortens the feedback loop to develop projects using a hybrid enviroment
Databricks-Rocket (short db-rockets), keeps your local Python scripts installed and synchronized with a Databricks notebook. Every change on your local machine is automatically reflected in the notebook. This shortens the feedback loop for developing git-based projects and eliminates the need to set up a local development environment.
databricks-rocket using pip:
pip install databricks-rocket
Ensure you've created a personal access token in Databricks (offical documentation). Afterward, set up the Databricks CLI by executing:
databricks configure --token
Alternatively, you can set the Databricks token and host in your environment variables:
export DATABRICKS_HOST="mydatabrickshost" export DATABRICKS_TOKEN="mydatabrickstoken"
If your project isn't already a pip package, you'll need to convert it into one. Use dbrocket for this:
Will create a setup.py for you.
To Sync Your Project
databricks-rocket syncs your project to DBFS automatically. This allows you to update your code and have
those changes reflected in your Databricks notebook without restarting the Python kernel. Simply execute:
You'll then receive the exact command to run in your notebook. Example:
stevenmi@MacBook db-rocket % rocket launch --watch=False >> Watch activated. Uploaded your project to databricks. Install your project in your databricks notebook by running: >> %pip install --upgrade pip >> %pip install -r /dbfs/temp/stevenmi/db-rocket/requirements.txt >> %pip install --no-deps -e /dbfs/temp/stevenmi/db-rocket and following in a new Python cell: >> %load_ext autoreload >> %autoreload 2
Finally, add the content in you databricks notebook:
To Upload Your Python Package
If you've disabled the watch feature,
databricks-rocket will only upload your project as a wheel to DBFS:
rocket launch --watch=False
stevenmi@MacBook db-rocket % rocket launch --watch=False >> Watch is disabled. Building creating a python wheel from your project >> Found setup.py. Building python library >> Uploaded ./dist/databricks_rocket-2.0.0-py3-none-any.whl to dbfs:/temp/stevenmi/db-rocket/dist/databricks_rocket-2.0.0-py3-none-any.whl >> Uploaded wheel to databricks. Install your library in your databricks notebook by running: >> %pip install --upgrade pip >> %pip install /dbfs/temp/stevenmi/db-rocket/databricks_rocket-2.0.0-py3-none-any.whl --force-reinstall
- Databricks: >=7
- Python: >=3.7
- Tested on Platform: Linux, MacOs. Windows will probably not work but contributions are welcomed!
- Thanks Leon Poli for the Logo :)
- Thanks Stephane Leonard for source-code and documentation improvements :)
- Thanks Malachi Soord for the CICD setup and README improvements
Contributions are welcomed!
- DB Rocket 1.0 post also gives more details about the rationale around dbrocket.
For security issues please contact firstname.lastname@example.org.
db-rocket is licensed under the Apache License, Version 2.0. See LICENSE for the full text.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for databricks_rocket-2.0.3-py3-none-any.whl