Databricks-Rocket
Databricks-Rocket (short db-rockets), keeps your local Python scripts installed and synchronized with a Databricks notebook. Every change on your local machine is automatically reflected in the notebook. This shortens the feedback loop for developing git-based projects and eliminates the need to set up a local development environment.
Installation
Install databricks-rocket using pip:
pip install databricks-rocket
Setup
Ensure you've created a personal access token in Databricks (offical documentation). Afterward, set up the Databricks CLI by executing:
databricks configure --token
Alternatively, you can set the Databricks token and host in your environment variables:
export DATABRICKS_HOST="mydatabrickshost"
export DATABRICKS_TOKEN="mydatabrickstoken"
If your project isn't already a pip package, you'll need to convert it into one. Use dbrocket for this:
rocket setup
Will create a setup.py for you.
Usage
To Sync Your Project
By default, databricks-rocket syncs your project to DBFS automatically. This allows you to update your code and have
those changes reflected in your Databricks notebook without restarting the Python kernel. Simply execute:
rocket launch
You'll then receive the exact command to run in your notebook. Example:
stevenmi@MacBook db-rocket % rocket launch --watch=False
>> Watch activated. Uploaded your project to databricks. Install your project in your databricks notebook by running:
>> %pip install --upgrade pip
>> %pip install -r /dbfs/temp/stevenmi/db-rocket/requirements.txt
>> %pip install --no-deps -e /dbfs/temp/stevenmi/db-rocket
and following in a new Python cell:
>> %load_ext autoreload
>> %autoreload 2
Finally, add the content in you databricks notebook:
Include non-python files
Upload all root level json files:
rocket launch --glob_path="*,json"
On top also upload all env files:
rocket launch --glob_path="[\"*.json\", \".env*\"]"
When specifying lists, be mindful about the formatting of the parameter string.
To Upload Your Python Package
If you've disabled the watch feature, databricks-rocket will only upload your project as a wheel to DBFS:
rocket launch --watch=False
Example:
stevenmi@MacBook db-rocket % rocket launch --watch=False
>> Watch is disabled. Building creating a python wheel from your project
>> Found setup.py. Building python library
>> Uploaded ./dist/databricks_rocket-2.0.0-py3-none-any.whl to dbfs:/temp/stevenmi/db-rocket/dist/databricks_rocket-2.0.0-py3-none-any.whl
>> Uploaded wheel to databricks. Install your library in your databricks notebook by running:
>> %pip install --upgrade pip
>> %pip install /dbfs/temp/stevenmi/db-rocket/databricks_rocket-2.0.0-py3-none-any.whl --force-reinstall
Blogposts
- DBrocket 2.0: A summary of the big improvements we made to the tool in the new release.
- DB Rocket 1.0 post also gives more details about the rationale around dbrocket.
Support
- Databricks: >=7
- Python: >=3.7
- Tested on Platform: Linux, MacOs. Windows will probably not work but contributions are welcomed!
- Supports uploading to Unity Catalog Volumes starting from version 3.0.0. Note that the underlying dependency,
databricks-sdk, is still in beta. We do not recommend using UC Volumes in production.
Acknowledgments
- Thanks Leon Poli for the Logo :)
- Thanks Stephane Leonard for source-code and documentation improvements :)
- Thanks Malachi Soord for the CICD setup and README improvements
Contributions are welcomed!
Security
For security issues please contact security@getyourguide.com.
Legal
db-rocket is licensed under the Apache License, Version 2.0. See LICENSE for the full text.
4 maintainers
Metadata
4 maintainers
Release files for databricks-rocket 3.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| databricks-rocket-3.1.0.tar.gz | 12.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| databricks_rocket-3.1.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 27.0 kB
Release files / databricks-rocket-3.1.0.tar.gz
| Download URL | databricks-rocket-3.1.0.tar.gz |
|---|---|
| Size | 12.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2ecb0868dff752c31546ff6d6ff20d19473f6e5b065eaf32bae69e66e345380d
|
|
BLAKE2b-256 checksum How to use checksums |
b6dcd8e2cd05881c96a83f6690585104d6bf1e20375388cf5810f1a7de1767c9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
Release files / databricks_rocket-3.1.0-py3-none-any.whl
| Download URL | databricks_rocket-3.1.0-py3-none-any.whl |
|---|---|
| Size | 14.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
896c79906526b83369deb04ab9af8fb22fcda5b02b792cc9048552f9db544aae
|
|
BLAKE2b-256 checksum How to use checksums |
49fad3834ae84d3dd89af7a041d8afb9a7f69069974b21ae7e90357796a829d5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.10.16
|
