databricks-job-runner

Reusable CLI for uploading, submitting, and cleaning Databricks job runs.

databricks-job-runner wraps the Databricks Python SDK into a small library that each project configures with a Runner instance. One Runner gives you nine CLI subcommands (upload, download, submit, validate, logs, clean, catalog, schema, volume) without writing any Databricks API code in your project.

Install

uv add databricks-job-runner

Or with pip:

pip install databricks-job-runner

Quick start

databricks-job-runner is a library, not a standalone CLI. There is no __main__ in this repo. Each project wires one Runner.

Create a cli/ package with two files:

cli/__init__.py

from databricks_job_runner import Runner

runner = Runner(
    run_name_prefix="my_project",
    wheel_package="my_package",  # optional
)

cli/__main__.py

from cli import runner
runner.main()

Add a .env to your project root with at least:

DATABRICKS_PROFILE=my-profile
DATABRICKS_CLUSTER_ID=0123-456789-abcdef
DATABRICKS_WORKSPACE_DIR=/Users/you@example.com/my_project

Then run the core lifecycle from your project root:

uv run python -m cli upload --all          # upload agent_modules/
uv run python -m cli submit test_hello.py  # submit a job and wait
uv run python -m cli logs                  # stdout/stderr from the last run
uv run python -m cli clean --yes           # tear down

.env + cli/  ->  upload  ->  submit  ->  (Databricks run)  ->  logs  ->  clean
                   |            |                               |
              workspace/     one-shot                        tail 5MB
              agent_modules  SubmitRun                        stdout/err

Documentation

Requirements

• Python 3.12+
• Databricks authentication: a Databricks CLI profile, env vars (DATABRICKS_HOST / DATABRICKS_TOKEN), or any other unified-auth method
• Either a Databricks all-purpose cluster (auto-started if terminated) or serverless compute enabled for the workspace
• uv (for wheel building only)