databricks-job-runner 0.6.1

Reusable CLI for uploading, submitting, validating, fetching logs, and cleaning Databricks job runs

databricks-job-runner

Reusable CLI for uploading, submitting, and cleaning Databricks job runs.

databricks-job-runner wraps the Databricks Python SDK into a small library that each project configures with a Runner instance. One Runner gives you nine CLI subcommands (upload, download, submit, validate, logs, clean, catalog, schema, volume) without writing any Databricks API code in your project.

Install

uv add databricks-job-runner

Or with pip:

pip install databricks-job-runner

Quick start

databricks-job-runner is a library, not a standalone CLI. There is no __main__ in this repo. Each project wires one Runner.

Create a cli/ package with two files:

cli/__init__.py

from databricks_job_runner import Runner

runner = Runner(
    run_name_prefix="my_project",
    wheel_package="my_package",  # optional
)

cli/__main__.py

from cli import runner
runner.main()

Add a .env to your project root with at least:

DATABRICKS_PROFILE=my-profile
DATABRICKS_CLUSTER_ID=0123-456789-abcdef
DATABRICKS_WORKSPACE_DIR=/Users/[e-mail]/my_project

Then run the core lifecycle from your project root:

uv run python -m cli upload --all          # upload agent_modules/
uv run python -m cli submit test_hello.py  # submit a job and wait
uv run python -m cli logs                  # stdout/stderr from the last run
uv run python -m cli clean --yes           # tear down

.env + cli/  ->  upload  ->  submit  ->  (Databricks run)  ->  logs  ->  clean
                   |            |                               |
              workspace/     one-shot                        tail 5MB
              agent_modules  SubmitRun                        stdout/err

Documentation

Page	What it covers
Getting started	Install, project-layout contract, first job end to end, architecture overview.
Configuration	Every .env key, precedence, compute modes, parameter injection, inject_params.
Workflows	Common workflows with diagrams: classic vs serverless, wheels, data, Unity Catalog.
Command reference	Every subcommand, flag, and positional argument.
Bootstrap-from-Volume	Run-startup wheel install, BootstrapConfig, per-run isolation.
Preflight hooks	Fail-fast compute checks before submit/validate, cluster-library helpers.
API reference	Runner, RunnerConfig, Compute, inject_params, RunnerError.
Examples and smoke tests	The two runnable example projects and the serverless test matrix.
Releasing	PyPI tag-based release flow.

Requirements

• Python 3.12+
• Databricks authentication: a Databricks CLI profile, env vars (DATABRICKS_HOST / DATABRICKS_TOKEN), or any other unified-auth method
• Either a Databricks all-purpose cluster (auto-started if terminated) or serverless compute enabled for the workspace
• uv (for wheel building only)