Skip to main content

databricks360

Installs Databricks course lab environments — notebooks, catalogs, datasets and governance objects — into your own workspace.

Modelled on dbdemos: you run it inside a Databricks notebook, so databricks-sdk picks up the notebook's own identity. There is no host, token or profile to configure.

Install

Not on PyPI yet, so install from the repo. In a Databricks notebook:

%pip install git+https://github.com/databrickslms/dbxdemos.git
dbutils.library.restartPython()

Once published, that becomes %pip install databricks360.

To pin a version, append a tag or commit:

%pip install git+https://github.com/databrickslms/dbxdemos.git@v0.1.0

Usage

import databricks360 as academy

academy.list_courses()
academy.install('genie-agents')

The bare call uses the course's own naming — a single genie_agent schema in your current catalog, with objects named mfg_core_*, mfg_ref_*, mfg_staging_*:

genie_agent.mfg_core_dim_date
genie_agent.mfg_core_fct_transactions
genie_agent.mfg_ref_documents

Override with schema=, table_prefix= or catalog= — see Regulated environments below.

install writes the lab notebooks into your workspace and prints the run order. Then you open them and run each in turn.

Installed 'genie-agents' → /Workspace/Users/you@corp.com/databricks360/genie-agents
  catalog: mfg    tier: small

  Run these in order:
    1. 01_catalog_and_schemas
    2. 02_dimensions
    3. 03_facts

Genie Agents

Courses that need Genie Agents ship them as serialized_space definitions, so an agent is never built by hand:

academy.create_agents('genie-agents', dry_run=True)   # check, create nothing
academy.create_agents('genie-agents')                 # create them

Notebook 08_agents does exactly this, so it is usually easier to run that in order with the rest. Object names come from the same layout as the notebooks, so the agents follow whichever catalog and schema you installed into.

This is the one call that creates something without you running a notebook. It is still cheap: an agent is metadata, and nothing touches a warehouse until someone asks it a question. It refuses to create an agent whose tables do not exist yet, because an agent pointed at a missing table fails on every question and the failure looks like a Genie problem rather than a notebook you skipped.

Options

academy.install(
    'genie-agents',
    path='/Workspace/Shared/labs',   # default: your home folder
    catalog='training_v2',           # default: the course's own catalog
    tier='large',                    # default: 'small'
    overwrite=True,                  # replace existing notebooks
)

Why it does not run the notebooks for you

dbdemos starts a job and loads the data on your behalf. This deliberately does not. Generating the data is the substance of Module 0 — the point is to watch a warehouse chew through 20M rows and see the flaws appear, not to have a finished catalog materialise. It also means nothing consumes your DBUs without you asking.

Regulated environments

A locked-down Unity Catalog is normal in a governed organisation, so the default assumes the least privilege that can still work: the lab lands in whatever current_catalog() returns, and nothing is created above schema level.

Notebook 01 opens by showing you exactly where that is:

SELECT current_catalog() AS default_catalog, current_schema() AS default_schema;

The four shapes

# 1. Default — current catalog, schemas core / ref / staging.
#    No CREATE CATALOG attempted. Table names are two-level (core.dim_date),
#    so SQL resolves the catalog itself.
academy.install('genie-agents')

# 2. A specific catalog you were granted. Adds USE CATALOG, still creates no catalog.
academy.install('genie-agents', catalog='main')

# 3. Create a catalog — opt in, only if you hold the privilege.
academy.install('genie-agents', catalog='mfg', create_catalog=True)

# 4. One schema you already own. Everything lands there.
academy.install('genie-agents', schema='training_you')
academy.install('genie-agents', catalog='main', schema='training_you')

create_catalog is off by default and raises if you pass it without a catalog name — current_catalog() exists by definition, so there would be nothing to create.

Passing schema assumes you cannot create that schema either, since that is the reason to reach for it. Override with create_schema=True if you can.

One schema shared with other content

If your schema also holds other things, prefix the object names so their provenance is visible:

academy.install('genie-agents', schema='genie_agent', table_prefix='mfg_')

The logical group folds into the name:

Default With table_prefix='mfg_'
core.dim_date genie_agent.mfg_core_dim_date
ref.documents genie_agent.mfg_ref_documents
staging.fct_txn_legacy genie_agent.mfg_staging_fct_txn_legacy

The prefix names objects, never the schema. table_prefix requires schema — without one, the schemas already namespace the objects and a prefix would just be noise.

Worth weighing before you use it. Genie reads table and column names as context, and Module 7 of the course is partly about removing noise so an agent has less to wade through. mfg_core_dim_date is measurably noisier than dim_date. Use a prefix when the schema is genuinely shared and provenance matters more; skip it when the schema is yours.

No CREATE VOLUME either

academy.install('genie-agents', schema='training_you', create_volume=False)

The volume only serves Agent mode over unstructured files (Modules 3 and 16). Skipping it costs those exercises and nothing else, and the notebook says so.

Why the single-schema case renames nothing

core holds the dimensions and facts, ref holds only the documents volume, and staging holds only the two decoy tables. The three sets of object names do not overlap, so collapsing them into one schema is safe — and notebooks 02 onward resolve their own paths, so they need no changes whichever layout you chose.

What you get back

install reports the resolved layout and anything it declined to create, so a restricted install is visible rather than quietly partial:

Installed 'genie-agents' → /Workspace/Users/you@corp.com/databricks360/genie-agents
  tier: small
  single schema: current_catalog().training_you
  skipping: CREATE SCHEMA, CREATE VOLUME

Unity Catalog is required. Genie Agents read Unity Catalog objects, so hive_metastore will not work as the target catalog.

Tiers

Tier Transactions Use
small 20M default. Every module except 13
large 900M Module 13 (latency) only. Left unclustered on purpose

Start small. The large tier exists because you cannot measure query latency on a toy dataset, and nowhere else needs it.

Adding a course

Each course is a subpackage under databricks360/courses/:

databricks360/courses/<course_id>/
  __init__.py
  manifest.json      # title, default catalog, tiers, notebooks in run order
  *.sql              # synced from content/courses/<id>/assets/lab/

Placeholders available in the SQL: {{CATALOG}}, plus anything declared under a tier's values (currently {{TXN_COUNT}}). An unresolved placeholder raises rather than silently rendering empty.

Dataset documentation lives in docs/.

Publishing

Releases go to PyPI via Trusted Publishing — GitHub Actions authenticates to PyPI with a short-lived OIDC identity, so no API token exists in repo secrets or on anyone's laptop. A leaked token is the usual way a package supply chain gets compromised; the safest token is one that was never created.

One-time PyPI setup

  1. Sign in at pypi.org → Your account → Publishing

  2. Under Add a new pending publisher, choose GitHub and enter exactly:

    Field Value
    PyPI Project Name databricks360
    Owner databrickslms
    Repository name dbxdemos
    Workflow name publish.yml
    Environment name pypi
  3. In GitHub → Settings → Environments → New environment → name it pypi. Add yourself as a required reviewer if you want to approve each release.

"Pending" publisher is correct — the project does not exist on PyPI yet, and the first successful run creates it.

Cutting a release

# bump version in pyproject.toml, commit, then:
git tag v0.1.0
git push origin v0.1.0

The workflow runs the tests, builds, checks the tag matches pyproject.toml, and publishes. A mismatched tag fails before anything reaches the index — versions on PyPI are immutable, so a wrong number cannot be taken back, only yanked.

Publishing by hand instead

python -m pip install build twine
python -m build
twine check dist/*
twine upload dist/*          # prompts for an API token

Test it against TestPyPI first if you want a dry run: twine upload --repository testpypi dist/*.

Tests

python3 run_tests.py

Eleven tests, no workspace required: manifest loading, notebook cell structure, catalog substitution, tier switching, unresolved-placeholder detection, and a dry_run install. It also asserts the flaw-teaching column comments survive into the generated notebooks — those comments are the curriculum, so losing them in rendering would be a silent failure.

Metadata

Release files for databricks360 0.9.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for databricks360 0.9.1
File Size Uploaded Provenance Actions
databricks360-0.9.1.tar.gz 70.7 kB Without provenance Details for databricks360-0.9.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for databricks360 0.9.1
File Interpreter ABI Platform Actions
databricks360-0.9.1-py3-none-any.whl Python 3 none any Details for databricks360-0.9.1-py3-none-any.whl

Total release size: 138.1 kB

Release files / databricks360-0.9.1.tar.gz

Download URL databricks360-0.9.1.tar.gz
Size 70.7 kB
Tags Source
SHA-256 checksum
How to use checksums
b2ecc2d4142c6f360ac1d31d9ac0bfb2e763776e27c430ebf48f56f7fb8f9682
BLAKE2b-256 checksum
How to use checksums
a3eea1a447a62ad9d019121a83f514dd069d08b2388fbb33a4c2a35ccf258dfe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.6

Release files / databricks360-0.9.1-py3-none-any.whl

Download URL databricks360-0.9.1-py3-none-any.whl
Size 67.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
47c445d3e21cf1fe4d7c90eb484b2ba43dd6171ca9b44223f9295a9711bcb10d
BLAKE2b-256 checksum
How to use checksums
a9f895ac85a6b9108da59c36f5365be0bd1c31497442db7e746728ece1279aa4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.9.6

Security for databricks360 0.9.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.9.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page