databricks360
Installs Databricks course lab environments — notebooks, catalogs, datasets and governance objects — into your own workspace.
Modelled on dbdemos: you run it inside a Databricks notebook, so
databricks-sdk picks up the notebook's own identity. There is no host, token or
profile to configure.
Install
Not on PyPI yet, so install from the repo. In a Databricks notebook:
%pip install git+https://github.com/databrickslms/dbxdemos.git
dbutils.library.restartPython()
Once published, that becomes %pip install databricks360.
To pin a version, append a tag or commit:
%pip install git+https://github.com/databrickslms/dbxdemos.git@v0.1.0
Usage
import databricks360 as academy
academy.list_courses()
academy.install('genie-agents')
The bare call uses the course's own naming — a single genie_agent schema in your
current catalog, with objects named mfg_core_*, mfg_ref_*, mfg_staging_*:
genie_agent.mfg_core_dim_date
genie_agent.mfg_core_fct_transactions
genie_agent.mfg_ref_documents
Override with schema=, table_prefix= or catalog= — see Regulated
environments below.
install writes the lab notebooks into your workspace and prints the run order.
Then you open them and run each in turn.
Installed 'genie-agents' → /Workspace/Users/you@corp.com/databricks360/genie-agents
catalog: mfg tier: small
Run these in order:
1. 01_catalog_and_schemas
2. 02_dimensions
3. 03_facts
Documents
Module 3's Agent-mode exercises read unstructured files alongside the tables:
academy.create_documents('genie-agents')
Forty Meridian documents — investment committee memos, advisor call notes and
client complaint resolutions — written into the course's Unity Catalog volume.
Each restates a planted flaw in the words someone would use in a meeting, so
answering a question means reconciling prose against numbers rather than reading
one or the other. PDFs where fpdf is installed, plain text otherwise.
Genie Agents
Courses that need Genie Agents ship them as serialized_space definitions, so an
agent is never built by hand:
academy.create_agents('genie-agents', dry_run=True) # check, create nothing
academy.create_agents('genie-agents') # create them
Notebook 08_agents does exactly this, so it is usually easier to run that in
order with the rest. Object names come from the same layout as the notebooks, so
the agents follow whichever catalog and schema you installed into.
This is the one call that creates something without you running a notebook. It is still cheap: an agent is metadata, and nothing touches a warehouse until someone asks it a question. It refuses to create an agent whose tables do not exist yet, because an agent pointed at a missing table fails on every question and the failure looks like a Genie problem rather than a notebook you skipped.
Options
academy.install(
'genie-agents',
path='/Workspace/Shared/labs', # default: your home folder
catalog='training_v2', # default: the course's own catalog
tier='large', # default: 'small'
overwrite=True, # replace existing notebooks
)
Why it does not run the notebooks for you
dbdemos starts a job and loads the data on your behalf. This deliberately does
not. Generating the data is the substance of Module 0 — the point is to watch a
warehouse chew through 20M rows and see the flaws appear, not to have a finished
catalog materialise. It also means nothing consumes your DBUs without you asking.
Regulated environments
A locked-down Unity Catalog is normal in a governed organisation, so the default
assumes the least privilege that can still work: the lab lands in whatever
current_catalog() returns, and nothing is created above schema level.
Notebook 01 opens by showing you exactly where that is:
SELECT current_catalog() AS default_catalog, current_schema() AS default_schema;
The four shapes
# 1. Default — current catalog, schemas core / ref / staging.
# No CREATE CATALOG attempted. Table names are two-level (core.dim_date),
# so SQL resolves the catalog itself.
academy.install('genie-agents')
# 2. A specific catalog you were granted. Adds USE CATALOG, still creates no catalog.
academy.install('genie-agents', catalog='main')
# 3. Create a catalog — opt in, only if you hold the privilege.
academy.install('genie-agents', catalog='mfg', create_catalog=True)
# 4. One schema you already own. Everything lands there.
academy.install('genie-agents', schema='training_you')
academy.install('genie-agents', catalog='main', schema='training_you')
create_catalog is off by default and raises if you pass it without a catalog
name — current_catalog() exists by definition, so there would be nothing to create.
Passing schema assumes you cannot create that schema either, since that is the
reason to reach for it. Override with create_schema=True if you can.
Which of these have actually been run
The spoiler sweep renders all six shapes at build time, but rendering is not running. These have been executed end to end against a real workspace:
| Layout | Executed |
|---|---|
Course default — one schema, mfg_ prefix |
all nine notebooks |
Multi-schema — core / ref / staging, no prefix |
all nine notebooks |
| Single schema, named catalog, no prefix | schemas + dimensions |
create_catalog=True |
catalog, schemas + dimensions |
scripts/run_lab.py is what does it — it executes a notebook's SQL against a workspace and
reports the first failure in each file, which is how six bugs in 06_curated and
07_metric_view were found after they had shipped.
One schema shared with other content
If your schema also holds other things, prefix the object names so their provenance is visible:
academy.install('genie-agents', schema='genie_agent', table_prefix='mfg_')
The logical group folds into the name:
| Default | With table_prefix='mfg_' |
|---|---|
core.dim_date |
genie_agent.mfg_core_dim_date |
ref.documents |
genie_agent.mfg_ref_documents |
staging.fct_txn_legacy |
genie_agent.mfg_staging_fct_txn_legacy |
The prefix names objects, never the schema. table_prefix requires schema —
without one, the schemas already namespace the objects and a prefix would just be
noise.
Worth weighing before you use it. Genie reads table and column names as context, and Module 7 of the course is partly about removing noise so an agent has less to wade through.
mfg_core_dim_dateis measurably noisier thandim_date. Use a prefix when the schema is genuinely shared and provenance matters more; skip it when the schema is yours.
No CREATE VOLUME either
academy.install('genie-agents', schema='training_you', create_volume=False)
The volume only serves Agent mode over unstructured files (Modules 3 and 16). Skipping it costs those exercises and nothing else, and the notebook says so.
Why the single-schema case renames nothing
core holds the dimensions and facts, ref holds only the documents volume, and
staging holds only the two decoy tables. The three sets of object names do not
overlap, so collapsing them into one schema is safe — and notebooks 02 onward
resolve their own paths, so they need no changes whichever layout you chose.
What you get back
install reports the resolved layout and anything it declined to create, so a
restricted install is visible rather than quietly partial:
Installed 'genie-agents' → /Workspace/Users/you@corp.com/databricks360/genie-agents
tier: small
single schema: current_catalog().training_you
skipping: CREATE SCHEMA, CREATE VOLUME
Unity Catalog is required. Genie Agents read Unity Catalog objects, so
hive_metastorewill not work as the target catalog.
Tiers
| Tier | Transactions | Use |
|---|---|---|
small |
20M | default. Every module except 13 |
large |
900M | Module 13 (latency) only. Left unclustered on purpose |
Start small. The large tier exists because you cannot measure query latency on a toy dataset, and nowhere else needs it.
Adding a course
Each course is a subpackage under databricks360/courses/:
databricks360/courses/<course_id>/
__init__.py
manifest.json # title, default catalog, tiers, notebooks in run order
*.sql # synced from content/courses/<id>/assets/lab/
Placeholders available in the SQL: {{CATALOG}}, plus anything declared under a
tier's values (currently {{TXN_COUNT}}). An unresolved placeholder raises rather
than silently rendering empty.
Dataset documentation lives in docs/.
Publishing
Releases go to PyPI via Trusted Publishing — GitHub Actions authenticates to PyPI with a short-lived OIDC identity, so no API token exists in repo secrets or on anyone's laptop. A leaked token is the usual way a package supply chain gets compromised; the safest token is one that was never created.
One-time PyPI setup
-
Sign in at pypi.org → Your account → Publishing
-
Under Add a new pending publisher, choose GitHub and enter exactly:
Field Value PyPI Project Name databricks360Owner databrickslmsRepository name dbxdemosWorkflow name publish.ymlEnvironment name pypi -
In GitHub → Settings → Environments → New environment → name it
pypi. Add yourself as a required reviewer if you want to approve each release.
"Pending" publisher is correct — the project does not exist on PyPI yet, and the first successful run creates it.
Cutting a release
# bump version in pyproject.toml, commit, then:
git tag v0.1.0
git push origin v0.1.0
The workflow runs the tests, builds, checks the tag matches pyproject.toml, and
publishes. A mismatched tag fails before anything reaches the index — versions on
PyPI are immutable, so a wrong number cannot be taken back, only yanked.
Publishing by hand instead
python -m pip install build twine
python -m build
twine check dist/*
twine upload dist/* # prompts for an API token
Test it against TestPyPI first if you want a dry run:
twine upload --repository testpypi dist/*.
Tests
python3 run_tests.py
Eleven tests, no workspace required: manifest loading, notebook cell structure,
catalog substitution, tier switching, unresolved-placeholder detection, and a
dry_run install. It also asserts the flaw-teaching column comments survive into
the generated notebooks — those comments are the curriculum, so losing them in
rendering would be a silent failure.
Metadata
Release files for databricks360 0.13.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| databricks360-0.13.2.tar.gz | 91.3 kB | Without provenance | Details for databricks360-0.13.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| databricks360-0.13.2-py3-none-any.whl | Python 3 | none | any | Details for databricks360-0.13.2-py3-none-any.whl |
Total release size: 183.3 kB
Release files / databricks360-0.13.2.tar.gz
| Download URL | databricks360-0.13.2.tar.gz |
|---|---|
| Size | 91.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dbf8941f4e3dd1692b755fea2aea51a049e2d32065e4ea326d5240ec209e0117
|
|
BLAKE2b-256 checksum How to use checksums |
e7a15d6ab0e189e85e82244058755bb78330d1dda23c22012b72de391d0c30b7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.6
|
Release files / databricks360-0.13.2-py3-none-any.whl
| Download URL | databricks360-0.13.2-py3-none-any.whl |
|---|---|
| Size | 92.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8a905d5856ce37c942983b2b463ea5d2d9a1eda7ee848c81f0c3a839db50552f
|
|
BLAKE2b-256 checksum How to use checksums |
90f7a5b4fd3608ea798a25a910e65115901554f74b9389cbd9c0fab9b7ab521c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.6
|
Security for databricks360 0.13.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.