dbx-flame 
A configuration-driven framework that loads governed, traceable data into Databricks. You onboard a dataset with a workflow YAML, not with Python.
Why
dbx-flame builds the data foundation that the "data intelligence" sits on: governed, traceable Bronze and Silver tables in Unity Catalog. Genie, agents and AI functions only work as well as the tables and metadata under them.
Most Databricks pipelines grow one notebook per dataset. dbx-flame uses one typed engine instead. It reads a workflow YAML and moves each dataset through the same five layers: Start, Pipeline, Typing, Policies, Output.
The project is beta. See Status.
Quick Look
Two tasks: CSV files into Bronze, then Bronze into Silver with an SCD Type 1 upsert on
CLAIM_ID.
# Inbound → Bronze
source.origin: csv
source.path: /Volumes/dev/source_data/inbound/
source.directory: CLAIMS
output.verb: append
output.schema_name: claims
output.table: CLAIMS_BRONZE
# Bronze → Silver
source.origin: table
source.schema_name: claims
source.table: CLAIMS_BRONZE
output.verb: upsert
output.schema_name: claims
output.table: CLAIMS_SILVER
output.keys: CLAIM_ID
output.event_time.column: __EXPORT_DATE
Both tasks are entries in a workflow YAML that the bundle deploys. All verbs: 03_write_verbs.md.
The possible data flows. Each number is one task. Shortcuts 4 and 5 skip Bronze, so no original record is kept. Which combinations each flow accepts: docs/examples.
What Every Table Gets
- Checked configuration. Start checks each task before it reads data. An invalid setting, or one with no effect, stops the run.
- Quality gates. A Databricks DQX ruleset checks
each batch before the framework writes it. An
errorrule stops the batch. Awarnrule logs the failure and lets the batch continue. A run never logs an error and reports success. - Row-level provenance. The framework sets
__SOURCE,__EXPORT_DATEand a__RECORD_HASHof the content when data enters, and never changes them. Each Silver row traces back to the export it came from, and a new version opens only when content changed. - An audit trail in your catalog. Each run that passes Start writes to
monitoring_{env}.audit.flame_logs. Each row names its job, run and task. A run that cannot log stops before it touches data. Schema: 00_overview.md. Each audit event also prints to the driver's stdout. If the cluster sends stdout to CloudWatch (for example, with the CloudWatch agent in an init script), the events also appear there. - One set of write verbs.
APPEND,FULL,UPSERT,SCD2andCOMPLETE_DELTAwork the same forInbound → BronzeandBronze → Silver.
Recommended Usage
The source, increment strategy (source.increment_strategy) and verb for each step:
| Step | Source | Strategy | Verb | Why |
|---|---|---|---|---|
| Inbound → Bronze | Files (csv, json) | checkpoint |
APPEND | Bronze keeps each export exactly as it arrived |
| External table → Bronze | A table we didn't create (e.g. a federated one) | full_read |
APPEND | One complete snapshot per run. Sees deletes |
delta_read |
APPEND | Only the rows changed since the last run, for a table too large to read whole. Misses deletes | ||
| Bronze → Silver | A table we stamped | checkpoint |
FULL | The newest export is the current state |
checkpoint |
UPSERT | Latest version per key (SCD1) | ||
checkpoint · watermark |
SCD2 | History of changes | ||
watermark |
COMPLETE_DELTA, snapshot_scope: delta |
Replays each export in order. A deletes feed retires keys |
Comprehensive list of combinations:
Documentation
Start with the quick start: the catalogs, volumes and naming rules that the framework needs. docs/README.md lists every document.
Development
The project targets Linux. On Windows, use WSL, because Spark does not run natively there. You need a JDK (11 or 17, for PySpark), Python 3.11 (the version in Databricks Runtime 15.4 LTS) and Poetry 2.x.
poetry env use python3.11 # once, to pin the interpreter
make install # runtime + dev dependencies
make test # unit tests with coverage
The Makefile lists the other targets. Platform tests are real Databricks jobs that run real-life scenarios: platform_tests/ and 05_testing.md.
Deploying to Databricks
The bundle in databricks.yml builds the Python wheel, then deploys it.
Authenticate with a CLI profile or with DATABRICKS_HOST / DATABRICKS_TOKEN. A new
workspace first needs the setup in the quick start.
databricks bundle deploy
Status
Implemented and tested: Start, Pipeline (CSV, JSON and table), Typing, Policies (DQX) and all
five write verbs. Each table's table_config file keeps its casts, and its Unity Catalog
comments and tags (02_config_schema.md). The export task writes a schema's objects to CSV files, with a checksum per
file and a manifest (09_export.md). Not implemented yet: the SAS origin
and the Gold example view. Status per phase: 06_roadmap.md.
License
Apache 2.0. See LICENSE.
Metadata
Release files for dbx-flame 0.2.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dbx_flame-0.2.3.tar.gz | 63.9 kB | With provenance | Details for dbx_flame-0.2.3.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dbx_flame-0.2.3-py3-none-any.whl | Python 3 | none | any | Details for dbx_flame-0.2.3-py3-none-any.whl |
Total release size: 148.7 kB
Release files / dbx_flame-0.2.3.tar.gz
| Download URL | dbx_flame-0.2.3.tar.gz |
|---|---|
| Size | 63.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
973342daf1e5253afa1e237f06c023b29bd047b7197502473734664a5d9479ab
|
|
BLAKE2b-256 checksum How to use checksums |
fe134e7529384bef823f41856693b0371c0d96e8879f8ec25d2bd6cafbc2936c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.
Transparency logRelease files / dbx_flame-0.2.3-py3-none-any.whl
| Download URL | dbx_flame-0.2.3-py3-none-any.whl |
|---|---|
| Size | 84.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2eb600eb36d43008614260beee34e913ab11368f7886c083dbba44ea8ea102d6
|
|
BLAKE2b-256 checksum How to use checksums |
75b2a173b88a40dd54bdb97ed1e9c85dbadb4ecc10259cbda6e3498fd8f6d9fe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.
Transparency logSecurity for dbx-flame 0.2.3
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for dbx_flame-0.2.3.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.