Skip to main content

dbx-flame

A configuration-driven framework that loads governed, traceable data into Databricks. You onboard a dataset with a workflow YAML, not with Python.

PyPI Status CI python Databricks Runtime DQX License

Why

dbx-flame builds the data foundation that the "data intelligence" sits on: governed, traceable Bronze and Silver tables in Unity Catalog. Genie, agents and AI functions only work as well as the tables and metadata under them.

Most Databricks pipelines grow one notebook per dataset. dbx-flame uses one typed engine instead. It reads a workflow YAML and moves each dataset through the same five layers: Start, Pipeline, Typing, Policies, Output.

The project is beta. See Status.

Quick Look

Two tasks: CSV files into Bronze, then Bronze into Silver with an SCD Type 1 upsert on CLAIM_ID.

# Inbound → Bronze
source.origin: csv
source.path: /Volumes/dev/source_data/inbound/
source.directory: CLAIMS
output.verb: append
output.schema_name: claims
output.table: CLAIMS_BRONZE

# Bronze → Silver
source.origin: table
source.schema_name: claims
source.table: CLAIMS_BRONZE
output.verb: upsert
output.schema_name: claims
output.table: CLAIMS_SILVER
output.keys: CLAIM_ID
output.event_time.column: __EXPORT_DATE

Both tasks are entries in a workflow YAML that the bundle deploys. All verbs: 03_write_verbs.md.

The possible data flows. Each number is one task. Shortcuts 4 and 5 skip Bronze, so no original record is kept. Which combinations each flow accepts: docs/examples.

Data flows from Inbound files and external tables to Bronze, Silver and Gold

What Every Table Gets

  • Checked configuration. Start checks each task before it reads data. An invalid setting, or one with no effect, stops the run.
  • Quality gates. A Databricks DQX ruleset checks each batch before the framework writes it. An error rule stops the batch. A warn rule logs the failure and lets the batch continue. A run never logs an error and reports success.
  • Row-level provenance. The framework sets __SOURCE and __EXPORT_DATE when data enters, and never changes them. Each Silver row traces back to the export it came from.
  • An audit trail in your catalog. Each run that passes Start writes to monitoring_{env}.audit.flame_logs. Each row names its job, run and task. A run that cannot log stops before it touches data. Schema: 00_overview.md. Each audit event also prints to the driver's stdout. If the cluster sends stdout to CloudWatch (for example, with the CloudWatch agent in an init script), the events also appear there.
  • One set of write verbs. APPEND, FULL, UPSERT, SCD2 and COMPLETE_DELTA work the same for Inbound → Bronze and Bronze → Silver.

The source, increment strategy (source.increment_strategy) and verb for each step:

Step Source Strategy Verb Why
Inbound → Bronze Files (csv, json) checkpoint APPEND Bronze keeps each export exactly as it arrived
External table → Bronze A table we didn't create (e.g. a federated one) full_read APPEND One complete snapshot per run. Sees deletes
delta_read APPEND Only the rows changed since the last run, for a table too large to read whole. Misses deletes
Bronze → Silver A table we stamped checkpoint FULL The newest export is the current state
checkpoint UPSERT Latest version per key (SCD1)
checkpoint · watermark SCD2 History of changes
watermark COMPLETE_DELTA, snapshot_scope: delta Replays each export in order. A deletes feed retires keys

Comprehensive list of combinations:

Documentation

Start with the quick start: the catalogs, volumes and naming rules that the framework needs. docs/README.md lists every document.

Development

The project targets Linux. On Windows, use WSL, because Spark does not run natively there. You need a JDK (11 or 17, for PySpark), Python 3.11 (the version in Databricks Runtime 15.4 LTS) and Poetry 2.x.

poetry env use python3.11   # once, to pin the interpreter
make install                # runtime + dev dependencies
make test                   # unit tests with coverage

The Makefile lists the other targets. Platform tests are real Databricks jobs that run real-life scenarios: platform_tests/ and 05_testing.md.

Deploying to Databricks

The bundle in databricks.yml builds the Python wheel, then deploys it. Authenticate with a CLI profile or with DATABRICKS_HOST / DATABRICKS_TOKEN. A new workspace first needs the setup in the quick start.

databricks bundle deploy

Status

Implemented and tested: Start, Pipeline (CSV, JSON and table), Typing, Policies (DQX) and all five write verbs. Each table's table_config file keeps its casts, and its Unity Catalog comments and tags (02_config_schema.md). The export task writes a schema's objects to CSV files, with a checksum per file and a manifest (09_export.md). Not implemented yet: the SAS origin and the Gold example view. Status per phase: 06_roadmap.md.

License

Apache 2.0. See LICENSE.

Metadata

Release files for dbx-flame 0.2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dbx-flame 0.2.2
File Size Uploaded Provenance Actions
dbx_flame-0.2.2.tar.gz 61.3 kB With provenance Details for dbx_flame-0.2.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dbx-flame 0.2.2
File Interpreter ABI Platform Actions
dbx_flame-0.2.2-py3-none-any.whl Python 3 none any Details for dbx_flame-0.2.2-py3-none-any.whl

Total release size: 143.3 kB

Release files / dbx_flame-0.2.2.tar.gz

Download URL dbx_flame-0.2.2.tar.gz
Size 61.3 kB
Tags Source
SHA-256 checksum
How to use checksums
7d18c84c34ef5304a97213b189ff599fe13868faa001fdf2650112da414701ed
BLAKE2b-256 checksum
How to use checksums
4fb6e9853d4172d127fd5b3d3e41321baaf60cd8371571305f3d4e0456bcc132
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.

Transparency log

Release files / dbx_flame-0.2.2-py3-none-any.whl

Download URL dbx_flame-0.2.2-py3-none-any.whl
Size 82.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a07c37d8c8041a62ca759891b0ee2a4fd064e09784bdc3104b4ea9986b7649ef
BLAKE2b-256 checksum
How to use checksums
df45b77487b02c669efd2b1d5319425291e86e15c4c9ec223877b7eb9441fee0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.

Transparency log

Security for dbx-flame 0.2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for dbx_flame-0.2.2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 30, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page