Skip to main content

dbx-flame

A configuration-driven framework that loads governed, traceable data into Databricks. You onboard a dataset with a workflow YAML, not with Python.

PyPI Status CI python Databricks Runtime DQX License

Why

dbx-flame builds the data foundation that the "data intelligence" sits on: governed, traceable Bronze and Silver tables in Unity Catalog. Genie, agents and AI functions only work as well as the tables and metadata under them.

Most Databricks pipelines grow one notebook per dataset. dbx-flame uses one typed engine instead. It reads a workflow YAML and moves each dataset through the same five layers: Start, Pipeline, Typing, Policies, Output.

The project is beta. See Status.

Quick Look

Two tasks: CSV files into Bronze, then Bronze into Silver with an SCD Type 1 upsert on CLAIM_ID.

# Inbound → Bronze
source.origin: csv
source.path: /Volumes/dev/source_data/inbound/
source.directory: CLAIMS
output.verb: append
output.schema_name: claims
output.table: CLAIMS_BRONZE

# Bronze → Silver
source.origin: table
source.schema_name: claims
source.table: CLAIMS_BRONZE
output.verb: upsert
output.schema_name: claims
output.table: CLAIMS_SILVER
output.keys: CLAIM_ID
output.event_time.column: __EXPORT_DATE

Both tasks are entries in a workflow YAML that the bundle deploys. All verbs: 03_write_verbs.md.

The possible data flows. Each number is one task. Shortcuts 4 and 5 skip Bronze, so no original record is kept. Which combinations each flow accepts: docs/examples.

Data flows from Inbound files and external tables to Bronze, Silver and Gold

What Every Table Gets

  • Checked configuration. Start checks each task before it reads data. An invalid setting, or one with no effect, stops the run.
  • Quality gates. A Databricks DQX ruleset checks each batch before the framework writes it. An error rule stops the batch. A warn rule logs the failure and lets the batch continue. A run never logs an error and reports success.
  • Row-level provenance. The framework sets __SOURCE, __EXPORT_DATE and a __RECORD_HASH of the content when data enters, and never changes them. Each Silver row traces back to the export it came from, and a new version opens only when content changed.
  • An audit trail in your catalog. Each run that passes Start writes to monitoring_{env}.audit.flame_logs. Each row names its job, run and task. A run that cannot log stops before it touches data. Schema: 00_overview.md. Each audit event also prints to the driver's stdout. If the cluster sends stdout to CloudWatch (for example, with the CloudWatch agent in an init script), the events also appear there.
  • One set of write verbs. APPEND, FULL, UPSERT, SCD2 and COMPLETE_DELTA work the same for Inbound → Bronze and Bronze → Silver.

The source, increment strategy (source.increment_strategy) and verb for each step:

Step Source Strategy Verb Why
Inbound → Bronze Files (csv, json) checkpoint APPEND Bronze keeps each export exactly as it arrived
External table → Bronze A table we didn't create (e.g. a federated one) full_read APPEND One complete snapshot per run. Sees deletes
delta_read APPEND Only the rows changed since the last run, for a table too large to read whole. Misses deletes
Bronze → Silver A table we stamped checkpoint FULL The newest export is the current state
checkpoint UPSERT Latest version per key (SCD1)
checkpoint · watermark SCD2 History of changes
watermark COMPLETE_DELTA, snapshot_scope: delta Replays each export in order. A deletes feed retires keys

Comprehensive list of combinations:

Documentation

Start with the quick start: the catalogs, volumes and naming rules that the framework needs. docs/README.md lists every document.

Development

The project targets Linux. On Windows, use WSL, because Spark does not run natively there. You need a JDK (11 or 17, for PySpark), Python 3.11 (the version in Databricks Runtime 15.4 LTS) and Poetry 2.x.

poetry env use python3.11   # once, to pin the interpreter
make install                # runtime + dev dependencies
make test                   # unit tests with coverage

The Makefile lists the other targets. Platform tests are real Databricks jobs that run real-life scenarios: platform_tests/ and 05_testing.md.

Deploying to Databricks

The bundle in databricks.yml builds the Python wheel, then deploys it. Authenticate with a CLI profile or with DATABRICKS_HOST / DATABRICKS_TOKEN. A new workspace first needs the setup in the quick start.

databricks bundle deploy

Status

Implemented and tested: Start, Pipeline (CSV, JSON and table), Typing, Policies (DQX) and all five write verbs. Each table's table_config file keeps its casts, and its Unity Catalog comments and tags (02_config_schema.md). The export task writes a schema's objects to CSV files, with a checksum per file and a manifest (09_export.md). Not implemented yet: the SAS origin and the Gold example view. Status per phase: 06_roadmap.md.

License

Apache 2.0. See LICENSE.

Metadata

Release files for dbx-flame 0.2.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dbx-flame 0.2.3
File Size Uploaded Provenance Actions
dbx_flame-0.2.3.tar.gz 63.9 kB With provenance Details for dbx_flame-0.2.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dbx-flame 0.2.3
File Interpreter ABI Platform Actions
dbx_flame-0.2.3-py3-none-any.whl Python 3 none any Details for dbx_flame-0.2.3-py3-none-any.whl

Total release size: 148.7 kB

Release files / dbx_flame-0.2.3.tar.gz

Download URL dbx_flame-0.2.3.tar.gz
Size 63.9 kB
Tags Source
SHA-256 checksum
How to use checksums
973342daf1e5253afa1e237f06c023b29bd047b7197502473734664a5d9479ab
BLAKE2b-256 checksum
How to use checksums
fe134e7529384bef823f41856693b0371c0d96e8879f8ec25d2bd6cafbc2936c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.

Transparency log

Release files / dbx_flame-0.2.3-py3-none-any.whl

Download URL dbx_flame-0.2.3-py3-none-any.whl
Size 84.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2eb600eb36d43008614260beee34e913ab11368f7886c083dbba44ea8ea102d6
BLAKE2b-256 checksum
How to use checksums
75b2a173b88a40dd54bdb97ed1e9c85dbadb4ecc10259cbda6e3498fd8f6d9fe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.

Transparency log

Security for dbx-flame 0.2.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for dbx_flame-0.2.3.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 4, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page