dbx-flame
A configuration-driven data engineering framework for Databricks. Onboarding a new dataset means writing a workflow YAML — never Python.
Why
Most Databricks pipelines grow one notebook per dataset. This framework inverts that: a single typed engine reads a workflow YAML and carries every dataset through the same five layers — Start, Pipeline, Typing, Policies, Output. A new source is a new YAML file, not a new code path.
The project is alpha — see Status for what's implemented today.
Quick Look
Two tasks, two YAML blocks — CSV into Bronze, then Bronze into Silver with SCD Type 1
upsert on CLAIM_ID:
# Inbound → Bronze
source.origin: csv
source.path: /Volumes/dev/source_data/inbound/
source.directory: CLAIMS
output.verb: append
output.schema_name: claims
output.table: CLAIMS_BRONZE
# Bronze → Silver
source.origin: table
source.schema_name: claims
source.table: CLAIMS_BRONZE
output.verb: upsert
output.schema_name: claims
output.table: CLAIMS_SILVER
output.keys: CLAIM_ID
output.event_time.column: __EXPORT_DATE
No code changes for either step — both are entries in a workflow YAML deployed through the bundle. See 03_write_verbs.md for the full verb matrix.
Recommended Combinations
The source, increment strategy (source.increment_strategy) and verb to use at each step.
Every other combination, and the traps a valid config doesn't rule out, are in
08_usage.md.
| Step | Source | Strategy | Verb | Why |
|---|---|---|---|---|
| Inbound → Bronze | Files (csv, json) | checkpoint |
APPEND | Bronze keeps every export exactly as it arrived |
| External table → Bronze | A table we didn't create (e.g. a federated one) | full_read |
APPEND | One complete snapshot per run. Sees deletes |
delta_read |
APPEND | Only the rows changed since the last run, for a table too large to read whole. Deletes are invisible | ||
| Bronze → Silver | A table we stamped | checkpoint |
FULL | The newest export is the current state |
checkpoint |
UPSERT | Latest version per key (SCD1) | ||
checkpoint · watermark |
SCD2 | History of changes | ||
watermark |
COMPLETE_DELTA, snapshot_scope: delta |
Replays every export in order; a deletes feed retires keys |
Key Capabilities
- Config-driven onboarding — declare an origin, a write verb and a target; the framework validates the combination and runs it. No per-dataset Python.
- Data handling verbs —
APPEND,FULL,UPSERT,SCD2,COMPLETE_DELTA— layer-agnostic, so the same mechanics serveInbound → BronzeorBronze → Silver. - Quality gates — every batch is checked against a
Databricks DQX ruleset you define before any data is propagated;
errorrefuses the batch,warnlogs and lets it through. - Fail fast, fail loud — Invalid Configs or Failed Data QA stops the run and logs the issue. Nothing logs an error and reports success.
Documentation
Start with the quick start: the catalogs, volumes and naming rules the framework needs. docs/README.md also lists every document.
Development
Targets Linux; on Windows, work inside WSL, since Spark doesn't run natively there (Required for running unit tests).
Needs a JDK (11 or 17, for PySpark), Python 3.11 (matches Databricks Runtime 15.4 LTS) and Poetry 2.x.
poetry env use python3.11 # once, to pin the interpreter
make install # runtime + dev dependencies
make test # unit tests with coverage
Review the Makefile to lists the rest.
Platform tests are real Databricks jobs under platform_tests/ - with real life scenarios; see 05_testing.md.
Deploying to Databricks
The bundle in databricks.yml is configured to work with a Python wheel: it
builds the wheel, then deploys the bundle. Authenticate with a CLI profile or with
DATABRICKS_HOST / DATABRICKS_TOKEN. A new workspace first needs its catalogs, schemas and
volumes; see the quick start.
databricks bundle deploy
Traceability
Every run that passes config validation writes an audit trail to your own catalog
(monitoring_{env}.audit.logs), not to vendor telemetry. Each row names the job, run and task
that produced it. The audit write comes first, so a run that can't log fails before touching
any data. Schema: 00_overview.md.
Status
Start, Pipeline (CSV + JSON + table), Typing, Policies (DQX), and all five write verbs are implemented and tested. The export task writes a schema's objects to CSV, leaving out or renaming columns, with a checksum per file and a manifest, and delivers them as a zip or loose files. The SAS origin and the Gold example view are not implemented yet. Full phase-by-phase status: 06_roadmap.md.
License
Apache 2.0 — see LICENSE.
Metadata
Release files for dbx-flame 0.1.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dbx_flame-0.1.1.tar.gz | 55.7 kB | With provenance | Details for dbx_flame-0.1.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dbx_flame-0.1.1-py3-none-any.whl | Python 3 | none | any | Details for dbx_flame-0.1.1-py3-none-any.whl |
Total release size: 132.0 kB
Release files / dbx_flame-0.1.1.tar.gz
| Download URL | dbx_flame-0.1.1.tar.gz |
|---|---|
| Size | 55.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1074e16ec3a172f637deaa30d8017a854c940ae2f9edbdbbbff094b51434b53f
|
|
BLAKE2b-256 checksum How to use checksums |
def4752edebc54588d0108c75b9aea807bb3dbcf7860c729da2e8fd3606b3a29
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.
Transparency logRelease files / dbx_flame-0.1.1-py3-none-any.whl
| Download URL | dbx_flame-0.1.1-py3-none-any.whl |
|---|---|
| Size | 76.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
db57586b600782ca3e9ba35c106320f9047f915d792709c2f96b39f44422f455
|
|
BLAKE2b-256 checksum How to use checksums |
85ce9ec00d1912a9b704376606e85f5200b1cabb5479ce05963039c560af9446
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.
Transparency logSecurity for dbx-flame 0.1.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for dbx_flame-0.1.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 26, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.