Skip to main content

dbx-flame

A configuration-driven data engineering framework for Databricks. Onboarding a new dataset means writing a workflow YAML — never Python.

PyPI Status CI python Databricks Runtime DQX License

Why

Most Databricks pipelines grow one notebook per dataset. This framework inverts that: a single typed engine reads a workflow YAML and carries every dataset through the same five layers — Start, Pipeline, Typing, Policies, Output. A new source is a new YAML file, not a new code path.

The project is alpha — see Status for what's implemented today.

Quick Look

Two tasks, two YAML blocks — CSV into Bronze, then Bronze into Silver with SCD Type 1 upsert on CLAIM_ID:

# Inbound → Bronze
source.origin: csv
source.path: /Volumes/dev/source_data/inbound/
source.directory: CLAIMS
output.verb: append
output.schema_name: claims
output.table: CLAIMS_BRONZE

# Bronze → Silver
source.origin: table
source.schema_name: claims
source.table: CLAIMS_BRONZE
output.verb: upsert
output.schema_name: claims
output.table: CLAIMS_SILVER
output.keys: CLAIM_ID
output.event_time.column: __EXPORT_DATE

No code changes for either step — both are entries in a workflow YAML deployed through the bundle. See 03_write_verbs.md for the full verb matrix.

The source, increment strategy (source.increment_strategy) and verb to use at each step. Every other combination, and the traps a valid config doesn't rule out, are in 08_usage.md.

Step Source Strategy Verb Why
Inbound → Bronze Files (csv, json) checkpoint APPEND Bronze keeps every export exactly as it arrived
External table → Bronze A table we didn't create (e.g. a federated one) full_read APPEND One complete snapshot per run. Sees deletes
delta_read APPEND Only the rows changed since the last run, for a table too large to read whole. Deletes are invisible
Bronze → Silver A table we stamped checkpoint FULL The newest export is the current state
checkpoint UPSERT Latest version per key (SCD1)
checkpoint · watermark SCD2 History of changes
watermark COMPLETE_DELTA, snapshot_scope: delta Replays every export in order; a deletes feed retires keys

Key Capabilities

  • Config-driven onboarding — declare an origin, a write verb and a target; the framework validates the combination and runs it. No per-dataset Python.
  • Data handling verbs — APPEND, FULL, UPSERT, SCD2, COMPLETE_DELTA — layer-agnostic, so the same mechanics serve Inbound → Bronze or Bronze → Silver.
  • Quality gates — every batch is checked against a Databricks DQX ruleset you define before any data is propagated; error refuses the batch, warn logs and lets it through.
  • Fail fast, fail loud — Invalid Configs or Failed Data QA stops the run and logs the issue. Nothing logs an error and reports success.

Documentation

Start with the quick start: the catalogs, volumes and naming rules the framework needs. docs/README.md also lists every document.

Development

Targets Linux; on Windows, work inside WSL, since Spark doesn't run natively there (Required for running unit tests).

Needs a JDK (11 or 17, for PySpark), Python 3.11 (matches Databricks Runtime 15.4 LTS) and Poetry 2.x.

poetry env use python3.11   # once, to pin the interpreter
make install                # runtime + dev dependencies
make test                   # unit tests with coverage

Review the Makefile to lists the rest.

Platform tests are real Databricks jobs under platform_tests/ - with real life scenarios; see 05_testing.md.

Deploying to Databricks

The bundle in databricks.yml is configured to work with a Python wheel: it builds the wheel, then deploys the bundle. Authenticate with a CLI profile or with DATABRICKS_HOST / DATABRICKS_TOKEN. A new workspace first needs its catalogs, schemas and volumes; see the quick start.

databricks bundle deploy

Traceability

Every run that passes config validation writes an audit trail to your own catalog (monitoring_{env}.audit.flame_logs), not to vendor telemetry. Each row names the job, run and task that produced it. The audit write comes first, so a run that can't log fails before touching any data. Schema: 00_overview.md.

Status

Start, Pipeline (CSV + JSON + table), Typing, Policies (DQX), and all five write verbs are implemented and tested. The export task writes a schema's objects to CSV, leaving out or renaming columns, with a checksum per file and a manifest, and delivers them as a zip or loose files. The SAS origin and the Gold example view are not implemented yet. Full phase-by-phase status: 06_roadmap.md.

License

Apache 2.0 — see LICENSE.

Metadata

Release files for dbx-flame 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dbx-flame 0.2.0
File Size Uploaded Provenance Actions
dbx_flame-0.2.0.tar.gz 58.5 kB With provenance Details for dbx_flame-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dbx-flame 0.2.0
File Interpreter ABI Platform Actions
dbx_flame-0.2.0-py3-none-any.whl Python 3 none any Details for dbx_flame-0.2.0-py3-none-any.whl

Total release size: 137.8 kB

Release files / dbx_flame-0.2.0.tar.gz

Download URL dbx_flame-0.2.0.tar.gz
Size 58.5 kB
Tags Source
SHA-256 checksum
How to use checksums
8f450ea159ca699ccda55056f944ad3ad1eb6a9b8fe96332da76a77af796a027
BLAKE2b-256 checksum
How to use checksums
ce95369221ae6fdca09744071680899f5dd4b007ddaa9b2c835a821a47c66aa3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.

Transparency log

Release files / dbx_flame-0.2.0-py3-none-any.whl

Download URL dbx_flame-0.2.0-py3-none-any.whl
Size 79.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
fe871a8566d70f90019a107a437315589f691c0108b734bb7b41fd7b992ff862
BLAKE2b-256 checksum
How to use checksums
60bdfbcc088714825369f48f040ed904ca898b813fc558751e92ae6e2c198588
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.

Transparency log

Security for dbx-flame 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for dbx_flame-0.2.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 28, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page