SQLBuild

Verify early. Test properly. Deploy reversibly. SQL pipelines with the rigor of real software.

Valid isn't the same as correct. Your SQL compiles, runs, and returns rows; none of that means the number is right, and a silently-wrong number a stakeholder already trusted is the bug that actually hurts.

SQLBuild brings software-engineering rigor to SQL pipelines: catch errors before the warehouse runs them, test your logic locally, and opt into change-aware execution when you need it. It is a standalone, open-source framework for building SQL and Python data pipelines.

All state is persisted as append-only tables in the warehouse alongside your data: no external state database, no manifest files, no paid add-on. Start with straightforward SQL models, then add ingestion, Python nodes, and opt-in virtual environments as your project grows.

Key features

  • Test your logic, not just your columns. Multi-model SQL tests resolve every intermediate model from its real SQL, plus end-to-end scenarios with local DuckDB replay for fast CI with no warehouse. Catch wrong logic before it ships, not just nulls.
  • Verify early. Define models as SQL files with MODEL() headers. SQLBuild resolves references, validates SQL, infers columns, checks contracts, and computes column lineage before anything runs, all offline. It fails at compile, not halfway through a warehouse run.
  • Fast and open static analysis. SQL parsing, validation, column inference, lineage, and transpilation run on Polyglot, a Rust SQL engine (MIT, 32+ dialects), so compile stays fast on large projects. The analysis is part of the Apache-2.0 core: no proprietary engine, no login, no paid tier.
  • Audits that block bad data. Audits run before data reaches the target table. Full table builds materialize into a staging table and only promote if audits pass; incremental models validate each batch before DML.
  • Deploy reversibly (opt-in). Virtual environments add instant low-copy branching, partial promotion, rollback, checkpoints, and reconciliation. Opt-in, not a tax you pay upfront.
  • Opt-in change-aware execution. Models, seeds, UDFs, and Python nodes are fingerprinted, and source freshness is tracked. In virtual environments, pass --changes-only or set changes_only = true to skip work that is already current; commands otherwise run the full selected scope.
  • Warehouse-native state. All change-tracking state lives in append-only tables (_sqlbuild_fingerprints, _sqlbuild_source_freshness, _sqlbuild_node_results) in your warehouse schemas. No external state machine, no corruption risk.
  • Cursor-based incremental processing. Automatic gap detection and resume, with microbatch mode for large ranges. No external checkpoint to maintain.
  • Ingestion and Python nodes. Load external data with Python @loader functions, and run @task, @asset, and @check nodes as first-class members of the same DAG as your SQL models.

See the documentation for the full feature set, including providers, lifecycle hooks, Python macros, UDFs, custom materializations, data diffs, zero-copy cloning, and virtual environments. To coordinate dbt and SQLBuild projects, see the dbt compatibility guide.

Quick start

pip install sqlbuild
# or
uv pip install sqlbuild

Create and run the included playground project:

sqb playground waffle-shop
cd waffle-shop
sqb plan
sqb build
sqb test

Example

A model is a SQL file with a MODEL() header and a SELECT. References use __ref() and __source(), and configuration, schema, and audits are declared inline:

MODEL (
  materialized table,
  columns (
    order_id (audits [not_null, unique]),
  ),
  tags [marts],
);

SELECT
  o.order_id,
  o.customer_id,
  p.amount_cents AS total_cents
FROM __ref("stg_orders") o
JOIN __ref("stg_payments") p USING (order_id)

A unit test mocks sources and asserts on the model, resolving every intermediate model automatically:

TEST();

WITH
__source__raw__orders AS (
  @mock_orders()
),
__source__raw__payments AS (
  SELECT
    1 AS payment_id,
    1 AS order_id,
    1500 AS amount_cents,
    'credit_card' AS method
),
__expected__fact_orders AS (
  SELECT 1 AS order_id, 100 AS customer_id, 1500 AS total_cents
)
SELECT 1

See the documentation for incremental models, scenarios, loaders, and more.

Python project layout

Project-owned Python must live in a supported extension location such as factories/, libs/, macros/, providers/, or another documented Python resource root. Factory locations contain normal Python: constants, classes, undecorated helper functions, and modules such as _helpers.py are allowed, while decorators determine which functions become SQLBuild resources. Compilation rejects Python under invented project roots so indirectly importable modules cannot create an unofficial project structure. Keep repository pytest tests outside the SQLBuild project's tests/ directory, which is reserved for SQLBuild SQL tests and scenarios. Documented integration paths such as dagster/, rivers_pipeline/, and their definitions.py modules are also supported.

Kata SQL architecture checks

Kata is SQLBuild's opt-in, error-only SQL model shape checker. It runs offline over the compiled project, reports coded faults with remediations, and never rewrites SQL. Its built-in lifecycle is native: Rust resolves rule policy, parses each model, evaluates built-ins, applies suppressions, and owns the persistent cache and deterministic result ordering.

Kata is disabled until the project selects at least one rule. Select the complete built-in policy in sqlbuild_project.toml with its namespace prefix:

[kata]
select = ["SQBK"]

SQBK activates every built-in rule. Narrower prefixes such as SQBKS activate one family, exact codes select individual rules, and ignore removes matching rules. Audit, unit-test, and custom-rule test-case minimums each default to one and can be overridden under [kata.thresholds].

Kata also keeps model ownership shallow and explicit. Configured level paths separate warehouse layers from domain ownership; every owner is a leaf or a branch, subdomain depth defaults to one, and declaration roles remain bounded flat-or-grouped containers:

[kata.layout]
levels = ["staging", "intermediate/clean", "intermediate/enriched", "mart"]
domain_roots = ["market/betfair", "model/horsenet/ratings"] # optional disambiguation

[kata.thresholds]
max_subdomain_depth = 1
min_shared_owner_prefix_directories = 2

Run sqb kata, inspect metadata with sqb kata rule SQBKS101, and generate agent guidance from the same active ruleset with sqb kata skills. Use sqb kata skills --check in CI to detect stale guidance. --json, --select, and --exclude are available for automation and model scoping.

Repository rules use the public API:

from sqlbuild.kata import RuleContext, kata


@kata(
    code="XSQBKP001",
    family="prices",
    slug="typed-currency",
    message="price models must declare a currency column",
    remediation="Declare currency in the MODEL columns contract at this model path.",
)
def typed_currency(*, model, ctx: RuleContext):
    return [] if any(column.name == "currency" for column in ctx.declared_columns) else [
        ctx.path_fault()
    ]

Load repository-owned files through rule_paths = ["kata/rules"] or dotted packages through rule_modules. Test each custom rule with RuleCase and evaluate_rule. Selecting custom rules disables caching unless [kata.cache] require_cacheable = true; cacheable rules may import only the supported pure modules and must access project files through RuleContext.

Python is used only for the SQLBuild compiler adapter and selected custom rules. Built-in-only runs cross into the native engine once as a compiled model batch and do not materialize or walk Python AST objects. A selected custom rule can still use the public RuleContext and raw Polyglot AST escape hatch; its findings rejoin native suppression, ordering, and cache policy.

Exact rule_exceptions require a rule, file, and reason and fail when stale. Broader rule_ignores and lone-star allowances also require reasons but are intentionally not stale-checked.

Supported adapters

Adapter Status
DuckDB Supported
MotherDuck Supported
Snowflake Supported
BigQuery Supported
Databricks Supported
PostgreSQL Supported
SQL Server Supported

ClickHouse, Redshift, Trino, Spark, and Athena are on the way.

Snowflake cost estimates

Native Snowflake builds automatically show a compact per-run busy-compute estimate. SQLBuild attributes visible overlapping query intervals fairly across active queries, converts attributed seconds using the warehouse-size credit rate, and estimates USD from the configured rate:

[cost]
usd_per_credit = 3.00

The default is 3.00 USD per credit and is visibly marked as a default. Configure the value with your Snowflake contract rate. Use sqb cost, sqb cost latest, sqb cost <run_id>, or sqb cost history --since 7d to inspect persisted records. --json and --json-output PATH provide a versioned, decimal-safe output contract. Pending detail records are refreshed from Snowflake when inspected again.

These values are attributed compute credits and estimated cost, not Snowflake-billed credits or invoice reconciliation. The estimate uses only query history visible to the executing role and does not reconstruct invisible concurrent work, warehouse resume or idle tail, the 60-second minimum, cloud-services credits, contract adjustments, or multi-cluster billing. Run metadata and query IDs are stored under target/executions/<run_id>/; that statement ledger stores only an SQL digest, not SQL text. Executed SQL artifacts are stored separately under the sensitive target/run/ tree.

Documentation

Full documentation is available at docs.sqlbuild.com.

Runtime operator and extension contracts:

Contributing

We welcome contributions. Please see CONTRIBUTING.md for guidelines.

License

SQLBuild is licensed under the Apache License 2.0.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.87.3)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for kvlonge from gravatar.com kvlonge

Requires

Python >=3.12

Provides Extra

bigquery dagster databricks dbt dlt ingestr postgres rivers snowflake sqlserver

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sqlbuild-0.85.0.tar.gz (1.5 MB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

sqlbuild-0.85.0-cp312-abi3-win_amd64.whl (10.5 MB view details)

Uploaded CPython 3.12+Windows x86-64

sqlbuild-0.85.0-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.3 MB view details)

Uploaded CPython 3.12+manylinux: glibc 2.17+ x86-64

sqlbuild-0.85.0-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (10.8 MB view details)

Uploaded CPython 3.12+manylinux: glibc 2.17+ ARM64

sqlbuild-0.85.0-cp312-abi3-macosx_11_0_arm64.whl (10.3 MB view details)

Uploaded CPython 3.12+macOS 11.0+ ARM64

sqlbuild-0.85.0-cp312-abi3-macosx_10_12_x86_64.whl (10.4 MB view details)

Uploaded CPython 3.12+macOS 10.12+ x86-64

File details

Details for the file sqlbuild-0.85.0.tar.gz.

File metadata

  • Download URL: sqlbuild-0.85.0.tar.gz
  • Upload date:
  • Size: 1.5 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for sqlbuild-0.85.0.tar.gz
Algorithm Hash digest
SHA256 7d799602176f522ce82bfd1e7c29965919b990509933f7d31da2191bd5671374
MD5 f82061e116c7e92e1b8142b3e7e0bd18
BLAKE2b-256 e72458dd397fa86efdefc12a3ce252ec33ea38bea3a932167bcce112163e2e0b

See more details on using hashes here.

Provenance

The following attestation bundles were made for sqlbuild-0.85.0.tar.gz:

Publisher: publish.yml on chio-labs/sqlbuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file sqlbuild-0.85.0-cp312-abi3-win_amd64.whl.

File metadata

  • Download URL: sqlbuild-0.85.0-cp312-abi3-win_amd64.whl
  • Upload date:
  • Size: 10.5 MB
  • Tags: CPython 3.12+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for sqlbuild-0.85.0-cp312-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 01dfd172ec88cc285fc36d17024c71ee007dc11bf3b050e46be0addbe4c86065
MD5 652c096d67774aeae19cc9a274a6e6e9
BLAKE2b-256 660e236d1e4eb4c341db178f9365ac69822baf1007cfab9dadaa6f83a5c10273

See more details on using hashes here.

Provenance

The following attestation bundles were made for sqlbuild-0.85.0-cp312-abi3-win_amd64.whl:

Publisher: publish.yml on chio-labs/sqlbuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file sqlbuild-0.85.0-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for sqlbuild-0.85.0-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 58fe38bfc319021849f05897fa71f110c4a2c706bc7408c02b009a65fe4dc903
MD5 5d66077212c0760082e155e96aaa4d9a
BLAKE2b-256 d126677a63bb11dba37af1c040a154d69fb1bc9dc710a9b1ed2adfd46661426c

See more details on using hashes here.

Provenance

The following attestation bundles were made for sqlbuild-0.85.0-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: publish.yml on chio-labs/sqlbuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file sqlbuild-0.85.0-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for sqlbuild-0.85.0-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 75936b28c0be0a1679b3b812c3bc033ae475b598ecdf2205034b336000176daf
MD5 26ac0c181c2ddea397120d82fcfdf885
BLAKE2b-256 4eb1cfd26cf88ff14fbaf2ab075629cb988c2c25bc496cda00d543df84e9b3d6

See more details on using hashes here.

Provenance

The following attestation bundles were made for sqlbuild-0.85.0-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: publish.yml on chio-labs/sqlbuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file sqlbuild-0.85.0-cp312-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for sqlbuild-0.85.0-cp312-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 24bd965bd10230f02d0bc8c19058a4f7588c6512625749a3a3404232f5706738
MD5 2ac7d2dae8c166d6e29b3b13f33f64a8
BLAKE2b-256 2ac545a915e57fe011a265d3f2ad8ed946e047d0cccb288d1a1187be4409f126

See more details on using hashes here.

Provenance

The following attestation bundles were made for sqlbuild-0.85.0-cp312-abi3-macosx_11_0_arm64.whl:

Publisher: publish.yml on chio-labs/sqlbuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file sqlbuild-0.85.0-cp312-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for sqlbuild-0.85.0-cp312-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 87ebd0a009a7e9f7fc910e62b18c918249a158a4badceacf4757525ff65272b0
MD5 260f73e3887a547c4023feb3fbdd8c35
BLAKE2b-256 d7c591d0e933bf2a9051eac863044634d0b17bf2a1d1bcc03f5fafedb7ca7ca7

See more details on using hashes here.

Provenance

The following attestation bundles were made for sqlbuild-0.85.0-cp312-abi3-macosx_10_12_x86_64.whl:

Publisher: publish.yml on chio-labs/sqlbuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.87.3

6 files

0.87.2

6 files

0.87.1

6 files

0.87.0

6 files

0.86.0

6 files

This release

0.85.0 This release

6 files

0.84.0

6 files

0.83.4

6 files

0.83.3

6 files

0.83.2

6 files

0.83.1

6 files

0.83.0

6 files

0.82.6

6 files

0.82.5

6 files

0.82.4

6 files

0.82.3

6 files

0.82.2

6 files

0.82.1

6 files

0.82.0

6 files

0.81.2

6 files

0.81.1

6 files

0.81.0

6 files

0.80.2

6 files

0.80.1

6 files

0.80.0

6 files

0.79.0

6 files

0.78.0

6 files

0.77.2

6 files

0.77.1

6 files

0.77.0

6 files

0.76.9

6 files

0.76.8

6 files

0.76.7

6 files

0.76.6

6 files

0.76.5

6 files

0.76.4

6 files

0.76.3

6 files

0.76.2

6 files

0.76.1

6 files

0.76.0

6 files

0.75.1

6 files

0.75.0

6 files

0.74.4

6 files

0.74.3

6 files

0.74.2

6 files

0.74.1

6 files

0.74.0

6 files

0.73.0

6 files

0.72.2

6 files

0.72.1

6 files

0.72.0

6 files

0.71.6

6 files

0.71.5

6 files

0.71.4

6 files

0.71.3

6 files

0.71.2

6 files

0.71.1

6 files

0.71.0

6 files

0.70.0

6 files

0.69.0

6 files

0.68.0

6 files

0.67.2

6 files

0.67.1

6 files

0.67.0

6 files

0.66.6

6 files

0.66.5

6 files

0.66.4

6 files

0.66.3

6 files

0.66.2

6 files

0.66.1

6 files

0.66.0

6 files

0.65.6

6 files

0.65.5

6 files

0.65.4

6 files

0.65.3

6 files

0.65.2

6 files

0.65.1

6 files

0.65.0

6 files

0.64.0

6 files

0.63.8

6 files

0.63.7

6 files

0.63.6

6 files

0.63.5

6 files

0.63.4

6 files

0.63.3

6 files

0.63.2

6 files

0.63.1

6 files

0.63.0

6 files

0.62.2

6 files

0.62.1

6 files

0.62.0

6 files

0.61.0

6 files

0.60.0

6 files

0.59.0

6 files

0.58.0

6 files

0.57.0

6 files

0.56.2

6 files

0.56.1

6 files

0.56.0

6 files

0.55.9

6 files

0.55.8

6 files

0.55.7

6 files

0.55.6

6 files

0.55.5

6 files

0.55.4

6 files

0.55.3

6 files

0.55.2

6 files

0.55.1

6 files

0.55.0

6 files

0.54.2

6 files

0.54.1

6 files

0.54.0

5 files

0.53.0

2 files

0.52.0

2 files

0.51.0

2 files

0.50.0

2 files

0.49.0

2 files

0.48.7

2 files

0.48.6

2 files

0.48.5

2 files

0.48.4

2 files

0.48.3

2 files

0.48.2

2 files

0.48.1

2 files

0.48.0

2 files

0.47.0

2 files

0.46.0

2 files

0.45.5

2 files

0.45.4

2 files

0.45.3

2 files

0.45.2

2 files

0.45.1

2 files

0.45.0

2 files

0.44.4

2 files

0.41.1

2 files

0.41.0

2 files

Yanked

0.40.1

2 files

Yanked reason: Just a shitty release

Yanked

0.40.0

2 files

Yanked reason: shitty release

0.39.3

2 files

0.39.2

2 files

0.39.1

2 files

0.39.0

2 files

0.38.6

2 files

0.38.5

2 files

0.38.4

2 files

0.38.3

2 files

0.38.2

2 files

0.38.1

2 files

0.38.0

2 files

0.37.7

2 files

0.37.6

2 files

0.37.5

2 files

0.37.4

2 files

0.37.3

2 files

0.37.2

2 files

0.37.1

2 files

0.36.0

2 files

0.35.0

2 files

0.34.0

2 files

0.33.0

2 files

0.32.0

2 files

0.31.0

2 files

0.30.1

2 files

0.30.0

2 files

0.29.0

2 files

0.28.1

2 files

0.28.0

2 files

0.27.0

2 files

0.26.2

2 files

0.26.1

2 files

0.26.0

2 files

0.25.1

2 files

0.25.0

2 files

0.24.0

2 files

0.23.0

2 files

0.22.1

2 files

0.22.0

2 files

0.21.0

2 files

0.20.1

2 files

0.20.0

2 files

0.19.0

2 files

0.18.0

2 files

0.16.2

2 files

0.16.1

2 files

0.15.0

2 files

0.14.0

2 files

0.13.0

2 files

0.12.0

2 files

0.10.0

2 files

0.9.0

2 files

0.8.0

2 files

0.7.0

2 files

0.4.0

2 files

0.3.0

2 files

0.2.1

2 files

0.2.0

2 files

0.0.1

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page