Verify early. Test properly. Deploy reversibly. SQL pipelines with the rigor of real software.
Valid isn't the same as correct. Your SQL compiles, runs, and returns rows; none of that means the number is right, and a silently-wrong number a stakeholder already trusted is the bug that actually hurts.
SQLBuild brings software-engineering rigor to SQL pipelines: catch errors before the warehouse runs them, test your logic locally, and opt into change-aware execution when you need it. It is a standalone, open-source framework for building SQL and Python data pipelines.
All state is persisted as append-only tables in the warehouse alongside your data: no external state database, no manifest files, no paid add-on. Start with straightforward SQL models, then add ingestion, Python nodes, and opt-in virtual environments as your project grows.
Key features
- Test your logic, not just your columns. Multi-model SQL tests resolve every intermediate model from its real SQL, plus end-to-end scenarios with local DuckDB replay for fast CI with no warehouse. Catch wrong logic before it ships, not just nulls.
- Verify early. Define models as SQL files with
MODEL()headers. SQLBuild resolves references, validates SQL, infers columns, checks contracts, and computes column lineage before anything runs, all offline. It fails at compile, not halfway through a warehouse run. - Fast and open static analysis. SQL parsing, validation, column inference, lineage, and transpilation run on Polyglot, a Rust SQL engine (MIT, 32+ dialects), so compile stays fast on large projects. The analysis is part of the Apache-2.0 core: no proprietary engine, no login, no paid tier.
- Audits that block bad data. Audits run before data reaches the target table. Full table builds materialize into a staging table and only promote if audits pass; incremental models validate each batch before DML.
- Deploy reversibly (opt-in). Virtual environments add instant low-copy branching, partial promotion, rollback, checkpoints, and reconciliation. Opt-in, not a tax you pay upfront.
- Opt-in change-aware execution. Models, seeds, UDFs, and Python nodes are fingerprinted, and source freshness is tracked. In virtual environments, pass
--changes-onlyor setchanges_only = trueto skip work that is already current; commands otherwise run the full selected scope. - Warehouse-native state. All change-tracking state lives in append-only tables (
_sqlbuild_fingerprints,_sqlbuild_source_freshness,_sqlbuild_node_results) in your warehouse schemas. No external state machine, no corruption risk. - Cursor-based incremental processing. Automatic gap detection and resume, with microbatch mode for large ranges. No external checkpoint to maintain.
- Ingestion and Python nodes. Load external data with Python
@loaderfunctions, and run@task,@asset, and@checknodes as first-class members of the same DAG as your SQL models.
See the documentation for the full feature set, including providers, lifecycle hooks, Python macros, UDFs, custom materializations, data diffs, zero-copy cloning, and virtual environments. To coordinate dbt and SQLBuild projects, see the dbt compatibility guide.
Quick start
pip install sqlbuild
# or
uv pip install sqlbuild
Create and run the included playground project:
sqb playground waffle-shop
cd waffle-shop
sqb plan
sqb build
sqb test
Example
A model is a SQL file with a MODEL() header and a SELECT. References use __ref() and __source(), and configuration, schema, and audits are declared inline:
MODEL (
materialized table,
columns (
order_id (audits [not_null, unique]),
),
tags [marts],
);
SELECT
o.order_id,
o.customer_id,
p.amount_cents AS total_cents
FROM __ref("stg_orders") o
JOIN __ref("stg_payments") p USING (order_id)
A unit test mocks sources and asserts on the model, resolving every intermediate model automatically:
TEST();
WITH
__source__raw__orders AS (
@mock_orders()
),
__source__raw__payments AS (
SELECT
1 AS payment_id,
1 AS order_id,
1500 AS amount_cents,
'credit_card' AS method
),
__expected__fact_orders AS (
SELECT 1 AS order_id, 100 AS customer_id, 1500 AS total_cents
)
SELECT 1
See the documentation for incremental models, scenarios, loaders, and more.
Python project layout
Project-owned Python must live in a supported extension location such as factories/, libs/,
macros/, providers/, or another documented Python resource root. Factory locations contain
normal Python: constants, classes, undecorated helper functions, and modules such as _helpers.py
are allowed, while decorators determine which functions become SQLBuild resources. Compilation
rejects Python under invented project roots so indirectly importable modules cannot create an
unofficial project structure. Keep repository pytest tests outside the SQLBuild project's tests/
directory, which is reserved for SQLBuild SQL tests and scenarios. Documented integration paths
such as dagster/, rivers_pipeline/, and their definitions.py modules are also supported.
SQL lint and Project Policy
SQL lint evaluates one plain SQL statement. Project Policy evaluates how SQLBuild resources are organised, configured, documented, tested, and connected. Both are deterministic: the boundary is the evidence a rule needs, not its severity.
sqb lint owns statement-local SQBL checks, including comment attachment, CTE shape, set
operations, and joins. Repository-defined statement-local checks use the separate XSQBL API:
from sqlbuild.lint import LintRuleContext, lint_rule
@lint_rule(
code="XSQBLS001",
family="shape",
slug="no-star",
message="Star projections are not allowed",
remediation="Enumerate the intended columns.",
)
def no_star(*, ctx: LintRuleContext):
if "*" not in ctx.source:
return ()
start = ctx.source.index("*")
return (ctx.finding(start=start, end=start + 1),)
Custom lint receives only SQL source, dialect, AST, source spans, and declared options. It cannot
observe models, paths, project configuration, the dependency graph, declarations, filesystem,
environment, process, network, or warehouse state. Configure repository-owned lint files with
[lint].rule_paths, select them with XSQBL..., and test them directly with
evaluate_lint_rule.
Project Policy is SQLBuild's opt-in, error-only project-aware checker. It runs offline over the compiled project, reports coded faults with remediations, and never rewrites SQL. Its built-in lifecycle is native: Rust resolves rule policy, parses each model, evaluates built-ins, applies suppressions, and owns the persistent cache and deterministic result ordering.
Project Policy is disabled until the project selects at least one rule. Select the complete built-in policy
in sqlbuild_project.toml with its namespace prefix:
[policy]
select = ["SQBP"]
SQBP activates every built-in rule. Narrower prefixes such as SQBPS activate one family, exact
codes select individual rules, and ignore removes matching rules. Audit, unit-test, and custom-rule
test-case minimums each default to one and can be overridden under [policy.thresholds].
Project Policy also keeps model ownership shallow and explicit. Configured level paths separate warehouse layers from domain ownership; every owner is a leaf or a branch, subdomain depth defaults to one, and declaration roles remain bounded flat-or-grouped containers:
[policy.layout]
levels = ["staging", "intermediate/clean", "intermediate/enriched", "mart"]
domain_roots = ["sales/partner", "inventory/forecasting"] # optional disambiguation
[policy.thresholds]
max_subdomain_depth = 1
min_shared_owner_prefix_directories = 2
Run sqb policy, inspect metadata with sqb policy rule SQBPS101, and generate agent guidance from
the same active ruleset with sqb policy skills. Use sqb policy skills --check in CI to detect stale
guidance. --json, --select, and --exclude are available for automation and model scoping.
Repository rules use the public API:
from sqlbuild.policy import RuleContext, policy
@policy(
code="XSQBPP001",
family="prices",
slug="typed-currency",
message="price models must declare a currency column",
remediation="Declare currency in the MODEL columns contract at this model path.",
)
def typed_currency(*, model, ctx: RuleContext):
return [] if any(column.name == "currency" for column in ctx.declared_columns) else [
ctx.path_fault()
]
Load repository-owned files through rule_paths = ["policy/rules"] or dotted packages through
rule_modules. Test each custom rule with RuleCase and evaluate_rule. Selecting custom rules
disables caching unless [policy.cache] require_cacheable = true; cacheable rules may import only
the supported pure modules and must access tracked .py, .sql, .toml, .yaml, or .yml
project files through RuleContext.
Custom Project Policy rules are model-local by default and receive incremental per-model cache
entries. Set project_wide=True on rules that inspect project-wide context or report findings for
other paths; cacheability validation rejects project-wide context access from a model-local rule.
Python is used only for the SQLBuild compiler adapter and selected custom rules. Built-in-only
runs cross into the native engine once as a compiled model batch and do not materialize or walk
Python AST objects. A selected custom rule can still use the public RuleContext and raw
Polyglot AST escape hatch; its findings rejoin native suppression, ordering, and cache policy.
Exact rule_exceptions require a rule, file, and reason and fail when stale. Broader
rule_ignores and lone-star allowances also require reasons but are intentionally not
stale-checked.
Run the neutral large-project benchmark locally after implementation changes:
uv run python -m scripts.benchmark_project_policy --models 3000 --iterations 3
uv run python -m scripts.benchmark_project_policy --models 5000 --iterations 3
The generated projects contain no company model names or SQL. They retain representative graph, SQL-complexity, contract, test, declaration, and custom-policy stressors and report cold, unchanged, leaf-edit, shared-ancestor, tracked-policy-input, custom-rule-edit, cache-disabled, and non-cacheable rejection median/p95 timings.
Supported adapters
| Adapter | Status |
|---|---|
| DuckDB | Supported |
| MotherDuck | Supported |
| Snowflake | Supported |
| BigQuery | Supported |
| Databricks | Supported |
| PostgreSQL | Supported |
| SQL Server | Supported |
ClickHouse, Redshift, Trino, Spark, and Athena are on the way.
Snowflake cost estimates
Native Snowflake builds automatically show a compact per-run busy-compute estimate. SQLBuild attributes visible overlapping query intervals fairly across active queries, converts attributed seconds using the warehouse-size credit rate, and estimates USD from the configured rate:
[cost]
usd_per_credit = 3.00
The default is 3.00 USD per credit and is visibly marked as a default. Configure the value with
your Snowflake contract rate. Use sqb cost, sqb cost latest, sqb cost <run_id>, or
sqb cost history --since 7d to inspect persisted records. --json and --json-output PATH
provide a versioned, decimal-safe output contract. Pending detail records are refreshed from
Snowflake when inspected again.
These values are attributed compute credits and estimated cost, not Snowflake-billed credits or
invoice reconciliation. The estimate uses only query history visible to the executing role and
does not reconstruct invisible concurrent work, warehouse resume or idle tail, the 60-second
minimum, cloud-services credits, contract adjustments, or multi-cluster billing. Run metadata and
query IDs are stored under target/executions/<run_id>/; that statement ledger stores only an SQL
digest, not SQL text. Executed SQL artifacts are stored separately under the sensitive
target/run/ tree.
Documentation
Full documentation is available at docs.sqlbuild.com.
Runtime operator and extension contracts:
- Execution observability and local troubleshooting
- SQLite and PostgreSQL execution history
- Typed lifecycle and command-output sinks
Contributing
We welcome contributions. Please see CONTRIBUTING.md for guidelines.
License
SQLBuild is licensed under the Apache License 2.0.
1 maintainer
Release files for sqlbuild 0.92.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| sqlbuild-0.92.0.tar.gz | 1.6 MB | Details |
Built distributions (wheels)
| File | Reset | |||
|---|---|---|---|---|
| sqlbuild-0.92.0-cp312-abi3-win_amd64.whl | CPython 3.12 | abi3 | Windows x86-64 | Details |
| sqlbuild-0.92.0-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl | CPython 3.12 | abi3 | Linux glibc 2.17+ x86-64 | Details |
| sqlbuild-0.92.0-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl | CPython 3.12 | abi3 | Linux glibc 2.17+ ARM64 | Details |
| sqlbuild-0.92.0-cp312-abi3-macosx_11_0_arm64.whl | CPython 3.12 | abi3 | macOS 11.0+ ARM64 | Details |
| sqlbuild-0.92.0-cp312-abi3-macosx_10_12_x86_64.whl | CPython 3.12 | abi3 | macOS 10.12+ x86-64 | Details |
Total release size: 60.4 MB
Release files / sqlbuild-0.92.0.tar.gz
| Download URL | sqlbuild-0.92.0.tar.gz |
|---|---|
| Size | 1.6 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
36b5c455b3c1a5f785b855e3357477704bb50e084d16050f0c86a5cc486eda2e
|
|
BLAKE2b-256 checksum How to use checksums |
b3237bc821f3052dedde765b94ce613c92fc9dca02651661c9b04489d227721c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 8, 2026.
Transparency logRelease files / sqlbuild-0.92.0-cp312-abi3-win_amd64.whl
| Download URL | sqlbuild-0.92.0-cp312-abi3-win_amd64.whl |
|---|---|
| Size | 11.7 MB |
| Tags | CPython 3.12 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
ba9205345e31a790383896564258fac90456b1b1477c4535d4571bcde71fcf8d
|
|
BLAKE2b-256 checksum How to use checksums |
1d3e8148e025fe1059534c569d510289799d448dd5237cb8ce7bd64a4d29ac84
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 8, 2026.
Transparency logRelease files / sqlbuild-0.92.0-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | sqlbuild-0.92.0-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 12.3 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
7e93b6f7d4e5deb6a1f081fce7a0c2f777c1d28131dcf466ca47b4c30ff667e3
|
|
BLAKE2b-256 checksum How to use checksums |
927a54d886bebf1966ac5aead8051311c455c8a4a38cb43cfa71218b72a2487f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 8, 2026.
Transparency logRelease files / sqlbuild-0.92.0-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | sqlbuild-0.92.0-cp312-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 11.9 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
f29652f1a1036730c8318b3093bc658f8a32f22719954862b4b1c1e8cff11a42
|
|
BLAKE2b-256 checksum How to use checksums |
0b53d2f4cc535c1b96db784641eb97fc52159d54462ff6e32f5406f853101781
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 8, 2026.
Transparency logRelease files / sqlbuild-0.92.0-cp312-abi3-macosx_11_0_arm64.whl
| Download URL | sqlbuild-0.92.0-cp312-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 11.3 MB |
| Tags | CPython 3.12 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
83541d232e1ec88559feec8184e5cbb1d3f106e56a2ec57bd38c9c22ed75ee68
|
|
BLAKE2b-256 checksum How to use checksums |
ae909e816f070f46dc72664a50c239e73b665ce57f135520b40d185c60ae2f08
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 8, 2026.
Transparency logRelease files / sqlbuild-0.92.0-cp312-abi3-macosx_10_12_x86_64.whl
| Download URL | sqlbuild-0.92.0-cp312-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 11.5 MB |
| Tags | CPython 3.12 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
27dd55cceb19d515701485ddf36dffd32184a874c597a006b114bb807f507c45
|
|
BLAKE2b-256 checksum How to use checksums |
2b255e3f6d4ccf48238f0cbf54ff6c063654d639465deb76b2302c3a662a792d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 8, 2026.
Transparency log
