StreamBuild

Declarative SQL streaming pipelines for ClickHouse, with replay-aware builds and staged deployments.

PyPI Verification Python versions License

StreamBuild lets data teams define continuously updating ClickHouse pipelines in typed SQL, inspect the affected graph before making changes, and rebuild it from retained streaming history when logic changes.

It brings dbt-style authoring and deployment workflows to streaming workloads:

  • Declarative models compile into ClickHouse tables and materialized views.
  • Replay-aware builds reconstruct affected models from retained stream history.
  • Safe planning shows the graph and warehouse operations before execution.
  • Staged deployments support review, audit, promotion, and graph-level rollback.
  • Built-in observability covers sources, lag, lineage, runs, quality checks, and sensors.

StreamBuild currently targets ClickHouse and supports managed Kafka landing or adopted external stream tables.

Install

Requires Python 3.12 or newer.

pip install streambuild
stb --help

Quickstart

The included orders demo runs locally with synthetic events, Redpanda, and ClickHouse. It requires Docker, curl, and uv:

git clone https://github.com/chio-labs/streambuild.git
cd streambuild
uv tool install --upgrade streambuild

docker compose -f examples/orders_demo/docker/compose.yml up -d --build

until docker compose -f examples/orders_demo/docker/compose.yml \
  exec -T redpanda rpk cluster health >/dev/null 2>&1; do sleep 1; done
until curl --fail --silent http://localhost:18123/ping >/dev/null; do sleep 1; done
curl --fail --silent --user clickhouse:clickhouse http://localhost:18123/ \
  --data-binary 'CREATE DATABASE IF NOT EXISTS orders_demo'

stb plan --project-dir examples/orders_demo
stb build --project-dir examples/orders_demo
stb dev --project-dir examples/orders_demo

Open http://127.0.0.1:8000 to inspect the live model graph, retained source messages, runs, quality checks, and Kafka lag. See the orders demo for the model DAG and staged deployment walkthrough.

How it works

Kafka or an adopted stream table
              |
              v
      retained landing data
              |
              v
 typed SQL models -> ClickHouse tables and materialized views
              |
              v
 plan -> build or stage -> audit -> promote

StreamBuild follows __source() and __ref() dependencies to compile the model graph. Live materialized views keep it current; retained replay columns let a later build reconstruct the affected scope after SQL changes.

Project

streambuild_project.toml
sources/
  orders.yml
macros/
  common.py
pipelines/
  pl__orders/
    pipeline.toml
    order_totals.sql
audits/
tests/

Minimal configuration:

name = "orders"
default_target = "dev"

[connection]
host = "localhost"
port = 8123
username = "default"
password = "${ENV:CLICKHOUSE_PASSWORD}"

[defaults]
pipeline_mode = "direct"

[targets.dev]
database = "analytics"

Developer-specific target and connection overrides belong in the gitignored streambuild_local.toml.

Sources

Managed Kafka source:

sources:
  - name: orders
    kind: kafka
    broker_list: kafka:9092
    topic: source.orders
    replay_boundary:
      mode: offsets

StreamBuild owns managed Kafka landing objects. It validates but never mutates adopted source tables. See the pipeline documentation for adopted stream-table configuration.

Pipelines

Each direct child of pipelines/ is a pipeline. Its directory name is its logical name:

pipelines/
  pl__orders/
    staging/
      orders_clean.sql
    order_totals.sql

Nested directories organize models but do not change pipeline identity. Pipeline, source, and model names share one namespace and must be unique.

Models

MODEL (
  engine "MergeTree()",
  order_by ["order_id", "_replay_partition", "_replay_offset"],
);

SELECT
  order_id::String AS order_id,
  _replay_partition::Int32 AS _replay_partition,
  _replay_offset::Int64 AS _replay_offset,
  _replay_timestamp::DateTime64(3) AS _replay_timestamp
FROM __source("orders")

Models must project explicit output types. Table models preserve replay lineage through normalized _replay_* columns. Terminal query views use MODEL (kind view).

Python functions under macros/ are available in model, test, and audit SQL as @function_name().

Workflow

stb discover                 # inspect authored resources
stb compile                  # offline validation and artifacts
stb plan                     # read-only warehouse plan
stb build                    # confirm and execute
stb test
stb audit
stb dev                      # local UI at 127.0.0.1:8000

Select a model or pipeline with repeatable selectors:

stb plan --select pipeline:pl__orders
stb build --select order_totals --start-time 2026-08-01T00:00:00Z

stb compile is offline. stb plan reads warehouse state but cannot mutate it. stb build replans immediately before execution so an approved command never relies on a stale plan.

Build modes

Mode Behavior Best for
Direct Rebuilds selected live relations immediately Development and explicitly controlled live changes
Virtual Builds deployment-specific relations before promotion Review, validation, and production releases
Mixed Stages virtual pipelines before applying direct pipelines Projects containing both deployment strategies

Deployments

Set pipeline_mode = "virtual" project-wide or mode = "virtual" in pipeline.toml.

stb build
stb deployment list
stb deployment show <deployment-id>
stb deployment diff <deployment-id>
stb deployment audit <deployment-id>
stb deployment promote <deployment-id>
stb deployment rollback --previous

Promotion and rollback switch stable views one relation at a time. Rollback restores a retained publication's bindings, not a historical data snapshot.

Development UI

stb dev serves a warehouse-backed interface for one resolved project and target. It provides:

StreamBuild lineage view showing the orders demo pipeline

  • overview, lineage, pipeline, catalog, source, topic, and message inspection
  • connected plan previews and protected-pipeline confirmation
  • direct, virtual, and mixed build execution
  • deployment inventory, diff, promotion, and cleanup
  • durable run timelines, statement progress, cancellation, and stale-run recovery guidance
  • quality history, scheduler health, sensors, and dead-letter recovery

Run observability is warehouse-backed. A silent run becomes unresponsive after 45 seconds and presumed_failed after [defaults].run_presumed_failed_after (default 10m). A new build is blocked until that safety window expires to prevent overlapping warehouse writes.

Shared installations support trusted-proxy or password authentication with project-scoped roles. See the documentation for access control and operational configuration.

Guarantees

  • stb compile is connection-free and writes disposable artifacts under target/.
  • stb plan is read-only; stb build always replans against current warehouse state.
  • Lifecycle state is append-only metadata in the selected target database.
  • Failed or cancelled builds are rerun, never resumed from copied SQL artifacts.
  • Workflow statements execute serially to avoid unbounded ClickHouse memory pressure.

Documentation

Development

Install the locked Python and UI dependencies before running the verification lanes:

uv sync --locked --all-groups
make ui-install ui-build
make check-ci
make ui-verify
make test

Run every verification lane with:

make test-all

The integration and browser lanes provision real ClickHouse and Redpanda containers, so Docker must be running. Install Chromium and its system dependencies before the first browser run:

uv run playwright install --with-deps chromium
make ui-install ui-build
make test-browser

Project links

Data verified by PyPI on Aug 23, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.39.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for kvlonge from gravatar.com kvlonge

GitHub Statistics

Data verified by PyPI on Aug 23, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

Requires

Python >=3.12

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

streambuild-0.26.16.tar.gz (3.2 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

streambuild-0.26.16-py3-none-any.whl (1.9 MB view details)

Uploaded Python 3

File details

Details for the file streambuild-0.26.16.tar.gz.

File metadata

  • Download URL: streambuild-0.26.16.tar.gz
  • Upload date:
  • Size: 3.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for streambuild-0.26.16.tar.gz
Algorithm Hash digest
SHA256 b87ab6452d1421c0f2f3071ca3750dbaba4af2a71f6eeb5864f778fd9c2a8f2e
MD5 8a758ece3c169e2b3c2392209b880940
BLAKE2b-256 a5e6ab183589ef610386ddc9b84a7860f3bceccb8a30756aad1ee777a88e6e88

See more details on using hashes here.

Provenance

The following attestation bundles were made for streambuild-0.26.16.tar.gz:

Publisher: publish.yml on chio-labs/streambuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file streambuild-0.26.16-py3-none-any.whl.

File metadata

  • Download URL: streambuild-0.26.16-py3-none-any.whl
  • Upload date:
  • Size: 1.9 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for streambuild-0.26.16-py3-none-any.whl
Algorithm Hash digest
SHA256 63bfc103390d9c9f3e86617d3dbaeea19ab64e699f99564371ef8e92e03a0170
MD5 221f0d8928a5dfdd59744562fa6c952f
BLAKE2b-256 4b91842e6ab69bfa304ef724d55ba075c8d168f71c5ea2d7ce1c843111dd5c62

See more details on using hashes here.

Provenance

The following attestation bundles were made for streambuild-0.26.16-py3-none-any.whl:

Publisher: publish.yml on chio-labs/streambuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.39.0

2 files

0.38.4

2 files

0.38.3

2 files

0.38.2

2 files

0.38.1

2 files

0.38.0

2 files

0.37.0

2 files

0.36.0

2 files

0.35.1

2 files

0.35.0

2 files

0.34.4

2 files

0.34.3

2 files

0.34.2

2 files

0.34.1

2 files

0.34.0

2 files

0.33.1

2 files

0.33.0

2 files

0.32.0

2 files

0.31.0

2 files

0.30.0

2 files

0.29.1

2 files

0.29.0

2 files

0.28.3

2 files

0.28.2

2 files

0.28.1

2 files

0.28.0

2 files

0.27.2

2 files

0.27.1

2 files

0.27.0

2 files

This release

0.26.16 This release

2 files

0.26.15

2 files

0.26.14

2 files

0.26.13

2 files

0.26.12

2 files

0.26.11

2 files

0.26.10

2 files

0.26.9

2 files

0.26.8

2 files

0.26.7

2 files

0.26.6

2 files

0.26.5

2 files

0.26.4

2 files

0.26.3

2 files

0.26.2

2 files

0.26.1

2 files

0.26.0

2 files

0.25.1

2 files

0.25.0

2 files

0.24.5

2 files

0.24.4

2 files

0.24.3

2 files

0.24.2

2 files

0.24.1

2 files

0.24.0

2 files

0.23.0

2 files

0.22.4

2 files

0.22.3

2 files

0.22.2

2 files

0.22.1

2 files

0.22.0

2 files

0.21.3

2 files

0.21.2

2 files

0.21.1

2 files

0.21.0

2 files

0.20.0

2 files

0.18.1

2 files

0.18.0

2 files

0.17.0

2 files

0.16.6

2 files

0.16.5

2 files

0.16.4

2 files

0.16.3

2 files

0.16.2

2 files

0.16.1

2 files

0.16.0

2 files

0.15.0

2 files

0.14.1

2 files

0.14.0

2 files

0.13.0

2 files

0.12.4

2 files

0.12.3

2 files

0.12.2

2 files

0.12.1

2 files

0.12.0

2 files

0.11.0

2 files

0.10.0

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.0

2 files

0.7.0

2 files

0.6.0

2 files

0.5.0

2 files

0.4.0

2 files

0.3.0

2 files

0.2.0

2 files

0.0.1

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page