StreamBuild

Declarative ClickHouse streaming pipelines with replay-aware builds and staged deployments.

StreamBuild compiles typed SQL models into ClickHouse tables and materialized views, plans the affected graph, and rebuilds it from retained streaming history.

  • Direct mode rebuilds selected live relations immediately.
  • Virtual mode builds deployment-specific relations for review, audit, promotion, and rollback.
  • Mixed mode stages virtual pipelines before applying direct pipelines in one invocation.

StreamBuild currently targets ClickHouse. It supports managed Kafka landing and adopted external stream tables.

Install

Requires Python 3.12 or newer.

pip install streambuild
stb --help

For repository development:

uv sync
uv run stb --help

Project

streambuild_project.toml
sources/
  orders.yml
macros/
  common.py
pipelines/
  orders/
    pipeline.toml
    order_totals.sql
audits/
tests/

Each direct child of pipelines/ is a pipeline. SQL filenames define logical model names, and StreamBuild infers pipeline sources by following __source() and __ref() dependencies.

Minimal configuration:

name = "orders"
default_target = "dev"

[connection]
host = "localhost"
port = 8123
username = "default"
password = "${ENV:CLICKHOUSE_PASSWORD}"

[defaults]
pipeline_mode = "direct"
run_presumed_failed_after = "10m"

[targets.dev]
database = "analytics"

Developer-specific target and connection overrides belong in the gitignored streambuild_local.toml.

Sources

Managed Kafka source:

sources:
  - name: orders
    kind: kafka
    broker_list: kafka:9092
    topic: source.orders
    replay_boundary:
      mode: offsets

Adopted ClickHouse source:

sources:
  - name: orders
    kind: stream_table
    table_name: orders_existing
    replay_boundary:
      mode: offsets
      columns:
        _replay_partition: event_partition
        _replay_offset: event_offset
        _replay_timestamp: event_time

StreamBuild owns managed Kafka landing objects. It validates but never mutates adopted source tables.

Models

MODEL (
  engine "MergeTree()",
  order_by ["order_id", "_replay_partition", "_replay_offset"],
);

SELECT
  order_id::String AS order_id,
  _replay_partition::Int32 AS _replay_partition,
  _replay_offset::Int64 AS _replay_offset,
  _replay_timestamp::DateTime64(3) AS _replay_timestamp
FROM __source("orders")

Models must project explicit output types. Table models preserve replay lineage through normalized _replay_* columns. Terminal query views use MODEL (kind view).

Python functions under macros/ are available in model, test, and audit SQL as @function_name().

Workflow

stb discover                 # inspect authored resources
stb compile                  # offline validation and artifacts
stb plan                     # read-only warehouse plan
stb build                    # confirm and execute
stb test
stb audit
stb dev                      # local UI at 127.0.0.1:8000

Select a model or pipeline with repeatable selectors:

stb plan --select pipeline:orders
stb build --select order_totals --start-time 2026-08-01T00:00:00Z

Protected pipelines require their exact configured --confirm value even with --auto-approve.

Deployments

Set pipeline_mode = "virtual" project-wide or mode = "virtual" in pipeline.toml.

stb build
stb deployment list
stb deployment show <deployment-id>
stb deployment diff <deployment-id>
stb deployment audit <deployment-id>
stb deployment promote <deployment-id>
stb deployment rollback --previous

Promotion and rollback switch stable views one relation at a time. Rollback restores a retained publication's bindings, not a historical data snapshot.

Development UI

stb dev serves one resolved project and target. The UI provides:

  • overview, lineage, pipeline, catalog, source, topic, and message inspection
  • connected plan previews and protected-pipeline confirmation
  • direct, virtual, and mixed build execution
  • deployment inventory, diff, promotion, and cleanup
  • durable run timelines, statement progress, cancellation, and stale-run recovery guidance
  • audit history and scheduler health

Run observability is warehouse-backed. A silent run becomes unresponsive after 45 seconds and presumed_failed after [defaults].run_presumed_failed_after (default 10m). A new build is blocked until that safety window expires to prevent overlapping warehouse writes.

Lineage activity is separate from replay freshness. StreamBuild prefers ClickHouse system.query_views_log, then insert-only system.part_log evidence. Enable the query_views_log and part_log server log sections and set log_query_views = 1 for ingestion users to get exact materialized-view activity. If both logs are unavailable, recent system.parts modification time is shown as approximate evidence because background merges can also modify parts. Missing evidence is reported as unknown rather than stalled.

Guarantees

  • stb compile is connection-free and writes disposable artifacts under target/.
  • stb plan is read-only; stb build always replans against current warehouse state.
  • Lifecycle state is append-only metadata in the selected target database.
  • Failed or cancelled builds are rerun, never resumed from copied SQL artifacts.
  • Workflow statements execute serially to avoid unbounded ClickHouse memory pressure.

Documentation

Development

make check-ci
make test
make test-all
make ui-build

Tests are split across tests/unit, tests/integration, and tests/e2e.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.39.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for kvlonge from gravatar.com kvlonge

Requires

Python >=3.12

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

streambuild-0.17.0.tar.gz (2.4 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

streambuild-0.17.0-py3-none-any.whl (1.7 MB view details)

Uploaded Python 3

File details

Details for the file streambuild-0.17.0.tar.gz.

File metadata

  • Download URL: streambuild-0.17.0.tar.gz
  • Upload date:
  • Size: 2.4 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for streambuild-0.17.0.tar.gz
Algorithm Hash digest
SHA256 88d5ab8ebc564aeb43133d6afb8cc10a53eb41bd8d0d099259527e8499485d29
MD5 708df2427275419887126361870302be
BLAKE2b-256 e339c452ef97334d184aa9614316abeea61107215e6c2d1aae2e2b503f171c15

See more details on using hashes here.

Provenance

The following attestation bundles were made for streambuild-0.17.0.tar.gz:

Publisher: publish.yml on chio-labs/streambuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file streambuild-0.17.0-py3-none-any.whl.

File metadata

  • Download URL: streambuild-0.17.0-py3-none-any.whl
  • Upload date:
  • Size: 1.7 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for streambuild-0.17.0-py3-none-any.whl
Algorithm Hash digest
SHA256 751d04ea11ce9c10fbf87d387c0cba5962c18b41956520b65df66bf1c757de3c
MD5 864040e69572da0b56b4a0c349787ecd
BLAKE2b-256 2cdca2e56a40ea7411c2208c27ca05629a1e99cd78be0676b6a3fa417e09e8c7

See more details on using hashes here.

Provenance

The following attestation bundles were made for streambuild-0.17.0-py3-none-any.whl:

Publisher: publish.yml on chio-labs/streambuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.39.0

2 files

0.38.4

2 files

0.38.3

2 files

0.38.2

2 files

0.38.1

2 files

0.38.0

2 files

0.37.0

2 files

0.36.0

2 files

0.35.1

2 files

0.35.0

2 files

0.34.4

2 files

0.34.3

2 files

0.34.2

2 files

0.34.1

2 files

0.34.0

2 files

0.33.1

2 files

0.33.0

2 files

0.32.0

2 files

0.31.0

2 files

0.30.0

2 files

0.29.1

2 files

0.29.0

2 files

0.28.3

2 files

0.28.2

2 files

0.28.1

2 files

0.28.0

2 files

0.27.2

2 files

0.27.1

2 files

0.27.0

2 files

0.26.16

2 files

0.26.15

2 files

0.26.14

2 files

0.26.13

2 files

0.26.12

2 files

0.26.11

2 files

0.26.10

2 files

0.26.9

2 files

0.26.8

2 files

0.26.7

2 files

0.26.6

2 files

0.26.5

2 files

0.26.4

2 files

0.26.3

2 files

0.26.2

2 files

0.26.1

2 files

0.26.0

2 files

0.25.1

2 files

0.25.0

2 files

0.24.5

2 files

0.24.4

2 files

0.24.3

2 files

0.24.2

2 files

0.24.1

2 files

0.24.0

2 files

0.23.0

2 files

0.22.4

2 files

0.22.3

2 files

0.22.2

2 files

0.22.1

2 files

0.22.0

2 files

0.21.3

2 files

0.21.2

2 files

0.21.1

2 files

0.21.0

2 files

0.20.0

2 files

0.18.1

2 files

0.18.0

2 files

This release

0.17.0 This release

2 files

0.16.6

2 files

0.16.5

2 files

0.16.4

2 files

0.16.3

2 files

0.16.2

2 files

0.16.1

2 files

0.16.0

2 files

0.15.0

2 files

0.14.1

2 files

0.14.0

2 files

0.13.0

2 files

0.12.4

2 files

0.12.3

2 files

0.12.2

2 files

0.12.1

2 files

0.12.0

2 files

0.11.0

2 files

0.10.0

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.0

2 files

0.7.0

2 files

0.6.0

2 files

0.5.0

2 files

0.4.0

2 files

0.3.0

2 files

0.2.0

2 files

0.0.1

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page