StreamBuild

Declarative ClickHouse streaming pipelines with replay-aware builds and staged deployments.

StreamBuild compiles typed SQL models into ClickHouse tables and materialized views, plans the affected graph, and rebuilds it from retained streaming history.

  • Direct mode rebuilds selected live relations immediately.
  • Virtual mode builds deployment-specific relations for review, audit, promotion, and rollback.
  • Mixed mode stages virtual pipelines before applying direct pipelines in one invocation.

StreamBuild currently targets ClickHouse. It supports managed Kafka landing and adopted external stream tables.

Install

Requires Python 3.12 or newer.

pip install streambuild
stb --help

For repository development:

uv sync
uv run stb --help

Project

streambuild_project.toml
sources/
  orders.yml
macros/
  common.py
pipelines/
  orders/
    pipeline.toml
    order_totals.sql
audits/
tests/

Each direct child of pipelines/ is a pipeline. SQL filenames define logical model names, and StreamBuild infers pipeline sources by following __source() and __ref() dependencies.

Minimal configuration:

name = "orders"
default_target = "dev"

[connection]
host = "localhost"
port = 8123
username = "default"
password = "${ENV:CLICKHOUSE_PASSWORD}"

[defaults]
pipeline_mode = "direct"
run_presumed_failed_after = "10m"

[targets.dev]
database = "analytics"

Developer-specific target and connection overrides belong in the gitignored streambuild_local.toml.

Sources

Managed Kafka source:

sources:
  - name: orders
    kind: kafka
    broker_list: kafka:9092
    topic: source.orders
    replay_boundary:
      mode: offsets

Adopted ClickHouse source:

sources:
  - name: orders
    kind: stream_table
    table_name: orders_existing
    replay_boundary:
      mode: offsets
      columns:
        _replay_partition: event_partition
        _replay_offset: event_offset
        _replay_timestamp: event_time

StreamBuild owns managed Kafka landing objects. It validates but never mutates adopted source tables.

Models

MODEL (
  engine "MergeTree()",
  order_by ["order_id", "_replay_partition", "_replay_offset"],
);

SELECT
  order_id::String AS order_id,
  _replay_partition::Int32 AS _replay_partition,
  _replay_offset::Int64 AS _replay_offset,
  _replay_timestamp::DateTime64(3) AS _replay_timestamp
FROM __source("orders")

Models must project explicit output types. Table models preserve replay lineage through normalized _replay_* columns. Terminal query views use MODEL (kind view).

Python functions under macros/ are available in model, test, and audit SQL as @function_name().

Workflow

stb discover                 # inspect authored resources
stb compile                  # offline validation and artifacts
stb plan                     # read-only warehouse plan
stb build                    # confirm and execute
stb test
stb audit
stb dev                      # local UI at 127.0.0.1:8000

Select a model or pipeline with repeatable selectors:

stb plan --select pipeline:orders
stb build --select order_totals --start-time 2026-08-01T00:00:00Z

Protected pipelines require their exact configured --confirm value even with --auto-approve.

Deployments

Set pipeline_mode = "virtual" project-wide or mode = "virtual" in pipeline.toml.

stb build
stb deployment list
stb deployment show <deployment-id>
stb deployment diff <deployment-id>
stb deployment audit <deployment-id>
stb deployment promote <deployment-id>
stb deployment rollback --previous

Promotion and rollback switch stable views one relation at a time. Rollback restores a retained publication's bindings, not a historical data snapshot.

Development UI

stb dev serves one resolved project and target. The UI provides:

  • overview, lineage, pipeline, catalog, source, topic, and message inspection
  • connected plan previews and protected-pipeline confirmation
  • direct, virtual, and mixed build execution
  • deployment inventory, diff, promotion, and cleanup
  • durable run timelines, statement progress, cancellation, and stale-run recovery guidance
  • audit history and scheduler health

Run observability is warehouse-backed. A silent run becomes unresponsive after 45 seconds and presumed_failed after [defaults].run_presumed_failed_after (default 10m). A new build is blocked until that safety window expires to prevent overlapping warehouse writes.

Guarantees

  • stb compile is connection-free and writes disposable artifacts under target/.
  • stb plan is read-only; stb build always replans against current warehouse state.
  • Lifecycle state is append-only metadata in the selected target database.
  • Failed or cancelled builds are rerun, never resumed from copied SQL artifacts.
  • Workflow statements execute serially to avoid unbounded ClickHouse memory pressure.

Documentation

Development

make check-ci
make test
make test-all
make ui-build

Tests are split across tests/unit, tests/integration, and tests/e2e.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.39.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for kvlonge from gravatar.com kvlonge

Requires

Python >=3.12

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

streambuild-0.16.6.tar.gz (2.4 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

streambuild-0.16.6-py3-none-any.whl (1.6 MB view details)

Uploaded Python 3

File details

Details for the file streambuild-0.16.6.tar.gz.

File metadata

  • Download URL: streambuild-0.16.6.tar.gz
  • Upload date:
  • Size: 2.4 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for streambuild-0.16.6.tar.gz
Algorithm Hash digest
SHA256 657ee92be551dfbb7d67ba351c4dfede223e9dea636158c0dcbc5dac20670cab
MD5 c65d454cac44b7a4f2770109e593ba1e
BLAKE2b-256 636b470e1c8f9425d3dd0775849b8bf43e0b32c5b856d75cbaf1876fd6231d78

See more details on using hashes here.

Provenance

The following attestation bundles were made for streambuild-0.16.6.tar.gz:

Publisher: publish.yml on chio-labs/streambuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file streambuild-0.16.6-py3-none-any.whl.

File metadata

  • Download URL: streambuild-0.16.6-py3-none-any.whl
  • Upload date:
  • Size: 1.6 MB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for streambuild-0.16.6-py3-none-any.whl
Algorithm Hash digest
SHA256 76efde6b7339f39d9be73f9e3e66bd3d224d09202c57b6c2f56e3adfc14e24c1
MD5 8426c0167cffb2de0050536def65cb70
BLAKE2b-256 987655b4beb2d11add199284e1a286946be270b2466efcd575e6223cfcdc5ac6

See more details on using hashes here.

Provenance

The following attestation bundles were made for streambuild-0.16.6-py3-none-any.whl:

Publisher: publish.yml on chio-labs/streambuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.39.0

2 files

0.38.4

2 files

0.38.3

2 files

0.38.2

2 files

0.38.1

2 files

0.38.0

2 files

0.37.0

2 files

0.36.0

2 files

0.35.1

2 files

0.35.0

2 files

0.34.4

2 files

0.34.3

2 files

0.34.2

2 files

0.34.1

2 files

0.34.0

2 files

0.33.1

2 files

0.33.0

2 files

0.32.0

2 files

0.31.0

2 files

0.30.0

2 files

0.29.1

2 files

0.29.0

2 files

0.28.3

2 files

0.28.2

2 files

0.28.1

2 files

0.28.0

2 files

0.27.2

2 files

0.27.1

2 files

0.27.0

2 files

0.26.16

2 files

0.26.15

2 files

0.26.14

2 files

0.26.13

2 files

0.26.12

2 files

0.26.11

2 files

0.26.10

2 files

0.26.9

2 files

0.26.8

2 files

0.26.7

2 files

0.26.6

2 files

0.26.5

2 files

0.26.4

2 files

0.26.3

2 files

0.26.2

2 files

0.26.1

2 files

0.26.0

2 files

0.25.1

2 files

0.25.0

2 files

0.24.5

2 files

0.24.4

2 files

0.24.3

2 files

0.24.2

2 files

0.24.1

2 files

0.24.0

2 files

0.23.0

2 files

0.22.4

2 files

0.22.3

2 files

0.22.2

2 files

0.22.1

2 files

0.22.0

2 files

0.21.3

2 files

0.21.2

2 files

0.21.1

2 files

0.21.0

2 files

0.20.0

2 files

0.18.1

2 files

0.18.0

2 files

0.17.0

2 files

This release

0.16.6 This release

2 files

0.16.5

2 files

0.16.4

2 files

0.16.3

2 files

0.16.2

2 files

0.16.1

2 files

0.16.0

2 files

0.15.0

2 files

0.14.1

2 files

0.14.0

2 files

0.13.0

2 files

0.12.4

2 files

0.12.3

2 files

0.12.2

2 files

0.12.1

2 files

0.12.0

2 files

0.11.0

2 files

0.10.0

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.0

2 files

0.7.0

2 files

0.6.0

2 files

0.5.0

2 files

0.4.0

2 files

0.3.0

2 files

0.2.0

2 files

0.0.1

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page