StreamBuild

Declarative ClickHouse streaming pipeline deployment for staged backfill, audit, and publish workflows.

streambuild is aimed at streaming data teams who want dbt-like authored models, but with deployment semantics that fit live ClickHouse pipelines:

  • plan rebuilds conservatively
  • create staged shadow objects
  • backfill history into the staged path
  • audit staged readiness
  • publish by switching stable logical views

The current product is centered on ClickHouse and streaming replay semantics. Kafka-backed sources work today, and adopted external streaming tables are now supported as replay roots.

Current Status

Current implemented workflow:

  • stb plan
  • stb build
  • stb backfill
  • stb audit backfill
  • stb publish
  • stb doctor
  • stb repair active-view
  • stb reconcile
  • stb compile
  • stb janitor

Current rollout model:

  • plan is read-only
  • backfill starts a real staged deployment
  • audit backfill inspects staged readiness
  • publish switches stable logical views to staged physical tables

Installation

Requirements:

  • Python >=3.12
  • ClickHouse

Local dev install:

uv sync

Run the CLI with:

uv run stb --help

Project Shape

StreamBuild projects are authored as a project root plus pipeline folders.

streambuild_project.toml
sources/
  orders.yml
macros/
  common.py
pipelines/
  orders/
    orders_enriched.sql
    order_rollups.sql

Rules:

  • each direct child folder under pipelines/ is one pipeline
  • recursive *.sql files under that folder belong to that pipeline
  • pipeline name is inferred from the folder name
  • pipeline source is inferred transitively from model driving inputs
  • model name is inferred from the SQL filename stem
  • optional pipeline.toml stores pipeline-wide virtual-environment policy

Macros

Public Python modules under macros/ are loaded once per project analysis. Functions defined by those modules are available in authored model, test, and audit SQL as @function_name(...). Imported functions, async functions, __init__.py, and modules or directories whose names start with _ are not registered.

from streambuild.compiler.macros.models import MacroContext


def qualified_source(ctx: MacroContext, table_name: str) -> str:
    return f"{ctx.database}.{table_name}"

SELECT * FROM @qualified_source("orders")

Macro modules are trusted project code, not a sandbox: module-level code runs during analysis, and a macro may perform anything allowed to that Python process. Calls accept only nested Python literals (str, bool, int, float, None, lists, tuples, and dictionaries with scalar keys) plus nested macro results. A first parameter named ctx must be annotated as MacroContext; StreamBuild supplies its immutable project target, adapter, database, virtual-environment, and variable values. Direct SQL macro calls must return strings. Errors report both the authored SQL call and the defining macro source.

Project Config

Committed project configuration lives in streambuild_project.toml. Developer-specific overrides may live in the gitignored streambuild_local.toml.

name = "orders_project"
default_target = "dev"

[settings]
virtual_environments = true

[connection]
host = "localhost"
port = 8123
username = "clickhouse"
password = "${ENV:CLICKHOUSE_PASSWORD}"

[defaults]
managed_source_ttl = "_replay_landed_at + INTERVAL 14 DAY"

[targets.dev]
database = "analytics"

Notes:

  • name and default_target are required; adapter defaults to clickhouse
  • target selection is CLI --target, local target, then project default_target
  • CLI --vars accepts one JSON object for ${name} interpolation
  • connection templates are expanded only for commands that connect
  • metadata lives in the same database by default
  • connection precedence is CLI flags, fixed STREAMBUILD_CLICKHOUSE_* environment variables, local config, selected target, then project config

Pipeline Sources

Reusable replay-driving sources live under sources/*.yml. StreamBuild follows each model's __source(...) or untyped __ref(...) driving input until it reaches a registered source. Every pipeline must resolve to exactly one source.

Managed Kafka Landing

sources:
  - kind: kafka
    name: orders
    broker_list: kafka:9092
    topic: source.orders.created
    ttl: _replay_landed_at + INTERVAL 30 DAY
    replay_boundary:
      mode: offsets

This is the managed source shape:

  • StreamBuild creates the Kafka table
  • StreamBuild creates the raw landing table and landing MV
  • source ttl overrides [defaults].managed_source_ttl; omitting both keeps data indefinitely
  • downstream models usually read the source via __source("orders")

Adopted External Source

sources:
  - kind: stream_table
    name: orders
    table_name: orders_existing
    replay_boundary:
      mode: offsets
      columns:
        _replay_partition: event_partition
        _replay_offset: event_offset
        _replay_timestamp: event_timestamp

This is the adopted-source shape:

  • StreamBuild does not create the source table
  • the source table must already exist in the resolved project database
  • table_name must currently be a bare table name
  • replay boundary columns are validated against the live table schema during planning/runtime commands

Current replay-boundary rules for adopted sources:

  • mode: offsets requires partition, offset, and timestamp
  • mode: offsets does not allow landed_at
  • mode: timestamp requires timestamp
  • mode: timestamp does not allow landed_at
  • mode: cursor requires cursor and timestamp

Currently supported external-source replay boundary modes:

  • offsets

  • timestamp

  • cursor

Virtual-environment projects can choose change-driven replay independently from the fallback used when bounded replay cannot preserve aggregate history:

bounded_replay_fallback = "bounded_without_history"

[replay_on_change]
breaking = "full"
non_breaking = "bounded-7d"

This optional pipeline.toml sits directly in the pipeline directory. The same policies can be defaults in streambuild_project.toml and overrides in a model MODEL(...) header. They are rejected when settings.virtual_environments is false.

Models

Each SQL model starts with a MODEL (...) header.

MODEL (
  engine "MergeTree()",
  order_by ["order_id", "_replay_partition", "_replay_offset"],
  partition_by "toYYYYMM(event_at)",
  ttl "event_at + INTERVAL 30 DAY",
  settings (
    index_granularity 8192,
  ),
  replay_anchor auto,
);

SELECT
  CAST(order_id AS UInt64) AS order_id,
  CAST(event_at AS DateTime64(3)) AS event_at,
  CAST(_replay_partition AS Int32) AS _replay_partition,
  CAST(_replay_offset AS Int64) AS _replay_offset
FROM __source("orders")

Notes:

  • the driving replay input may be declared with __source(...) for source roots or __ref(...) for managed upstream models
  • additional managed dependencies are declared with __ref(...)
  • additional __ref(...) dependencies must declare ref_type
  • header fields use SQLBuild syntax: whitespace-separated key value entries, lists in [...], and nested mappings in (...)
  • omitted SQL storage settings default to engine "MergeTree()" and order_by ["_replay_timestamp"]
  • both CAST(expr AS Type) and expr::Type are accepted

Replay Lineage

StreamBuild exposes a normalized replay lineage surface.

Current intent:

  • _replay_* is the normalized source-agnostic replay vocabulary

Current generic replay columns:

  • _replay_partition
  • _replay_offset
  • _replay_timestamp
  • _replay_landed_at
  • _replay_cursor

Current behavior:

  • managed Kafka landing populates the normalized _replay_* lineage columns directly
  • adopted sources map declared physical source columns into the normalized replay surface
  • downstream managed outputs should preserve _replay_* when they need replay lineage

Core Commands

From a project directory:

uv run stb plan
uv run stb build
uv run stb backfill
uv run stb audit backfill
uv run stb publish
uv run stb doctor
uv run stb repair active-view --table tbl__orders
uv run stb reconcile
uv run stb compile
uv run stb janitor

From outside the project directory:

uv run stb plan --project-dir examples/orders_demo

Compile Artifacts

stb compile writes artifacts under project-level target/.

Static compile products and runtime evidence have separate owners:

target/
  manifest.json
  streambuild_dag.json
  compiled/
    models/<pipeline>/
    resources/
      sources/<source>/
      models/<pipeline>/
    workflows/<pipeline>/
      steps/
      workflow.sql
      workflow.json
    audits/
    tests/
  run/
    tests/

stb compile atomically replaces only the static owners and never writes under target/run/. Runtime commands own their command-specific subtrees.

The compile manifest includes:

  • resolved database
  • relations
  • source metadata
  • model specs
  • logical tests and audits
  • realized adapter resources
  • every emitted static artifact path
  • workflow paths and logical DAG identity

Example

See examples/orders_demo/ for a runnable local demo using:

  • Redpanda
  • ClickHouse
  • a synthetic producer
  • a real streambuild project

Demo README:

  • examples/orders_demo/README.md

Development

Useful commands:

make format
make lint
make type
make test
make test-all
make check
make verify

Current meanings:

  • make check: fast structural and static validation
  • make verify: full validation including tests

Testing

The repo uses:

  • unit tests under tests/unit
  • integration tests under tests/integration
  • end-to-end tests under tests/e2e

Recent coverage includes:

  • staged backfill / audit / publish flows
  • active-view diagnosis and repair
  • adopted external replay sources
  • normalized replay lineage behavior

Scope Notes

Current intentional limitations:

  • ClickHouse-only runtime
  • external adopted sources must resolve in the project database
  • managed Kafka sources support offsets, timestamp, and landed_at; adopted relations support offsets, timestamp, and cursor

This repo is actively evolving around staged rollout correctness, replay semantics, and migration/adoption support for existing ClickHouse streaming tables.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.39.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for kvlonge from gravatar.com kvlonge

Requires

Python >=3.12

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

streambuild-0.2.0.tar.gz (698.2 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

streambuild-0.2.0-py3-none-any.whl (449.5 kB view details)

Uploaded Python 3

File details

Details for the file streambuild-0.2.0.tar.gz.

File metadata

  • Download URL: streambuild-0.2.0.tar.gz
  • Upload date:
  • Size: 698.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for streambuild-0.2.0.tar.gz
Algorithm Hash digest
SHA256 5866c1bc86c003f36adf03a2a27ced6d04c2121ca32bef2a405b0028aa1d3192
MD5 189f9b9f1e0c3bc3bb8f167dfae2f7e2
BLAKE2b-256 001f64bc35f7af1d5053d5bb1de15c3fd3bdde59b451d4d769c254c486055b59

See more details on using hashes here.

Provenance

The following attestation bundles were made for streambuild-0.2.0.tar.gz:

Publisher: publish.yml on chio-labs/streambuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file streambuild-0.2.0-py3-none-any.whl.

File metadata

  • Download URL: streambuild-0.2.0-py3-none-any.whl
  • Upload date:
  • Size: 449.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for streambuild-0.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 4f7b0b5d36cc21d30407933cee24664fced627e86d692bf761279b419cd41a14
MD5 f29f4648f917a3e52bc1231fcb31891e
BLAKE2b-256 9a8fa2b59b4310903755673de2de4b02c235b334453a5e4824df1836f5de780a

See more details on using hashes here.

Provenance

The following attestation bundles were made for streambuild-0.2.0-py3-none-any.whl:

Publisher: publish.yml on chio-labs/streambuild

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.39.0

2 files

0.38.4

2 files

0.38.3

2 files

0.38.2

2 files

0.38.1

2 files

0.38.0

2 files

0.37.0

2 files

0.36.0

2 files

0.35.1

2 files

0.35.0

2 files

0.34.4

2 files

0.34.3

2 files

0.34.2

2 files

0.34.1

2 files

0.34.0

2 files

0.33.1

2 files

0.33.0

2 files

0.32.0

2 files

0.31.0

2 files

0.30.0

2 files

0.29.1

2 files

0.29.0

2 files

0.28.3

2 files

0.28.2

2 files

0.28.1

2 files

0.28.0

2 files

0.27.2

2 files

0.27.1

2 files

0.27.0

2 files

0.26.16

2 files

0.26.15

2 files

0.26.14

2 files

0.26.13

2 files

0.26.12

2 files

0.26.11

2 files

0.26.10

2 files

0.26.9

2 files

0.26.8

2 files

0.26.7

2 files

0.26.6

2 files

0.26.5

2 files

0.26.4

2 files

0.26.3

2 files

0.26.2

2 files

0.26.1

2 files

0.26.0

2 files

0.25.1

2 files

0.25.0

2 files

0.24.5

2 files

0.24.4

2 files

0.24.3

2 files

0.24.2

2 files

0.24.1

2 files

0.24.0

2 files

0.23.0

2 files

0.22.4

2 files

0.22.3

2 files

0.22.2

2 files

0.22.1

2 files

0.22.0

2 files

0.21.3

2 files

0.21.2

2 files

0.21.1

2 files

0.21.0

2 files

0.20.0

2 files

0.18.1

2 files

0.18.0

2 files

0.17.0

2 files

0.16.6

2 files

0.16.5

2 files

0.16.4

2 files

0.16.3

2 files

0.16.2

2 files

0.16.1

2 files

0.16.0

2 files

0.15.0

2 files

0.14.1

2 files

0.14.0

2 files

0.13.0

2 files

0.12.4

2 files

0.12.3

2 files

0.12.2

2 files

0.12.1

2 files

0.12.0

2 files

0.11.0

2 files

0.10.0

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.0

2 files

0.7.0

2 files

0.6.0

2 files

0.5.0

2 files

0.4.0

2 files

0.3.0

2 files

This release

0.2.0 This release

2 files

0.0.1

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page