This release is a pre-release and may not be stable for production use.
SegmentStream pipeline SDK
Declare source tables with s.ingestion and warehouse SQL transformations with
s.model. The SDK resolves inputs, validates explicit partitioning and capabilities,
and compiles native Dagster definitions using the existing warehouse loader.
from collections.abc import Iterator
import ibis.expr.types as ir
import segmentstream as s
@s.ingestion(
tier=s.Tier.Bronze,
partitioning=s.Partitioning.Unpartitioned,
schema={"amount": "float64"},
)
def orders() -> Iterator[s.IngestionRow]:
yield {"amount": 25.0}
@s.model(
tier=s.Tier.Gold,
partitioning=s.Partitioning.Unpartitioned,
schema={"total_revenue": "float64"},
)
def revenue(orders: ir.Table) -> ir.Table:
return orders.aggregate(total_revenue=orders.amount.sum())
defs = s.definitions(
assets=[orders, revenue],
jobs=[s.job("revenue", assets=[revenue], include_upstream=True)],
)
The workspace documentation is organized by concept: connections, ingestion, models, partitioning, jobs, execution, checks, runtime values, and warehouse storage. The CLI bundles these same pages into every starter workspace.
s.validate(defs) checks the graph and named job configuration without executing
assets. s.describe(defs) explains resolved inputs, per-asset policies, storage,
capabilities, and job selections without resolving credentials. See the
modular commerce starter
for daily row ingestion, batched models, and a partitioned check.
Compatibility
The SDK contains authoring, graph inspection, Ibis resources, connection HTTP clients, and managed secret access. Managed orchestration, workers, PostgreSQL setup, and the metadata webserver live in the internal pipeline runtime.
In 0.1.0a28, use s.get_connection(key).get_http_client() for provider requests.
The client forwards requests through the backend, where provider credentials and
refresh remain. The access-token and full-credential accessors have been removed.
See connections for declaring API destinations.
Workspace pipelines declare segmentstream-pipeline[bigquery]; our image supplies
the runtime. The old SDK cloud-run and webserver extras and
segmentstream.runtime module are removed in a22.
segmentstream.dagster retains its native exports. Native assets and high-level
declarations can coexist in s.definitions. Root s.resource and
s.IngestionResource remain available for legacy ingestion. Import their helpers
from segmentstream; the old public segmentstream.ingestion module is retired
so s.ingestion can be the decorator. Migration examples
cover the mapping and IO defaults.
Managed Google clients
The Sandbox runtime integration supplies refreshable credentials to the built-in BigQuery warehouse and GCS IO manager. Ibis remains the user-facing warehouse API; its internal official Google client renews credentials automatically throughout the authorized run. Ten minutes is the lifetime of each access token, not a limit on pipeline duration. Managed execution does not provide general ADC support. See runtime lifecycle and credential brokering. Managed Dagster execution uses the backend compute provider API for every step, including SQL assets and checks. The coordinator owns scheduling and retries; the backend owns execution identity, scoped credentials and cleanup. See the compute provider contract.
Releases
This change targets 0.1.0a36. Native BigQuery connections support managed
service-account impersonation and direct SQL writes to workspace tables. See
native BigQuery connections.
Publish the SDK before releasing a CLI whose starter
pins it. PyPI versions are immutable; do not republish an existing version.
The protected pipeline-sdk-release.yml workflow builds a wheel and source
distribution from a pipeline-sdk-v<version> tag, then uses PyPI Trusted Publishing
from the pypi GitHub environment. No long-lived publishing token is stored in GitHub.
1 maintainer
Metadata
1 maintainer
Release files for segmentstream-pipeline 0.1.0a36
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| segmentstream_pipeline-0.1.0a36.tar.gz | 175.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| segmentstream_pipeline-0.1.0a36-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 268.7 kB
Release files / segmentstream_pipeline-0.1.0a36.tar.gz
| Download URL | segmentstream_pipeline-0.1.0a36.tar.gz |
|---|---|
| Size | 175.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
70dfdca9b6a21c835a8b81864b730243627f53deb878cc173bfa900b5fca5be8
|
|
BLAKE2b-256 checksum How to use checksums |
ebe7772185c27179fed1852d030b83e9463cc59f1ed98934aec80aa72ad2c54a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 23, 2026.
Transparency logRelease files / segmentstream_pipeline-0.1.0a36-py3-none-any.whl
| Download URL | segmentstream_pipeline-0.1.0a36-py3-none-any.whl |
|---|---|
| Size | 93.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6932135de6b60891b6002212cf43edbe8bbcc76c180fcea3f863c003854a726a
|
|
BLAKE2b-256 checksum How to use checksums |
fd42301e95dee79f17008f05855a2815ddc20e74f97c6e82ef5d9da3c030b258
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 23, 2026.
Transparency log
