This release is a pre-release and may not be stable for production use.
SegmentStream pipeline SDK
segmentstream-pipeline is the small runtime contract between a workspace's
Dagster definitions and the warehouse provisioned by SegmentStream. Dagster
continues to own assets and jobs, while Ibis continues to own relational
expressions. This package supplies lazy runtime configuration and durable
warehouse I/O.
The initial connector supports BigQuery and full-table replacement. It reads the following non-secret configuration when a pipeline first accesses the warehouse:
SEGMENTSTREAM_WAREHOUSE_ENGINESEGMENTSTREAM_WAREHOUSE_CATALOGSEGMENTSTREAM_WAREHOUSE_DEFAULT_NAMESPACESEGMENTSTREAM_WAREHOUSE_LOCATION(optional)
Configuration and authentication are deliberately lazy. Importing and
validating definitions.py during a deployment build does not connect to a
warehouse. In Cloud Run, the BigQuery connector uses the attached workload
identity through Application Default Credentials.
import dagster as dg
import ibis.expr.types as ir
from segmentstream import external_table_asset, warehouse_resources
raw_orders = external_table_asset(key=["raw", "orders"])
@dg.asset(
key=["analytics", "normalized_orders"],
io_manager_key="warehouse_io",
kinds={"ibis"},
)
def normalized_orders(raw_orders: ir.Table) -> ir.Table:
return raw_orders.filter((raw_orders.status == "completed") & (raw_orders.amount > 0))
defs = dg.Definitions(
assets=[raw_orders, normalized_orders],
resources=warehouse_resources(),
)
Asset keys map to relations using a small convention:
["orders"]uses the configured default namespace.["analytics", "orders"]uses the explicitanalyticsnamespace.- Other key shapes are rejected.
For local package development, install this project in editable mode rather than adding a relative path dependency to a deployable pipeline.
Published workspace pipelines will install the provider extra:
python -m pip install "segmentstream-pipeline[bigquery]"
Releases
Releases use the version declared in pyproject.toml and are published from the
pipeline-sdk-v<version> Git tag by the protected pipeline-sdk-release.yml
workflow. The workflow builds the wheel and source distribution in a job without
publishing credentials, then uses PyPI Trusted Publishing from the pypi GitHub
environment. No long-lived PyPI token is stored in GitHub.
PyPI releases are immutable. Increment the package version before creating a new release tag; do not reuse a version that has already been uploaded.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file segmentstream_pipeline-0.1.0a1.tar.gz.
File metadata
- Download URL: segmentstream_pipeline-0.1.0a1.tar.gz
- Upload date:
- Size: 76.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
92d996749669cc5a8a3eab74da7fe78e20510f8e8acc78dca5670169058f8157
|
|
| MD5 |
f8930907be47d4b1ffc394bd57f02a98
|
|
| BLAKE2b-256 |
2ee7b9f36c48084b181aace130d6ead3b5bcd9af0aa5408036f3541851482cc0
|
Provenance
The following attestation bundles were made for segmentstream_pipeline-0.1.0a1.tar.gz:
Publisher:
pipeline-sdk-release.yml on segmentstream/segmentstream
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
segmentstream_pipeline-0.1.0a1.tar.gz -
Subject digest:
92d996749669cc5a8a3eab74da7fe78e20510f8e8acc78dca5670169058f8157 - Sigstore transparency entry: 2623686229
- Sigstore integration time:
-
Permalink:
segmentstream/segmentstream@85a838788ffd2b1e988c39c0244495d6f4a541e2 -
Branch / Tag:
refs/tags/pipeline-sdk-v0.1.0a1 - Owner: https://github.com/segmentstream
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
pipeline-sdk-release.yml@85a838788ffd2b1e988c39c0244495d6f4a541e2 -
Trigger Event:
push
-
Statement type:
File details
Details for the file segmentstream_pipeline-0.1.0a1-py3-none-any.whl.
File metadata
- Download URL: segmentstream_pipeline-0.1.0a1-py3-none-any.whl
- Upload date:
- Size: 13.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ac3e6f092baf87634ed8d4f8d10235d8408fb1610962131e14ccc419c2649a50
|
|
| MD5 |
bfa43d5ac8274a0a575dc24d1a48006a
|
|
| BLAKE2b-256 |
9c2fc6aa6ea11b162920226b3cf7abe6973d581b689b06f036990f696d389b49
|
Provenance
The following attestation bundles were made for segmentstream_pipeline-0.1.0a1-py3-none-any.whl:
Publisher:
pipeline-sdk-release.yml on segmentstream/segmentstream
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
segmentstream_pipeline-0.1.0a1-py3-none-any.whl -
Subject digest:
ac3e6f092baf87634ed8d4f8d10235d8408fb1610962131e14ccc419c2649a50 - Sigstore transparency entry: 2623686360
- Sigstore integration time:
-
Permalink:
segmentstream/segmentstream@85a838788ffd2b1e988c39c0244495d6f4a541e2 -
Branch / Tag:
refs/tags/pipeline-sdk-v0.1.0a1 - Owner: https://github.com/segmentstream
-
Access:
private
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
pipeline-sdk-release.yml@85a838788ffd2b1e988c39c0244495d6f4a541e2 -
Trigger Event:
push
-
Statement type:
