Skip to main content

StreamWright

PyPI Python License Docs

Declarative data pipelines in YAML: read from APIs, files, databases and ads platforms, shape with DuckDB SQL, write to files, DuckDB, DuckLake, Singer or dlt.

  • No per-connector code: authentication, paging, retries, rate limits and incremental state are configuration.
  • Connectors on demand from the StreamWright hub: sw connectors -i <name>.
  • Jobs: kind: DuckDBJob SQL pipelines with replay-safe writes; kind: SparkJob with streamwright-spark.

Install

pip install streamwright
sw connectors -i restapi

Example

A source is a folder: source.yaml and one file per stream.

# my_api/source.yaml
kind: source
name: my_api
spec:
  secrets:
    api_token: {type: string}
auth:
  provider: restapi
  type: bearer
  token: "{{ secrets.api_token }}"
http:
  base_url: https://api.example.com
# my_api/streams/items.yaml
requests:
  - name: raw_items
    http: {path: /v1/items}
transform:
  mode: page
  steps:
    - name: items
      select: SELECT record->>'id' AS id, record->>'name' AS name FROM raw_items
export:
  items: {step: items, primary_key: [id]}
sw validate my_api
STREAMWRIGHT_SECRET_API_TOKEN=... sw run my_api --output jsonl:out

Documentation · Quickstart · Source code

Metadata

Release files for streamwright 0.1.12

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for streamwright 0.1.12
File Size Uploaded Provenance Actions
streamwright-0.1.12.tar.gz 444.7 kB Without provenance Details for streamwright-0.1.12.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for streamwright 0.1.12
File Interpreter ABI Platform Actions
streamwright-0.1.12-py3-none-any.whl Python 3 none any Details for streamwright-0.1.12-py3-none-any.whl

Total release size: 894.9 kB

Release files / streamwright-0.1.12.tar.gz

Download URL streamwright-0.1.12.tar.gz
Size 444.7 kB
Tags Source
SHA-256 checksum
How to use checksums
3e7dbb4f2fc3adc4c9c0e83418f9662f7a46fdb8b1856d3c8737c3f63cc97eed
BLAKE2b-256 checksum
How to use checksums
4ddd90202269a1765bdc0e04c503e8ef0767add392b8332f5def33c955827b90
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.7

Release files / streamwright-0.1.12-py3-none-any.whl

Download URL streamwright-0.1.12-py3-none-any.whl
Size 450.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
7bb5b42b9ff8b800f55dc17895295bbf990af4f67cc6716afccf12a462e77f5b
BLAKE2b-256 checksum
How to use checksums
aa25ffa6e9f6b1ab5bb5b827d6e62cd7daaba2a2d8de5c1442940cd9c7fb3a84
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.7

Security for streamwright 0.1.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page