Skip to main content

StreamWright Spark

PyPI Python License Docs

kind: SparkJob for StreamWright: declarative batch pipelines on Apache Spark.

  • Read and write object storage, Delta Lake, databases, Kafka and SQS with StreamWright connectors.
  • Exactly-once batches: batch state, leases and output receipts; replay-safe writes.
  • Any source connector (REST APIs, ads platforms) through driver-side call steps.
  • Runs locally, on Spark standalone and on AWS EMR on EC2.

Install

pip install "streamwright-spark[spark]"    # needs Java 11+
sw connectors -i files

Example

# orders.yaml
kind: SparkJob
version: 1
name: daily_orders
connections:
  lake: {provider: files, roots: ["/data/lake/"]}
steps:
  - name: orders
    connection: lake
    operation:
      service: object
      method: read
      arguments: {path: "raw/orders/*.csv", format: csv, options: {header: true}}
  - name: daily_totals
    sql: SELECT CAST(ordered_at AS DATE) AS day, COUNT(*) AS orders FROM orders GROUP BY 1
outputs:
  - name: save_totals
    connection: lake
    operation:
      service: object
      method: write
      arguments: {path: "/data/lake/curated/daily_totals/", format: parquet, mode: overwrite}
    sql: SELECT * FROM daily_totals
runtime:
  state: {sqlite: /data/state/daily_orders.db}
  platform: {type: local}
sw spark validate orders.yaml
sw spark run orders.yaml --once

Documentation · StreamWright

Metadata

Release files for streamwright-spark 0.1.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for streamwright-spark 0.1.5
File Size Uploaded Provenance Actions
streamwright_spark-0.1.5.tar.gz 127.0 kB Without provenance Details for streamwright_spark-0.1.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for streamwright-spark 0.1.5
File Interpreter ABI Platform Actions
streamwright_spark-0.1.5-py3-none-any.whl Python 3 none any Details for streamwright_spark-0.1.5-py3-none-any.whl

Total release size: 253.1 kB

Release files / streamwright_spark-0.1.5.tar.gz

Download URL streamwright_spark-0.1.5.tar.gz
Size 127.0 kB
Tags Source
SHA-256 checksum
How to use checksums
d1ade909fbe7361bdbdfccc2b719f4ad022eb8ec9b62ceafc5e995b2f90e6880
BLAKE2b-256 checksum
How to use checksums
0684e1693f898a8d9b50de9f789749bcb76012a618a42a0abe27bf39a4f583de
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.11

Release files / streamwright_spark-0.1.5-py3-none-any.whl

Download URL streamwright_spark-0.1.5-py3-none-any.whl
Size 126.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0519d0f97e8bc0965a010eb9052369e3c6814f0d16f477a2efc0b61059e65a11
BLAKE2b-256 checksum
How to use checksums
56271920a0fc60d6ccd98a7dd4f2a6f9f4e945ef5fc238b66310d481ddbf3652
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.11.11

Security for streamwright-spark 0.1.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page