StreamWright Spark
kind: SparkJob for StreamWright: declarative batch pipelines on Apache Spark.
- Read and write object storage, Delta Lake, databases, Kafka and SQS with StreamWright connectors.
- Exactly-once batches: batch state, leases and output receipts; replay-safe writes.
- Any source connector (REST APIs, ads platforms) through driver-side call steps.
- Runs locally, on Spark standalone and on AWS EMR on EC2.
Install
pip install "streamwright-spark[spark]" # needs Java 11+
sw connectors -i files
Example
# orders.yaml
kind: SparkJob
version: 1
name: daily_orders
connections:
lake: {provider: files, roots: ["/data/lake/"]}
steps:
- name: orders
connection: lake
service: object
method: read
arguments: {path: "raw/orders/*.csv", format: csv, options: {header: true}}
- name: daily_totals
sql: SELECT CAST(ordered_at AS DATE) AS day, COUNT(*) AS orders FROM orders GROUP BY 1
outputs:
- name: save_totals
connection: lake
service: object
method: write
arguments: {path: "/data/lake/curated/daily_totals/", format: parquet, mode: overwrite}
sql: SELECT * FROM daily_totals
runtime:
state: {sqlite: /data/state/daily_orders.db}
platform: {type: local}
sw spark validate orders.yaml
sw spark run orders.yaml --once
Links
Metadata
Release files for streamwright-spark 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| streamwright_spark-0.1.0.tar.gz | 103.7 kB | Without provenance | Details for streamwright_spark-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| streamwright_spark-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for streamwright_spark-0.1.0-py3-none-any.whl |
Total release size: 193.5 kB
Release files / streamwright_spark-0.1.0.tar.gz
| Download URL | streamwright_spark-0.1.0.tar.gz |
|---|---|
| Size | 103.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
2bb88962e175153b2717768be22a1888d22e106514d47c75782c067dc8c78fdd
|
|
BLAKE2b-256 checksum How to use checksums |
aa6cbb87bb4965b7cc4aa734cdaef26b507debf298633351f216b7a69082f285
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.7
|
Release files / streamwright_spark-0.1.0-py3-none-any.whl
| Download URL | streamwright_spark-0.1.0-py3-none-any.whl |
|---|---|
| Size | 89.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
47a9ba60a4af12a7476e58827af4301bd98c7c55e8d2bbc0efd09041964f02b9
|
|
BLAKE2b-256 checksum How to use checksums |
6cfb59f926aef721671575bd4a2816a6619717b0eb3a8d5ad645558e6d8c0fe4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.11.7
|
Security for streamwright-spark 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.