pyspark-streaming-base 0.1.0

This project provides a set of base classes that simplify the art of crafting bullet-proof Spark Structured Streaming applications.

pyspark-streaming-base

This project provides a set of base classes that simplify the art of crafting bullet-proof Spark Structured Streaming applications.

The Environment

• uv
• Spark 4.0.1. This should match the pyspark version in the pyproject.toml

Java Home

Setup on Mac

brew install openjdk@17
brew install openjdk@21

Here is an example from my local install. I needed to manually symlink since java 23 was taking over the mac

ln -s /opt/homebrew/Cellar/openjdk@21/21.0.8/libexec/openjdk.jdk /Users/scotthaines/Library/Java/JavaVirtualMachines/openjdk@21.0.8

Check that your version exists

/usr/libexec/java_home -V

The following I added to my ~/.zshrc. Spark 4.0.1 supports Java 17 or 21, and Scala 2.13. Use jdk 21.

export JAVA_HOME=$(/usr/libexec/java_home -v 21)

Using uv : https://docs.astral.sh/uv/getting-started/features/#projects

uv python install 3.13

Sync to match the uv.lock

uv sync

uv build

uv run pytest --cov=pyspark_streaming_base --cov-report term