Streamcase

Streamcase is an early-development pytest-oriented toolkit for deterministic Apache Spark Structured Streaming tests.

Alpha status: the first prerelease is available on PyPI. Its Spark runner, including checkpoint-preserving restart actions, scenario model, results, and assertions are implemented. The API is experimental and may change in later alphas; the first-alpha scope is narrower than the longer-term roadmap.

The first-alpha release notes describe the tested support matrix and limitations. Check PyPI for available versions.

Problem

Ordinary DataFrame assertions do not exercise streaming semantics such as micro-batch boundaries, late data, watermarks, checkpoint restarts, or state growth. Streamcase will make those behaviors explicit and reproducible in Python tests while executing against Spark's public Structured Streaming interfaces.

Installation

Install the lightweight backend-independent package from PyPI. The --pre option allows alpha releases; pin a version when reproducibility matters:

python -m pip install --pre streamcase

For the Spark runner, install the optional spark extra and provide a compatible Java JDK separately:

python -m pip install --pre "streamcase[spark]"

The Spark extra keeps PySpark out of the core dependency set. See the Spark compatibility policy for the supported Python, Java, and Spark combination. Contributors working from a checkout can use the editable-install instructions.

Available today

Streamcase can describe immutable logical input batches and checkpoint-preserving restart boundaries, represent captured output, and assert row equality, batch counts, and unique keys without importing PySpark. The public Spark runner executes batches and restarts against a caller-owned session. See the scenario model guide and results and assertions guide, plus the two-batch Spark quick start and runner contract and checkpoint restart example.

Planned capabilities

• progress, watermark, and state-store assertions;
• pytest fixtures;
• compatibility testing across supported Spark releases.

These items are roadmap goals, not current package features. Each capability has its own issue, tests, documentation, and pull request.

Repository status

This bootstrap establishes:

• Python packaging with a src layout;
• Apache License 2.0 and project governance;
• issue and pull-request conventions;
• formatting, linting, type checking, tests, and package validation in CI;
• security and release policies.

See ROADMAP.md for the planned delivery sequence and docs/architecture.md for the proposed technical shape.

Local checks

python -m venv .venv
python -m pip install -e ".[dev]"
ruff format --check .
ruff check .
mypy
pytest -m "not spark"
python -m build
python -m twine check dist/*

Spark contributors can run the isolated integration suite with the setup in the contribution guide.

Contributing

All material work starts with a GitHub issue and reaches main only through a reviewed pull request. Read CONTRIBUTING.md before making a change.

Trademark notice

Apache Spark, Spark, Apache Flink, and Flink are trademarks of the Apache Software Foundation. Streamcase is independent software and is not endorsed by the Apache Software Foundation.

License

Licensed under the Apache License 2.0.