PySpark - Fine-grained data provenance - PySpark part

Table of Content (ToC)

• PySpark - Fine-grained data provenance - PySpark part
  • Table of Content (ToC)
  • Overview
  • Quick Start
  • Project Layout
  • Common commands
  • Development life-cycle
  • PyPI package
  • CI/CD

Created by gh-md-toc

Overview

Python package in this monorepo for enabling Spark data provenance features in PySpark jobs.

This project is managed with uv.

Quick Start

From the repository root:

cd pyspark-wringlet
make init-uv-python
make init
make check
make test
make run

Project Layout

pyspark-wringlet/
    src/wringlet/
    tests/
    main.py
    pyproject.toml
    Makefile

Common commands

• make init - Create/refresh lock file and sync dependencies
• make update - Upgrade dependencies and sync environment
• make check - Run lint and type checks
• make test - Run unit tests
• make build - Build package wheel
• make publish - Publish package (PyPI)

Development life-cycle

• In order to switch (bump) to a newer version of the Python package, the following are the main options, ordered by the general probability of occurrence in the development life-cycle, from the highest to the lowest)
  • Increment the dev version (e.g., from 2.4.3.dev5 to 2.4.3.dev6 or from 2.4.3 to 2.4.4.dev0): make increment-dev-version
  • Bump to minor version (e.g., from 2.4.3.dev5 to 2.4.3): make bump-to-minor-version
  • Bump to patch version (e.g., from 2.4.3.dev5 to 2.4.4): make bump-to-patch-version
  • Bump to major version (e.g., from 2.4.3.dev5 to 3.0.0): make bump-to-major-version
• Then, the version bump has to be cascaded to related files (typically, pyproject.toml)

PyPI package

• The Python package is published on PyPI

• Optionally, the Python wheel may be published manually onto Pypi.org

  • The UV_PUBLISH_TOKEN environment variable then needs to be specified (see the uv documentation for further details)

make publish

• However, the easiest way to publish the Python wheel is through the CI/CD pipeline, that is, the GitHub Actions Python publishing pipeline.

  • That CI/CD pipeline is automatically triggered when creating a release on the Git repository
  • It may also be triggered manually by contributors of the Git repository

• Check the versions of the Python wheel on Pypi.org

• Install the Python wheel locally, in the main/global Python environment (as spark-submit does not seem to work properly with uv):

make install-local

• For information, that actually install the Python wheel with pip:

python -mpip install -U pyspark-data-provenance

• Run the PySpark job locally, in the main/global Python environment (as spark-submit does not seem to work properly with uv):

make run-local

CI/CD

Repository-level workflows are provided for:

• CI: lint, type-check, and tests on pushes/PRs affecting this package
• Publish: build and publish on release