PipeWhy: data integrity and lineage diagnostics

Capture data. Detect drift. Follow the evidence.

What is it?

PipeWhy is a Python package for finding changes in versioned data and explaining them using captured observations and declared lineage. It normalizes typed values, builds composable fingerprints, stores immutable local snapshots and compares historical captures with explicit uncertainty.

The first release targets accidental drift and corruption detection. Missing reads, unknown upstream lineage and incomparable observations stay visible in reports. Capture time alone does not establish an operation's timing or cause. Fingerprints do not provide an adversarial integrity guarantee.

Table of contents

• Main features
• Where to get it
• Dependencies
• Installation from sources
• A first snapshot
• Documentation
• Background
• Getting help
• Development and contributing
• License

Main features

• Explicit equality rules: strict representation by default, with a separate opt-in logical normalization profile.
• Versioned fingerprints: reference Python execution, optional native CPU execution and an experimental CUDA backend with recorded hardware qualification.
• Durable local snapshots: saved schema, identity, coverage and retention context support comparisons and queries after the source goes offline.
• Qualified diagnostics: historical comparison, declared lineage, incident analysis and integrity reports preserve the limits of their evidence.
• Useful CLI output: scan, diff, lineage, why, report integrity and export, with text, JSON, checksummed CSV and graph formats where supported.
• Pinned Enron integration: the public corbt/enron-emails dataset is read at an exact revision, with verified cache contents and explicit unknown upstream lineage.
• Resumable local workers: the Python execution API coordinates partitions, finite retries, durable attempts and replay-safe snapshot publication.

Where to get it

Source and tagged releases are maintained in oakplume/pipewhy. The source repository is private; repository and release downloads require access.

The first release series is a development prerelease. To install from PyPI:

python -m pip install --pre pipewhy

See PyPI for published versions and GitHub Releases for wheel downloads and checksums. The installation guide also describes building and installing a local wheel.

Dependencies

The core wheel has no runtime dependencies. Optional extras enable specific features; Poetry locks the development and documentation environments separately.

Python 3.14 and Poetry 2.5.x are required for development. Current qualification uses CPython 3.14.7. Exact dependency constraints live in pyproject.toml and poetry.lock. Read the CUDA guide before enabling that backend.

Installation from sources

From the repository root:

poetry config virtualenvs.in-project true --local
poetry sync
poetry run pipewhy --help

This creates the project .venv. Use the Poetry executable from your tool environment if poetry is not on your PATH.

For native CPU execution and the Enron source:

poetry sync --extras cpu --extras enron

Include every extra you intend to keep when running poetry sync. Fixtures and the Enron corpus are external source data; they are not bundled in the wheel.

A first snapshot

Capture the fixture supplied with this checkout using the reference backend:

poetry run pipewhy scan fixture --config examples/local-fixture.toml --source-version baseline-v1 --retain value --format json

The example writes to .artifacts/local-snapshots and uses repository test data. Value retention here is explicit. The quickstart continues with changed and missing observations, comparisons and saved reports. For a repeatable demonstration, use the PowerShell 7 local slice script.

Documentation

Read the PipeWhy documentation for getting started, the user guide, API reference and developer guide. It includes search, section navigation and light/dark modes.

To build the static HTML and offline ZIP from the source checkout:

poetry sync --with docs
poetry run python tools/build_docs.py

Open docs/_build/html/index.html, or extract dist/pipewhy-docs.zip and open index.html. The documentation build guide describes local previews and the generated files.

Background

PipeWhy separates source identity, normalized representation, fingerprints, observations and lineage evidence. Comparisons carry that context forward so a report can distinguish detected drift from missing or incompatible information. Saved queries work with the core package without reopening source data.

The architecture guide links the versioned contracts and acceptance records. Local benchmark results retain their measured hardware, source and dependency provenance; they are not universal performance claims.

Getting help

Use the issue tracker for bugs, questions and proposals. Include the package version, command or API call, backend, expected result and a minimal reproducible example. Keep private data and credentials out of reports. poetry run pipewhy --help lists CLI commands.

Development and contributing

Follow the contributor requirements and developer guide. Work in small, meaningful commits tied to issues. Tests must check behavior, including failure paths.

poetry run python tools/quality.py
poetry run python -m pytest tests/gates tests/integration
poetry run python tools/build_docs.py
poetry build

The quality gate enforces Black, ty, dependency checks and pytest under Coverage.py, with at least 95% statement and combined coverage and no coverage exclusions. CI also builds the documentation with warnings treated as errors, rebuilds the source distribution and checks clean installed packages for silent runtime breaks. See the packaging guide.

License

Current package metadata is LicenseRef-Proprietary. No public open-source license has been selected. Distribution preparation does not grant redistribution rights or authorize publication.