Parity

Behavioural compatibility testing for software migrations.

Parity tries to disprove that a candidate preserves the behaviour of a reference. It sends the same canonical inputs to both sides, compares their observable outcomes, shrinks generated counterexamples, classifies independent differences, and preserves evidence for exact replay.

canonical inputs
      ├──> reference environment ──┐
      └──> candidate environment ──┴──> compare ──> shrink ──> persist/replay

The two sides do not need the same dependency versions, API, implementation, language, runtime or architecture. They need a shared behavioural contract. That makes Parity useful for dependency upgrades, refactors, branch/worktree regression testing, backend changes, rewrites and cross-language replacements.

Parity is an open-source verification engine, not an AI migration product. A human, script or migration agent can use its deterministic results as a gate, but Parity does not generate or repair the candidate.

Parity produces evidence, not a proof of equivalence. PASSED means no difference was found in the configured domain and search budget.

Quick start

The controller requires Python 3.11 or later.

python -m pip install parity-check
parity init
parity check

parity init creates a runnable parity.toml and example module. Replace the example functions with small reference and candidate adapters around the behaviour you care about, then run parity check. Parity handles process isolation, generation, shrinking, artifacts and reporting.

A compact real-project case is:

version = 1

[[cases]]
name = "orders"
fixture = "tests/fixtures/orders.parquet"

[cases.reference]
target = "migration_adapters:reference_orders"
adapter = "pandas"
record_distributions = ["orders-lib"]

[cases.candidate]
target = "migration_adapters:candidate_orders"
adapter = "polars"
record_distributions = ["orders-lib"]

[cases.comparison]
row_order = "keyed"
row_keys = ["order_id"]
dtype = "compatible"
rtol = 1e-7

[cases.generation]
max_examples = 250
max_findings = 4
seed = 20260820

Run the suite, one case, or a tag:

parity check
parity check --case orders --max-examples 1000
parity check --tag critical --json .parity/report.json --junit .parity/junit.xml

Exit 0 means PASSED, exit 1 means FAILED because behaviour or an enforced performance gate differed, and exit 2 means ERROR because Parity could not perform a reliable comparison.

What a run provides

• Fixtures, deterministic boundary cases, Hypothesis search and shrinking.
• First-class Return(canonical_value) and Raise(exception_type, normalized_message, structured_details) outcomes.
• Stable ms3: finding signatures that separate unrelated value, API and exception changes while ignoring volatile paths, addresses, timestamps, IDs and version text.
• Several independently confirmed findings per campaign, with deduplication and tiny witnesses where shrinking applies.
• Isolated reference and candidate processes, timeouts, mutation tracking and runtime/dependency provenance.
• Replayable Arrow inputs, integrity-bound manifests and the effective comparison contract.
• Ordered case-level parallelism without parallel Hypothesis shrinking.
• Median performance ratios with deterministic bootstrap confidence intervals and optional gates.
• Terminal, JSON, Markdown, JUnit and GitHub step-summary output.
• A CLI, Python API, pytest fixture and migration-inventory gate.

Different APIs are normal

Reference and candidate functions do not need matching signatures. Put the translation at the boundary and keep the canonical contract stable:

def reference_quote(frame):
    # Import only the implementation available in the reference environment.
    from legacy import calculate as old_calculate

    row = frame.iloc[0]
    return old_calculate(row.x, row.y, row.currency)


def candidate_quote(frame):
    # Import only the implementation available in the candidate environment.
    from rewritten import Data, Engine

    row = frame.iloc[0]
    return Engine(row.currency).calculate(Data(row.x, row.y))

Shared static_args/static_kwargs and side-specific reference_kwargs/candidate_kwargs cover simple declarative differences. For richer changes, these small project-owned functions are the explicit adapters; Parity does not require old and new APIs to resemble each other. When the sides have conflicting dependencies, keep their imports inside the side-specific functions as above, or put the functions in separate modules. Importing both implementations at the top of one shared module would make preflight fail in an environment that intentionally contains only one of them.

If a target returns a domain object, add an output canonicalizer in that target environment:

[cases.candidate]
target = "migration_adapters:candidate_quote"
canonicalizer = "migration_adapters:quote_to_contract"
adapter = "arrow"

The canonicalizer receives the successful raw return value and produces an Arrow-compatible frame or JSON-compatible value. Exceptions raised by the target remain semantic outcomes; a failure to import or execute the adapter/canonicalizer is infrastructure ERROR.

Target environments stay independent

For Python targets in another environment, set python on each side. The target environment needs only PyArrow, the selected adapter dependency, and the application under test. It does not need parity-check, Pydantic, Hypothesis, Rich or Typer. The controller launches a dependency-light portable worker and validates transport, imports and requested distribution constraints before invoking user code.

[cases.reference]
target = "migration_adapters:reference_orders"
python = ".venv-reference/bin/python"
adapter = "pandas"

[cases.candidate]
target = "migration_adapters:candidate_orders"
python = ".venv-candidate/bin/python"
adapter = "pandas"

This supports conflicting dependencies and older Python target environments without coupling them to the controller's dependency graph. Use parity doctor --config parity.toml for two-phase preflight: it validates both transports/runtimes and declared requirements before importing either endpoint, then checks target, canonicalizer and adapter imports. It never invokes the target. If one transport fails, the other endpoint is reported as not_checked with error code TargetEndpointNotChecked, rather than producing misleading one-sided import evidence.

An arbitrary executable can instead be a first-class target:

[cases.reference]
command = ["./bin/legacy-adapter"]

[cases.candidate]
command = ["./bin/new-adapter", "--mode", "compatibility"]

Command targets implement the small versioned process protocol; they may be Python, Rust, Java, C/C++, Fortran or anything else that can read/write JSON and Arrow IPC. See the target protocol.

Inputs and domain generation

A fixture anchors the campaign in realistic structure. A reviewed schema then makes the explored domain explicit: numeric and string bounds, nullability, enums, dates/datetimes, time zones, useful examples, uniqueness, ordering and row relationships.

[cases.schema]
min_rows = 0
max_rows = 50

[[cases.schema.constraints]]
kind = "row_comparison"
left = "start_date"
operator = "le"
right = "end_date"

[[cases.schema.columns]]
name = "status"
dtype = "string"
nullable = false
categories = ["open", "closed"]

For domain objects or cross-object rules that do not fit the compact schema, use project-owned generation code:

[cases.generation]
generator = "tests.generators:portfolios"
max_examples = 500

The factory may return a Hypothesis strategy, preserving shrinking through the ordinary finding pipeline, or a bounded iterable from an existing corpus/generator. Parity deliberately keeps this escape hatch first-class instead of growing parity.toml into a proprietary data language.

Findings and replay

A mismatch creates an isolated directory:

.parity/orders/<timestamp>-<input-hash>/
├── input.arrow
├── input.parquet        # when the schema is representable
├── manifest.json
├── replay.json
└── result.json

Multi-input cases use opaque Arrow filenames bound to their logical names in replay.json. Replay checks artifact hashes, effective configuration and recorded runtime/source identities before running trusted project code:

parity replay .parity/orders/<timestamp>-<input-hash>
parity evidence verify .parity/report.json --json .parity/evidence-status.json

parity replay preserves the finding's semantic status: a successfully reproduced incompatibility is still FAILED and exits 1. Use parity evidence verify when the question is whether report-referenced findings reproduced; that command exits 0 when every one did.

Each finding explains what class of behaviour changed and carries an ms3: mismatch-shape fingerprint. It is a deterministic deduplication/replay key, not a cryptographic signature, source attestation, root-cause claim or bug ID. Exception findings show data-safe reference/candidate outcomes and well-known qualified types plus allow-listed Pydantic error codes/location shapes and NumPy API tokens. Custom identifier-shaped metadata remains opaque; raw messages and witness values remain private. Terminal output prints the complete replay signature.

Parallelism and performance

Run independent cases concurrently:

parity check --jobs 8 --native-threads 1

Results return in configuration order and each case owns separate target sessions and artifact paths. Search and shrinking inside a case remain serial and deterministic. --native-threads caps common BLAS/OpenMP pools in target processes, avoiding jobs × native-thread oversubscription. Parallel fail-fast is rejected because its result would depend on scheduling.

Performance starts only after semantic success. Parity alternates nearby reference/candidate invocations, reports paired median speed and peak-memory ratios with deterministic bootstrap confidence intervals, and fails an enforced gate only when the interval's lower bound exceeds its threshold. Use jobs = 1, enough repeats and a controlled runner for meaningful performance evidence; concurrent cases contend for the same host even when their performance policy is report-only. Point estimates from busy shared hosts are not a defensible release gate.

Managed migrations and local regression testing

For a declared library migration, keep the cases and any wrappers in migrations/. The initializer creates a starter inventory at migrations/migration.toml when it is absent; review that inventory before using it as a completion gate. The resulting workspace prepares separately locked environments and runs every lane:

python -m pip install "parity-check[workspace]"
parity init migrations/parity.toml \
  --reference your_library.api:transform \
  --candidate your_library.api:transform \
  --fixture tests/fixtures/input.parquet
parity migration init --reference "$REFERENCE_PACKAGE_SPEC"
parity migration run

For a branch or worktree comparison, create the same migrations/parity.toml contract first, then point both sides at existing checkouts:

parity migration init \
  --reference-path ../main-worktree \
  --candidate ../feature-worktree
parity migration run

The reference is either an exact released requirement or a local checkout; the candidate is always a local checkout. migration init creates the active workspace and, when needed, a starter ledger that maps every configured case to core-regression; review that inventory before relying on it. migration run resolves separate hash-pinned dependency locks, prepares a reference/candidate environment for each declared lane, verifies the installed package identities and writes a data-safe JSON report for each lane. Local/local runs additionally verify editable import origins, record path-free Git HEAD/dirty/source-digest provenance and fail if either checkout changes during the run. Findings retain those identities, so replay rejects same-version source drift before target invocation. Target environments need the package, its adapter dependencies and PyArrow, not Parity; use --reference-python and --candidate-python for different Python 3.8+ runtimes. Parity never creates, switches or modifies worktrees. Set explicit reference.python and candidate.python paths when another system provisions the environments. See the user guide.

Migrations are one active adjacent pair. Keep reusable controls/core cases, replace transition-specific cases, and advance after promoting the candidate:

parity migration advance --reference "$NEXT_REFERENCE_PACKAGE_SPEC"
parity migration run

Historical transitions do not accumulate in the completion gate. See the user guide and migration completion protocol.

Other supported uses

The same contracts currently support dependency-version checks, large refactors, local versus local Git comparisons, release regressions, alternative backends, Python rewrites and external command implementations. Cross-language verification is available through the target protocol, not a language-specific plugin. Capturing files, databases, HTTP calls and other effects is a future contract extension; today those effects should be projected into an explicit return value by a reviewed adapter. See use cases and boundaries.

Python and pytest

from parity import check, verify

suite = check("parity.toml", cases={"orders"})
assert suite.passed

suite = verify(
    reference_orders,
    candidate_orders,
    fixture=sample,
    reference_adapter="pandas",
    candidate_adapter="polars",
    artifact_dir=".parity/live-orders",
)
assert suite.passed

The pytest plugin exposes the same assertion as a fixture:

def test_orders_migration(parity):
    parity.check("parity.toml", cases={"orders"})

Python APIs return typed result models and never terminate the process. The CLI adds the stable 0/1/2 exit contract.

GitHub Actions

permissions:
  contents: read

steps:
  - uses: actions/checkout@v4
  - uses: leighshepperson/parity@v0
    with:
      config: parity.toml
      upload-artifact: "false"

Reports are data-safe projections, but replay artifacts contain input values. Upload them only under an appropriate access and retention policy. Pin a reviewed full commit SHA when the Action revision itself must be immutable. See the Action guide.

Boundaries

Parity answers: did these executable contracts differ anywhere we looked? It does not decide which side expresses business intent, prove correctness outside the input domain, infer a complete public API, justify a weakened policy, migrate code, or securely sandbox hostile targets.

Current comparison understands canonical frames, series, arrays, mappings, sequences, scalars, returns, raises and input mutation. Broader effect capture is intentionally future work. Run unknown code in a container or hardened runner.

Documentation

• User guide
• Use cases and current boundaries
• Configuration reference
• External target protocol
• Architecture and artifact contracts
• Migration completion protocol
• Security and privacy and threat model
• Pytest integration and GitHub Action
• External validation log and fault corpus
• Roadmap, development guide and release notes

Development status

Parity is pre-1.0 and supports its latest minor release. Minor releases may simplify configuration, artifacts and APIs; patch releases preserve their minor line's contracts.

Licensed under the Apache License 2.0.