parity-ai 0.1.16

Automatically generate evals for every AI change

Parity

Parity analyzes behavior-defining AI changes in pull requests, discovers the most relevant existing eval target, validates the real coverage gaps, and proposes native eval additions that fit the target suite.

Parity is not an eval runner. It does not create or mutate hosted evaluator infrastructure. It reuses the eval system you already have.

What Parity Does

For each PR that changes prompts, instructions, guardrails, judges, validators, or similar behavior-defining assets, Parity:

1. Detects the behavioral change.
2. Resolves the best matching eval target and method.
3. Validates which gaps are actually uncovered.
4. Synthesizes native eval additions for that target.
5. Writes only native_ready evals after explicit approval.

Support

Path	Status	Notes
Promptfoo	Strong	Best fully native path. Assertions are row-local and writeback is straightforward.
LangSmith	Strong	Strong dataset discovery and writeback. Evaluator reuse is supported; evaluator mutation is out of scope.
Braintrust	Supported with limitations	Writeback works. Target discovery is weaker and evaluator recovery depends more on repo assets.
Arize Phoenix	Supported with limitations	Dataset read/write works. Evaluator discovery is weaker than Promptfoo and LangSmith.
Bootstrap mode	Built in	If no safe target is found, Parity proposes starter evals and abstains from unsafe writeback.

More detail: docs/platforms.md

Public Commands

These are the commands most users need:

• parity init — scaffold parity.yaml, the GitHub Actions workflow, and context/ stubs
• parity doctor — verify your setup and environment
• parity run-stage 1 — detect behavioral artifact changes in a PR
• parity run-stage 2 — analyze coverage gaps against existing evals
• parity run-stage 3 — synthesize native eval proposals
• parity write-evals — write approved evals to your platform after merge
• parity setup-mcp — generate an MCP server config from parity.yaml (for local agent tooling)

Internal CI commands (post-comment, resolve-run-id, etc.) are used by the generated workflow and are not intended to be called directly.

Quick Start

pip install parity-ai
parity init

Then:

1. Fill in the generated context/ files.
2. Add GitHub secrets: ANTHROPIC_API_KEY, OPENAI_API_KEY, and any platform keys you use.
3. Commit parity.yaml, .github/workflows/parity.yml, and context/.
4. Open a PR that changes AI behavior.
5. Add the fixed approval label parity:approve before merging if you want Parity to write approved evals back after merge.

Docs

• Configuration
• Architecture
• Platform support
• Example quickstart
• Maintainer guide

License

MIT