PromptDrift

CI regression testing for LLM prompts.

Catch AI behavior regressions before they reach production.

PromptDrift turns the behavior you need from a prompt into explicit, reviewable contracts. When a prompt changes, it runs those contracts in CI and tells you exactly what broke — without treating every wording change as a regression.

prompt change  →  PromptDrift  →  behavioral contracts  →  PR check
                                                        ↳ PASS · WARN · FAIL

Why PromptDrift?

git diff tells you what changed in the prompt text. PromptDrift tells you whether the AI behavior changed in a way that matters.

Quick Start

pip install promptdrift-ci
promptdrift init
promptdrift test

init creates a fully local example using the deterministic mock provider. No API key needed — validate the workflow first, then connect OpenAI or Ollama.

Capture approved behavior:

promptdrift baseline
git add promptdrift.baseline.json

Define Behavioral Contracts

promptdrift.yaml stays small, readable, and version-controlled:

version: 1

provider:
  type: openai
  model: gpt-4.1-mini
  api_key_env: OPENAI_API_KEY

tests:
  - id: refund_request
    prompt: prompts/support.txt
    variables:
      policy: Refunds are available within 30 days.
      question: Can I get a refund for my order?
    assertions:
      - type: contains
        value: 30 days
      - type: not_contains
        value: guaranteed
      - type: max_length
        value: 600

❌ FAIL refund_request

Assertion: contains
Expected: 30 days
Actual:   You may be eligible for a refund.
Reason:   Required text does not appear in the output.

Contracts, Not Cosmetic Diffs

Built-in Assertions

exact_match · contains · not_contains · regex · not_regex · json_valid · json_schema · min_length · max_length · max_tokens · latency_ms · cost_usd

Each assertion supports severity: warn for non-blocking warnings.

Run in CI

name: PromptDrift

on:
  pull_request:
    paths:
      - "prompts/**"
      - "promptdrift.yaml"

permissions:
  contents: read

jobs:
  promptdrift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: tanveer-arch/promptdrift/action@v1
        with:
          config: promptdrift.yaml
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

The Action writes a Step Summary, uploads a JSON report, and fails only after reports are available. For PR comments, add comment: 'true' and grant pull-requests: write.

Security: Use pull_request for untrusted forks. Never expose provider secrets with pull_request_target while executing code from a pull request.

Commands

All commands support --json for machine-readable output. Exit codes: 0 success, 1 behavioral failure, 2 config/usage error, 3 provider/runtime error.

Architecture

promptdrift.yaml ─→ Config ─→ Template ─→ Provider ─→ Evaluator ─→ Report
                     Loader    Renderer    Adapter     Engine       Formatter
                                                        │
                                                  Baseline ←── Git
                                                  Comparison

• Providers are isolated adapters (OpenAI, Ollama, Mock) — adding one is a single file
• Assertions are pure, deterministic functions — same input always produces the same result
• Baselines store hashes and metrics, never raw outputs or secrets
• Reports output to terminal, JSON, HTML, or GitHub PR comments

Privacy by Default

• No telemetry, accounts, or hosted service
• Prompt content goes only to the provider you configure
• API keys are environment variables — never config values
• Baselines contain hashes and metrics, not raw outputs
• Local SQLite history suppresses prompts and outputs by default

Documentation

Roadmap

• Semantic similarity assertions (embedding-based)
• LLM-as-judge evaluator
• Multi-provider comparison (same prompt, different models)
• Parallel test execution
• PyPI trusted publisher release
• Cost tracking dashboard

License

Released under the MIT License.