PromptDrift

Git-native AI behavior change detection.

Change a prompt. PromptDrift automatically shows what changed in AI behavior, what broke, what improved, and whether the change is safe to merge.

PromptDrift turns your real application traffic and prompt changes into automatic regression coverage with Impact Radius analysis, without requiring manual authoring of dozens of YAML tests.

application / prompt / traffic
          ↓
      discovery
          ↓
   scenario library
          ↓
   baseline behavior
          ↓
 prompt change detected
          ↓
 replay + evaluation
          ↓
 regression analysis
          ↓
 GitHub PR / CLI report

3-Minute Quick Start

pip install promptdrift-ci
promptdrift init

PromptDrift scans your repository context and sets up starter files.

1. Log example interactions from Production

Use the PromptDrift CLI or the built-in async/sync CaptureRecorder SDK to capture your production traffic.

from promptdrift.capture.recorder import CaptureRecorder, CaptureConfig

recorder = CaptureRecorder(CaptureConfig(enabled=True))
# Safely wraps your OpenAI client, capturing representative traffic with a rolling retention cap (default 10,000 interactions).
# Redacts raw outputs and uses deterministic sampling to avoid duplicating identical edge cases.

2. Turn captures into candidate regression scenarios

promptdrift learn

3. Review and promote scenarios to the active suite

promptdrift scenarios
promptdrift suggest cancellation_how_do_i_cancel
promptdrift promote cancellation_how_do_i_cancel

4. Create your baseline

promptdrift baseline

5. Run Git-aware check

promptdrift check

Impact Radius & Behavioral Change Report

When you modify your prompt templates and run promptdrift check:

PromptDrift Impact Report
Behavior changed in 8 / 43 scenarios.

  ✓ 5 improved
  ⚪ 35 unchanged
  💬 17 changed but valid
  ✗ 3 regressed

Impact radius: 18.6%
Latency change: +7.2%
Cost change:    +11.3%

Affected Categories:
  Refunds          5 affected
  Cancellation     2 affected
  Escalation       1 affected

Regressions:
  ✗ refund_request
    Required policy condition failed: 30 days window missing.
  ✗ cancellation_001
    JSON schema failed: 'order_id' is a required property.

Define Explicit Behavioral Contracts (Advanced Mode)

You can always define explicit behavioral contracts in promptdrift.yaml:

version: 2

project:
  name: support-bot

provider:
  type: openai
  model: gpt-4.1-mini
  api_key_env: OPENAI_API_KEY

tests:
  - id: refund_request
    prompt: prompts/support.txt
    variables:
      policy: Refunds are available within 30 days.
      question: Can I get a refund for my order?
    assertions:
      - type: contains
        value: 30 days
      - type: not_contains
        value: guaranteed
      - type: max_length
        value: 600

Built-in Deterministic Assertions

exact_match · contains · not_contains · regex · not_regex · json_valid · json_schema · min_length · max_length · max_tokens · latency_ms · cost_usd

GitHub Action & PR Workflow

name: PromptDrift Check

on:
  pull_request:
    paths:
      - "prompts/**"
      - "promptdrift.yaml"

permissions:
  contents: read
  pull-requests: write

jobs:
  promptdrift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: tanveer-arch/promptdrift/action@main  # Change to @v1 upon stable release
        with:
          config: promptdrift.yaml
          comment: 'true'
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

Complete CLI Command Suite

Privacy by Default

• No telemetry, tracking, or cloud accounts
• Prompts are evaluated against the provider you configure
• Captured traffic remains local in SQLite and is excluded from Git by default
• Baseline files store hashes and metrics, never raw user inputs or prompt secrets
• Clean wipe available at any time via promptdrift purge

License

Released under the MIT License.