promptreg 0.1.0

Automated LLM prompt regression testing using local models

🎯 Prompt Regression Tracker (promptreg)

promptreg is an enterprise-grade automated CLI tool designed to prevent quality regressions in LLM prompts. By integrating with local models (via Ollama) and cloud APIs (OpenAI/Anthropic), it systematically runs evaluation suites against pre-configured rubrics scored by a judge model (e.g., phi3 or custom choices), alerting you if a prompt optimization or model update degrades generation quality.

📊 Visual Demos: Explore the generated Interactive HTML Report or review the structured JSON Report Output.

---

🏗️ Architecture

graph TD
    A[CLI / CI Trigger] --> B[Test Case Loader]
    B --> C[Runner - Ollama / OpenAI / Anthropic]
    C --> D[Judge - LLM-as-a-Judge]
    D --> E[Database Layer - SQLite or Postgres]
    E --> F[Differ - Regression Checker]
    F --> G[Reporter - Terminal / JSON / HTML]

---

✨ Features

• LLM-as-a-Judge Evaluation: Scores responses from 0 to 10 against semantic rubrics using local or cloud models.
• Dual Database Adapter: Saves execution history, baselines, and runs to SQLite (local) or PostgreSQL (shared team database).
• Parallel Execution: Speeds up prompt testing via concurrent thread workers.
• Golden Baselines: Lock in your high-performance outputs to compare future changes against.
• CI/CD Integration: Returns non-zero exit codes on regressions for pre-commit hooks and Github workflows.
• Premium Reporting: Generates interactive HTML dashboards, structured JSON files, and beautiful terminal reports.

---

🚀 Getting Started

1. Install & Pull Ollama (Local LLM Support)

Download Ollama from ollama.com and pull your evaluation models:

ollama pull phi3

2. Install the Package

python3 -m venv venv
source venv/bin/activate
pip install -e ".[dev]"

---

📝 Writing Test Cases

Create YAML test files in tests/suite/ (e.g., tests/suite/summarize.yaml):

id: summarize_basic
description: Single-sentence summary of a news paragraph
model: phi3
prompt: |
  Summarize the following in one sentence:
  "The Federal Reserve raised interest rates by 0.25% on Wednesday,
  citing persistent inflation concerns despite signs of cooling in
  the labour market."
rubric:
  - must_include: ["Federal Reserve", "interest rates"]
  - max_words: 30
  - tone: neutral

---

🛠️ CLI Reference

1. Run the Evaluation Suite

Evaluate all test cases in the directory, compare outputs to baselines, and check for regressions:

promptreg run --suite tests/suite --threshold 1.5

💡 Exits with code 1 if any regression >= threshold is detected.

Command Options:

• --suite PATH: Directory containing test files (Default: tests/suite)
• --threshold FLOAT: Score drop threshold to trigger a regression (Default: 1.5)
• --db-url URL: Postgres or SQLite database connection URL (Default: promptreg.db)
• --judge-model MODEL: Model to use for judging (e.g. phi3, openai/gpt-4o)
• --concurrency INT: Number of concurrent workers (Default: 4)
• --output-json PATH / --output-html PATH: Export report paths

2. Promote Run to baseline

Mark the scores of a run as the benchmark golden baselines:

promptreg baseline <run_id>

3. Review Historical Scores

Display a tabular history of previous evaluation runs:

promptreg history [--test-id <test_id>] [--last 10]

4. Diff Two Runs Side-by-Side

Compare the scores of two specific runs:

promptreg diff <run_a> <run_b>

---

🤝 Git Integration (Pre-commit Hook)

Automate tests on commit by adding this to .pre-commit-config.yaml:

repos:
  - repo: local
    hooks:
      - id: prompt-regression
        name: Prompt regression check
        entry: venv/bin/promptreg run
        language: system
        pass_filenames: false
        always_run: true

Install the hook:

pre-commit install

---

🔒 Centralized Team Operations & Security

When deploying promptreg in a team environment with a shared PostgreSQL database, you can secure write operations and track attribution using the built-in authentication and auditing system.

1. Database Configuration

Configure promptreg to use a central PostgreSQL database. This can be defined via CLI option --db-url or the PROMPTREG_DATABASE_URL environment variable:

export PROMPTREG_DATABASE_URL="postgresql://postgres:password@shared-db-host:5432/promptreg"

Alternatively, configure it inside promptreg.yaml:

db_url: "postgresql://postgres:password@shared-db-host:5432/promptreg"

2. Token-Based Authentication

To prevent unauthorized users from overriding baseline benchmarks or committing runs to the shared database, enable token authentication in promptreg.yaml:

auth:
  enabled: true
  admin_token: "your-team-secret-admin-token"

Or configure the expected token via the PROMPTREG_ADMIN_TOKEN environment variable on your database server.

Running Commands with Auth:

Once authentication is active, write operations (run and baseline promotion) require providing the token. Specify it via the --auth-token flag:

promptreg run --auth-token "your-team-secret-admin-token"

Or by setting the PROMPTREG_AUTH_TOKEN environment variable in your developer terminal or CI/CD workflow:

export PROMPTREG_AUTH_TOKEN="your-team-secret-admin-token"
promptreg baseline 42

3. Username Auditing

To identify which developer or pipeline triggered an evaluation or promoted a baseline, you can configure username attribution. promptreg tracks:

• created_by for each run
• set_by for each baseline

Specify your username via the --username CLI option or the PROMPTREG_USERNAME environment variable:

promptreg run --username "alice"
# or
export PROMPTREG_USERNAME="bob"
promptreg baseline 12

If not provided, promptreg automatically falls back to your OS login name.

---

🧪 Pytest Integration

promptreg exposes a built-in pytest plugin so you can run regression evaluations directly within your existing Python unit/integration tests.

When you install promptreg, the plugin is automatically registered. You can immediately inject the prompt_regression fixture.

Example:

# tests/test_prompts.py

def test_summarization_prompt_quality(prompt_regression):
    prompt = "Summarize the article..."
    model_output = call_your_llm_function(prompt)
    
    prompt_regression.assert_no_regression(
        test_id="summarize_basic",
        prompt=prompt,
        output=model_output,
        rubric=[
            {"must_include": ["Federal Reserve"]},
            {"max_words": 30},
            "tone should be objective"
        ],
        model="gpt-4o"
    )

Running pytest will execute the evaluation, track the score history in the promptreg database, compare results against baseline, and trigger failures if a quality regression occurs.