promptreg 0.1.1

Automated LLM prompt regression testing using local models

Prompt Regression Tracker (promptreg)

promptreg is an enterprise-grade automated CLI tool and testing framework designed to prevent quality regressions in Large Language Model (LLM) prompts. By integrating with local models (via Ollama) and cloud APIs (OpenAI/Anthropic), it systematically executes evaluation suites against pre-configured rubrics. Using an LLM-as-a-Judge architecture, promptreg alerts you immediately if a prompt optimization or model update degrades response quality.

Visual Demos: Explore the generated Interactive HTML Report or review the structured JSON Report Output.

---

Architecture

graph TD
    A[CLI / CI Trigger] --> B[Test Case Loader]
    B --> C[Runner - Ollama / OpenAI / Anthropic]
    C --> D[Judge - LLM-as-a-Judge]
    D --> E[Database Layer - SQLite or Postgres]
    E --> F[Differ - Regression Checker]
    F --> G[Reporter - Terminal / JSON / HTML]

---

Key Features

• LLM-as-a-Judge Evaluation: Scores responses quantitatively (0 to 10) against semantic rubrics utilizing either local or cloud models.
• Dual Database Adapter: Persists execution history, benchmarks, and historical runs to SQLite for local development or PostgreSQL for shared team environments.
• Parallel Execution: Accelerates prompt testing via concurrent thread workers for scalable evaluation.
• Golden Baselines: Establishes high-performance output benchmarks to continually compare future iterations against.
• CI/CD Integration: Supports pre-commit hooks and GitHub Actions with non-zero exit codes upon regression detection.
• Premium Reporting: Generates interactive HTML dashboards, structured JSON files, and comprehensive terminal output.

---

Getting Started

1. Model Provisioning (Local Execution)

To utilize local models, download and install Ollama from ollama.com. Pull the necessary evaluation models before execution:

ollama pull phi3

2. Installation

Initialize a virtual environment and install the package with development dependencies:

python3 -m venv venv
source venv/bin/activate
pip install -e ".[dev]"

---

Writing Test Cases

Define test cases using YAML syntax within the tests/suite/ directory (e.g., tests/suite/summarize.yaml):

id: summarize_basic
description: Single-sentence summary of a news paragraph
model: phi3
prompt: |
  Summarize the following in one sentence:
  "The Federal Reserve raised interest rates by 0.25% on Wednesday,
  citing persistent inflation concerns despite signs of cooling in
  the labour market."
rubric:
  - must_include: ["Federal Reserve", "interest rates"]
  - max_words: 30
  - tone: neutral

---

Command Line Interface (CLI) Reference

1. Evaluation Suite Execution

Evaluate all test cases within a targeted directory, compare outputs to established baselines, and detect regressions:

promptreg run --suite tests/suite --threshold 1.5

Note: Exits with code 1 if any regression exceeds or equals the defined threshold.

Options:

• --suite PATH: Directory containing test files (Default: tests/suite)
• --threshold FLOAT: Score drop threshold to trigger a regression (Default: 1.5)
• --db-url URL: Postgres or SQLite database connection URL (Default: promptreg.db)
• --judge-model MODEL: Model utilized for evaluation logic (e.g. phi3, openai/gpt-4o)
• --concurrency INT: Number of concurrent workers (Default: 4)
• --output-json PATH / --output-html PATH: Export report destinations

2. Baseline Promotion

Promote the scores of a specific run to serve as the benchmark golden baseline:

promptreg baseline <run_id>

3. Historical Data Review

Display a tabular history of preceding evaluation runs:

promptreg history [--test-id <test_id>] [--last 10]

4. Differential Comparison

Perform a side-by-side comparison of the scores of two specific runs:

promptreg diff <run_a> <run_b>

---

CI/CD and Pre-commit Integration

Automate testing pipelines by adding promptreg to your .pre-commit-config.yaml:

repos:
  - repo: local
    hooks:
      - id: prompt-regression
        name: Prompt regression check
        entry: venv/bin/promptreg run
        language: system
        pass_filenames: false
        always_run: true

Install the hook locally:

pre-commit install

---

Team Operations & Security

For enterprise deployments utilizing a shared PostgreSQL database, promptreg includes built-in authentication and auditing systems to secure write operations.

1. Database Configuration

Configure a centralized PostgreSQL database via the --db-url CLI option or the PROMPTREG_DATABASE_URL environment variable:

export PROMPTREG_DATABASE_URL="postgresql://postgres:password@shared-db-host:5432/promptreg"

Alternatively, specify it within promptreg.yaml:

db_url: "postgresql://postgres:password@shared-db-host:5432/promptreg"

2. Token-Based Authentication

To restrict unauthorized modifications to baseline benchmarks or test results, enable token authentication in promptreg.yaml:

auth:
  enabled: true
  admin_token: "your-team-secret-admin-token"

Executing Authenticated Commands: Write operations (run and baseline promotion) mandate token validation when authentication is active. Provide the token via the --auth-token flag:

promptreg run --auth-token "your-team-secret-admin-token"

Or via the PROMPTREG_AUTH_TOKEN environment variable:

export PROMPTREG_AUTH_TOKEN="your-team-secret-admin-token"
promptreg baseline 42

3. User Auditing

Maintain auditability by tracking the identity of the developer or CI pipeline executing operations. promptreg tracks:

• created_by for each execution run
• set_by for baseline promotions

Specify the identity using the --username CLI option or the PROMPTREG_USERNAME environment variable:

promptreg run --username "alice"
# or
export PROMPTREG_USERNAME="bob"
promptreg baseline 12

If omitted, promptreg automatically falls back to the active operating system login name.

---

Pytest Integration

promptreg exposes a native pytest plugin to seamlessly execute regression evaluations within standard Python test suites.

Installation automatically registers the plugin. The prompt_regression fixture can be immediately injected into test cases.

Implementation Example:

# tests/test_prompts.py

def test_summarization_prompt_quality(prompt_regression):
    prompt = "Summarize the article..."
    model_output = call_your_llm_function(prompt)
    
    prompt_regression.assert_no_regression(
        test_id="summarize_basic",
        prompt=prompt,
        output=model_output,
        rubric=[
            {"must_include": ["Federal Reserve"]},
            {"max_words": 30},
            "tone should be objective"
        ],
        model="gpt-4o"
    )

Executing pytest will run the evaluation, log historical scoring data, compare the results against the designated baseline, and force test failure upon quality regression detection.