promptry 0.6.0

Regression protection for LLM pipelines

promptry

Sentry for prompts. Local-first regression testing for LLM pipelines — track prompt versions, run eval suites, catch regressions before your users do.

from promptry import track

prompt = track(system_prompt, "rag-qa")
# promptry automatically versions prompts, runs evals, and flags regressions

When something regresses, promptry tells you what changed, when, and whether it caused it:

Overall score: 0.910 -> 0.720  REGRESSION

Probable cause:
  -> Prompt changed (v3 -> v4)

Install

pip install promptry                       # core
pip install promptry[semantic]             # + semantic assertions (sentence-transformers)
pip install promptry[dashboard]            # + web dashboard
pip install promptry[semantic,dashboard]   # everything

Quick start

promptry init                              # scaffold project + starter eval
promptry run smoke-test --module evals     # run it

PASS test_basic_quality (142ms)
  semantic (0.891) ok

Overall: PASS  score: 0.891

Features

Feature	What it does
Prompt versioning	Content-hashed, automatic dedup
Eval suites	Semantic, schema, LLM-as-judge, JSON, regex, grounding assertions
Regression detection	Compare against baselines, get root cause hints
Drift detection	Catch slow quality degradation over time
Model comparison	Statistical comparison against historical baseline (not just snapshots)
Cost tracking	Token usage and cost per prompt, aggregated reports
Safety templates	25+ built-in jailbreak / injection / PII tests
MCP server	Expose everything as tools for Claude, Cursor, VS Code, etc.
Dashboard	Web UI for eval history, prompt diffs, model comparison, cost
JS/TS client	Ship prompt events from frontend/Node apps

Dashboard

pip install promptry[dashboard]
promptry dashboard

How it differs

	Promptfoo	LangSmith	RAGAS	promptry
Approach	External YAML + CLI	Hosted platform	Metrics library	Python-native, instruments your code
Production tracking	No	Yes	No	Yes (track())
Drift detection	No	No	No	Yes
Root cause hints	No	No	No	Yes
Model comparison	Snapshot (A vs B now)	No	No	Historical (statistical)
Local-first	Yes	No (SaaS)	Yes	Yes (SQLite)
Cost	Free	Paid	Free	Free

GitHub Action

Run eval suites in CI with one line. View on Marketplace.

# .github/workflows/eval.yml
name: Eval
on: [push]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: bihanikeshav/promptry@v0.5.0
        with:
          suite: rag-regression
          module: evals
          compare: prod  # optional — compare against baseline

Input	Required	Default	Description
suite	Yes		Eval suite name
module	Yes		Python module containing the suite
compare	No		Baseline tag to compare against
python-version	No	3.12	Python version
extras	No	semantic	pip extras to install

MCP server

claude mcp add promptry -- promptry mcp    # Claude Code

Works with Claude Desktop, Cursor, Windsurf, VS Code. See full setup.

Documentation

The full guide covers all assertions, cost tracking, model comparison, safety templates, notifications, storage modes, JS client, CLI reference, MCP setup, and config options.

Known limitations

• No auto-instrumentation. You add track() calls manually. Explicit > magic.
• Local-first. No hosted multi-user UI. For that, look at LangSmith or Arize.
• Early-stage. v0.6 — API is stable but the project is young. Issues welcome.

License

MIT