promptry

Local-first prompt observability that lives in your repo. Version your prompts, write eval suites in Python, track the cost of every call, edit prompts live, and catch regressions in CI. One pip install, one SQLite file, zero services — your prompts never leave your laptop.

Try the live demo → · Integration guide · Docs

from promptry import track, suite, assert_semantic

# track() content-hashes your prompt and stores a new version if it changed
prompt = track(system_prompt, "rag-qa")
response = llm.chat(system=prompt, ...)

# suites are regular Python functions. run them via CLI or in CI.
@suite("rag-regression")
def test_quality():
    response = my_pipeline("What is photosynthesis?")
    assert_semantic(response, "Converts light into chemical energy")

When a suite regresses against its baseline, promptry reports what changed:

Overall score: 0.910 -> 0.720  REGRESSION

Probable cause:
  -> Prompt changed (v3 -> v4)

Install

pip install promptry

Quick start

promptry init                              # scaffold project + starter eval
promptry run smoke-test --module evals     # run it

PASS test_basic_quality (142ms)
  semantic (0.891) ok

Overall: PASS  score: 0.891

Prefer YAML, or don't want to hand-write a suite? promptry new suite scaffolds one for you, YAML or Python:

promptry new suite --name my-suite --yaml \
  --model gpt-4o-mini --prompt "Answer: {input}" \
  --case "What is 2+2?::contains::4"
promptry run my-suite --module evals.yaml  # run it

Features

Dashboard

promptry dashboard

Eval health and spend at a glance — drill into evals or cost for detail.

The prompt registry, grouped by module. Click any prompt to inspect versions, diffs, and stats.

A prompt detail view: edit the live $-placeholder template, with variable pills and promotion tags.

Cost, drilled module → prompt → the priciest individual calls.

A single call, broken into fixed template overhead vs the variable payload you fed in.

The playground: render a prompt and compare it across models before promoting to a suite.

Why promptry

Three things you won't get elsewhere — together, in one tool:

1. Code, not YAML. Suites are pytest-style decorators. Loops, fixtures, debugger breakpoints, IDE autocomplete. Promptfoo makes you generate YAML from Python scripts once your suite grows past a few dozen tests. Just skip the round trip.
2. Local by design. One SQLite file. No account, no API key for the framework, no cloud to trust. LangSmith and DeepEval's flagship features push your prompts and outputs to their servers — disqualifying for regulated industries, IP-sensitive work, or anyone who reads their procurement policy.
3. No per-run judge tax. Most assertions are deterministic: semantic similarity, schema, JSON, regex, grounding, tool-use. CI runs cost $0. RAGAS's headline metrics (faithfulness, answer relevancy, context precision) all need judge-model calls — every run costs tokens, adds latency, and drifts when the judge model updates. We treat LLM-as-judge as an opt-in, not a default.

GitHub Action

Run eval suites in CI with one line. On pull requests it posts (or updates) a single comment summarizing the eval: overall score, pass/fail counts, and any regressed tests vs. the previous run. View on Marketplace.

# .github/workflows/eval.yml
name: Eval
on: [push, pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write  # required for PR comments
    steps:
      - uses: actions/checkout@v4
      - uses: bihanikeshav/promptry@v0.10.0
        with:
          suite: rag-regression
          module: evals
          compare: prod  # optional — compare against baseline

Example PR comment on a regression:

## promptry eval: rag-regression

| | Current | Baseline | Delta |
|---|---|---|---|
| Overall score | 0.891 | 0.910 | -0.019 |
| Passed | 8/10 | 9/10 | -1 |
| Status | REGRESSED | PASS | |

**Regressions:**
- `test_photosynthesis_answer`: semantic 0.89 -> 0.72 (-0.17)
- `test_schema_validation`: passed -> **failed**

_Generated by [promptry](https://github.com/bihanikeshav/promptry)_

Subsequent pushes edit the same comment instead of spamming new ones.

MCP server

claude mcp add promptry -- promptry mcp    # Claude Code

Works with Claude Desktop, Cursor, Windsurf, VS Code. See full setup.

Documentation

The full guide covers all assertions, cost tracking, model comparison, safety templates, notifications, storage modes, JS client, CLI reference, MCP setup, and config options.

Scope

Promptry is local-first by design. If you need a hosted, always-on observability product for production traffic with team seats and SSO, use LangSmith or Arize — different product category. Promptry runs against one SQLite file on your machine: wire it into CI so a bad prompt change never reaches production, manage your live prompts from the dashboard, and keep a per-call ledger of cost and traces without sending anything to a vendor.

Shipped: everything in the feature table above, across Python + JS + CLI + dashboard + MCP + GitHub Action — including the live prompt CMS with environment promotion, the per-call invocations ledger with opt-in request/response capture and feedback ingest, cost-by-module drill-down with budgets, and regression bisect.

Reroute-aware pricing ships in 0.10: when a provider retires a slug and silently serves a pricier model (e.g. xAI's 2026-05-15 grok-4-fast → grok-4.3 at ~6x), the cost engine prices by the model that actually billed, not the requested name. Prices are a bundled snapshot you refresh on your terms — promptry prices lists them, promptry prices --refresh pulls a static published feed (or your local litellm) into ~/.promptry/prices.json, and promptry prices --check flags ledger models with no rate. No hosted service, no daily phone-home: nothing leaves your machine unless you ask it to.

On the roadmap: agent trajectory analysis and LLM-powered root cause.

License

MIT