promptry

Local-first prompt observability that lives in your repo. Version your prompts, write eval suites in YAML or Python, track the cost of every call, edit prompts live, and catch regressions in CI. One pip install, one SQLite file, zero services — your prompts never leave your laptop.

Try the live demo → · Integration guide · Docs

pip install promptry

Quick start

1. See what every call costs — change one import.

from promptry.openai import OpenAI   # was: from openai import OpenAI

client = OpenAI()
client.chat.completions.create(model="gpt-4o-mini", messages=[...])
# ^ recorded: cost, tokens, cached-token split, latency, model, call-site name.
# Streaming, failures, embeddings, and the Responses API are captured too.

promptry dashboard          # cost, drift, and per-call traces at 127.0.0.1:8420

Not on the OpenAI SDK? track_invocation(name, metadata=...) records the same row from any provider's response. On the frontend, wrapOpenAI(new OpenAI()) from promptry-js ships the same events to the same store.

2. Catch regressions before they ship — write an eval.

from promptry import suite, assert_semantic

@suite("rag-regression")           # a plain function; runs via CLI or in CI
def test_quality():
    response = my_pipeline("What is photosynthesis?")
    assert_semantic(response, "Converts light into chemical energy")

promptry init                            # scaffold project + a starter eval
promptry run rag-regression --module evals

When a suite regresses against its baseline, promptry reports what changed — not just that it broke:

Overall score: 0.910 -> 0.720  REGRESSION

Probable cause:
  -> Prompt changed (v3 -> v4)

Prefer no code? promptry new suite --yaml scaffolds a declarative evals.yaml from --case flags. Both paths run through the same CLI.

Install extras

Core stays small on purpose — CLI, prompt registry, deterministic assertions (exact / text / schema / JSON / rouge / levenshtein), drift, cost tracking from the bundled price snapshot, the local dashboard, and the MCP server. Add an extra only when you need it:

pip install 'promptry[semantic]'   # assert_semantic, embedding distance, RAG context, clustering (sentence-transformers + chromadb)
pip install 'promptry[llm]'        # run real model completions / assert_llm judging / live price refresh (litellm + openai + anthropic)
pip install 'promptry[postgres]'   # point the store at a shared PostgreSQL server (alpha) instead of the default SQLite file
pip install 'promptry[full]'       # everything

If you call a feature whose extra isn't installed, promptry raises a clear error telling you the exact pip install to run — it never fails with a bare ModuleNotFoundError.

Features

Dashboard

promptry dashboard

Binds 127.0.0.1 only. Reverse-proxy it for remote access, then lock it with a single shared secret (one API key for the whole team — not per-user tokens):

# generate once; put in a password manager / team vault
export PROMPTRY_AUTH_TOKEN="$(openssl rand -hex 32)"
promptry dashboard --no-open

There is no self-serve “mint me a token” endpoint (that would defeat the lock). Distribute the secret via vault; operators with shell can read the env file. See Dashboard auth.

Prices: the dashboard pulls the published feed (prices.json on main) at startup and every 24h. Opt out with PROMPTRY_PRICES_AUTO_REFRESH=0. Manual: promptry prices --refresh.

Eval health and spend at a glance — drill into evals or cost for detail.

The prompt registry, grouped by module. Click any prompt to inspect versions, diffs, and stats.

A prompt detail view: edit the live $-placeholder template, with variable pills and promotion tags.

Cost, drilled module → prompt → the priciest individual calls.

A single call, broken into fixed template overhead vs the variable payload you fed in.

The playground: render a prompt and compare it across models before promoting to a suite.

The Evals page also builds suites: New suite assembles a YAML suite from manual cases, golden examples, or positive-feedback logs, and Edit reopens any YAML-declared suite in the same builder (Python-defined suites are read-only). A Cache optimization page (also promptry cache on the CLI) ranks prompts by prefix-cache reorder opportunity, diffs near-duplicate prompts for consolidation, and flags redundant/filler wording to shorten — all read from your local prompt registry and invocations ledger, no LLM calls. See the guide.

Why promptry

Three things you won't get elsewhere — together, in one tool:

1. YAML when it's simple, code when it's not. Declarative evals.yaml covers the no-code cases; pytest-style @suite decorators cover the rest — loops, fixtures, debugger breakpoints, IDE autocomplete. Promptfoo makes you generate YAML from Python scripts once your suite grows past a few dozen tests. Here Python is native, not a code-generation round trip.
2. Local by design. One SQLite file. No account, no API key for the framework, no cloud to trust. LangSmith and DeepEval's flagship features push your prompts and outputs to their servers — disqualifying for regulated industries, IP-sensitive work, or anyone who reads their procurement policy.
3. No per-run judge tax. Most assertions are deterministic: semantic similarity, schema, JSON, regex, grounding, tool-use. CI runs cost $0. RAGAS's headline metrics (faithfulness, answer relevancy, context precision) all need judge-model calls — every run costs tokens, adds latency, and drifts when the judge model updates. We treat LLM-as-judge as an opt-in, not a default.

GitHub Action

Run eval suites in CI with one line. On pull requests it posts (or updates) a single comment summarizing the eval: overall score, pass/fail counts, and any regressed tests vs. the previous run. View on Marketplace.

# .github/workflows/eval.yml
name: Eval
on: [push, pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write  # required for PR comments
    steps:
      - uses: actions/checkout@v4
      - uses: promptry/promptry@v1
        with:
          suite: rag-regression
          module: evals
          compare: prod  # optional — compare against baseline

Example PR comment on a regression:

## promptry eval: rag-regression

| | Current | Baseline | Delta |
|---|---|---|---|
| Overall score | 0.891 | 0.910 | -0.019 |
| Passed | 8/10 | 9/10 | -1 |
| Status | REGRESSED | PASS | |

**Regressions:**
- `test_photosynthesis_answer`: semantic 0.89 -> 0.72 (-0.17)
- `test_schema_validation`: passed -> **failed**

_Generated by [promptry](https://github.com/promptry/promptry)_

Subsequent pushes edit the same comment instead of spamming new ones.

MCP server

claude mcp add promptry -- promptry mcp    # Claude Code

Works with Claude Desktop, Cursor, Windsurf, VS Code. Agents don't just run evals — they create them: list_suite_candidates surfaces cases from golden examples or positively-rated production logs, create_eval_suite writes them into a runnable evals.yaml, and run_eval executes it. Everything the agent creates appears (and stays editable) on the dashboard. See full setup.

Documentation

The full guide covers all assertions, cost tracking, model comparison, safety templates, notifications, storage modes, JS client, CLI reference, MCP setup, and config options.

Scope

Promptry is local-first by design. If you need a hosted, always-on observability product for production traffic with team seats and SSO, use LangSmith or Arize — different product category. Promptry runs against one SQLite file on your machine: wire it into CI so a bad prompt change never reaches production, manage your live prompts from the dashboard, and keep a per-call ledger of cost and traces without sending anything to a vendor.

Maturity

promptry is Beta: the core is stable and tested, the surrounding surfaces are usable but still moving. Honest status per component:

Reroute-aware pricing ships in 0.10: when a provider retires a slug and silently serves a pricier model (e.g. xAI's 2026-05-15 grok-4-fast → grok-4.3 at ~6x), the cost engine prices by the model that actually billed, not the requested name. Prices are a bundled snapshot you refresh on your terms — promptry prices lists them, promptry prices --refresh pulls a static published feed (or your local litellm) into ~/.promptry/prices.json, and promptry prices --check flags ledger models with no rate. No hosted service, no daily phone-home: nothing leaves your machine unless you ask it to.

On the roadmap: agent trajectory analysis and LLM-powered root cause.

License

MIT