promptops
I changed one word in a system prompt and broke production. I had no diff, no rollback, no way to know which version was running. So I built Git for prompts.
promptops is a framework-agnostic, Git-native CLI that version-controls LLM prompts. Semantic diff. Thompson Sampling A/B routing. LLM-as-judge regression eval. MCP server. Drop-in GitHub Actions gate.
pip install "git+https://github.com/vineetha00/promptops@v0.3.0"
Don't
pip install promptopsfrom PyPI. That name belongs to an unrelated project, which installs its ownpromptopscommand and Python package. This project's package ispromptops-git, its command ispromptops-git, and it imports aspromptops_git. It isn't on PyPI yet, so install it from a GitHub release tag as shown above. Its data stays in your repo's.promptops/directory.
60-second quickstart
promptops-git init
promptops-git commit payment-agent "You are a payment assistant. Be precise." --tag v1
promptops-git commit payment-agent "You are a helpful payment assistant. Be precise and friendly." --tag v2
promptops-git diff payment-agent v1..v2
promptops-git ab-test payment-agent v1 v2 --metric quality
promptops-git rollback payment-agent --to v1
That's six commands from zero to versioned, A/B-tested, instantly-rollback-able prompts.
Architecture
┌──────────────────────────────────────────────────────────┐
│ promptops │
│ │
CLI ──┤ store.py diff.py ab.py eval.py │
│ ─────── ─────── ───── ─────── │
│ .promptops/ sentence- Thompson Claude │
│ registry.json transformers Sampling as judge │
│ prompts/*.txt cosine sim Beta(α,β) 1-5 score │
│ │ │ │ │ │
│ └──────┬───────┴────────────┴──────────────┘ │
│ │ │
│ mcp_server.py ◄───── stdio ─────► Claude Code │
│ │
│ prompt-regression.yml ◄────► GitHub Actions PR │
└──────────────────────────────────────────────────────────┘
Each prompt is a plain .txt file under .promptops/prompts/. Metadata — versions, scores, Beta posteriors, traffic — lives in .promptops/registry.json. Git already knows how to diff, branch, and merge that.
Commands
| Command | What it does |
|---|---|
promptops-git init |
Create .promptops/ in the current directory. |
promptops-git commit NAME TEXT --tag v1 |
Commit a new version of a prompt. |
promptops-git log NAME |
Version history with scores, traffic, and active marker. |
promptops-git diff NAME v1..v2 |
Semantic diff: cosine similarity + sentence-level changes. |
promptops-git ab-test NAME v1 v2 --metric quality |
Thompson Sampling routing; declares a winner at p > 0.95. |
promptops-git rollback NAME --to v1 |
Instant rollback to a prior version. |
promptops-git status |
All tracked prompts, active versions, total traffic. |
promptops-git eval --compare main..branch --fail-on-regression |
Run changed prompts over their eval set, judge the outputs against main; CI gate. |
promptops-git serve --port 3333 |
Start the MCP server (stdio). |
How it works
Semantic diff — both prompt versions are embedded with sentence-transformers/all-MiniLM-L6-v2. Cosine similarity gives an overall semantic distance. A sentence-level diff highlights what actually changed, colored by rich.
Thompson Sampling A/B — every version maintains a Beta(α, β) posterior over its success rate. A routing call samples once from each posterior and picks the highest. Wins bump α, losses bump β. A winner is declared when a Monte-Carlo estimate of P(version is best) exceeds 0.95 — i.e., statistically significant before declaring.
Output-based regression eval — each prompt has a fixed eval set of test inputs. Both the baseline and the candidate prompt are run on every case (generator: claude-opus-5-5, override with PROMPTOPS_GEN_MODEL to match production), and claude-sonnet-5-5 (PROMPTOPS_JUDGE_MODEL) scores each output 1-5 on quality, relevance, and safety, using a rubric with level anchors and few-shot calibration examples. Judge output is schema-constrained and validated. A version's score is the mean over the eval set. promptops-git eval --prompt NAME scores versions locally and feeds the A/B posteriors.
Fails closed — if generation or judging fails (no credentials, API error, malformed judge output) or the baseline can't be read, eval exits 2 with no scores. There is no fallback scorer. Exit codes: 0 pass, 1 regression (with --fail-on-regression), 2 eval could not run.
MCP server — stdio transport exposing three tools to any MCP client (Claude Code included):
get_active_version(prompt_name)— current winner + textlist_prompts()— all tracked prompts and versionsget_metrics(prompt_name)— A/B scores, win rates, traffic splits
vs. the alternatives
| promptops | Langfuse | LangSmith | Weights & Biases | |
|---|---|---|---|---|
| Pip-installable, no account | ✅ | ❌ | ❌ | ❌ |
| Git-native storage (diffable, branchable) | ✅ | ❌ | ❌ | ❌ |
| Semantic diff between versions | ✅ | ❌ | partial | ❌ |
| Thompson Sampling A/B router | ✅ | ❌ | ❌ | ❌ |
| LLM-as-judge eval | ✅ | ✅ | ✅ | partial |
| MCP server out of the box | ✅ | ❌ | ❌ | ❌ |
| Framework-agnostic | ✅ | ✅ | weighted to LangChain | ✅ |
| Drop-in GitHub Actions gate | ✅ | ❌ | ❌ | ❌ |
| Local-first, no telemetry | ✅ | self-host | ❌ | ❌ |
MCP integration with Claude Code
Add this to your Claude Code MCP config:
{
"mcpServers": {
"promptops": {
"command": "promptops-git",
"args": ["serve"]
}
}
}
Now Claude Code can list_prompts, fetch the active version, and read live A/B metrics directly from your repo.
GitHub Actions: drop-in regression gate
Copy .github/workflows/prompt-regression.yml from this repo. Add ANTHROPIC_API_KEY to your repo secrets. Every PR touching prompts now runs:
promptops-git eval --compare origin/main..HEAD --fail-on-regression
The baseline is the active version as committed on main, read with git show, and the eval set is also read from main. Nothing in the PR's checkout can change what it is measured against. A PR that edits an eval set gets a note, and the edit applies only after it merges. Candidates are the versions the PR adds or edits, plus its active version if that differs from main. The report lists which models actually served the run, plus token usage. If the judge or generator model changes partway through a run, the run fails with exit 2.
When does a change count as a regression?
Prompt regressions are sparse and severe, not uniform and mild. A bad prompt edit rarely makes every answer slightly worse. More often it breaks a few inputs completely and leaves the rest alone. In testing, a prompt that told the assistant to ask for card details made it refuse 2 of 6 ordinary requests. Those cases fell from about 4.7 to 1.0, while the other four cases were unaffected. A mean-based test can't see that shape, because the uneven drops look like noise to it. That run had a -19% mean drop at p = 0.08, and the mean rule alone passed it. So the gate has three triggers, and any one of them blocks the merge:
| Trigger | Fires when | Catches |
|---|---|---|
| mean | mean score drops more than 10% (--threshold) and a one-sided paired t-test across cases gives p < 0.05 |
broad, uniform degradation |
| collapse | one case drops by ≥ 2.0 points (1–5 scale) | a prompt that breaks specific inputs |
| floor | one case scores 1 on any dimension (below 2.0) where the baseline did not | refusals, unusable answers, and unsafe answers, whatever the baseline was |
The floor checks each dimension separately, not the total. A refusal scores quality 1, relevance 1, safety 5, which averages 2.33, so a floor on the total would never catch it. A polished but unsafe answer (5, 5, 1) averages 3.67 and would slip through too. In testing, a refusal on a case whose baseline was only 3.0 dropped too little for the collapse rule. Only the per-dimension floor caught it.
Before collapse or floor fires, the same outputs are judged a second time. The trigger fires only if the second judgment agrees, so one flaky judgment can't fail the gate. That costs nothing unless a trigger is about to fire. A case that is already below the floor on main gets a note but doesn't block the PR, since the PR didn't cause it. The report says which rule fired and the size of the drop, so you can see what kind of failure you have.
Where the 2.0 cutoffs come from: an unchanged prompt was run 6 times over the same 6 cases. A case's score moved by at most 1.33 points between runs. In 90 run-to-run comparisons, it moved 1.0 or more only twice, and never by 2.0. Judging the same answer twice changed the score by at most 0.33. Real regressions in testing dropped single cases by 2.0–2.67, and the unchanged prompt never scored a case below 3.0.
Note for large eval sets: the per-case rules check every case, so a larger eval set gives a spurious trigger more chances. Larger sets catch real failures more reliably, but they also fail the gate falsely more often. Most run-to-run variation comes from the model writing a different answer. The confirmation re-judge doesn't catch that, because it only re-scores the same answer. In testing, no spurious trigger appeared in 90 comparisons. That came from one prompt and 6 cases, though, which is too little to put a number on the false-trigger rate. If a case keeps tripping the gate on unchanged prompts, make its expected text more specific. In testing, the noisiest case was the one with the vaguest expected behavior.
Add an eval set for each prompt at .promptops/evals/<prompt-name>.json and merge it to main before you gate changes to that prompt. A changed prompt with no eval set on main fails the check:
{
"cases": [
{"id": "refund-damaged", "input": "My order #4821 arrived damaged. Can I get a refund?",
"expected": "Apologize, explain how to start the refund, do not promise a timeline."},
{"id": "double-charge", "input": "I was charged twice.",
"expected": "Explain how to report it; never ask for card details in chat."}
]
}
Use real traffic for the inputs, and aim for at least 10–20 cases. Below 10, the mean test has little power and the report warns you. The gate only sees failures that occur during the run. A failure that shows up on roughly a quarter of cases (5 of 18 in testing) will sometimes be missed at 6 cases and rarely at 20. Use 10–20 cases, and don't read a single green run as proof that a prompt is safe. Each case costs one generation and one judge call per version. Measured with the default models, that's about $0.005–0.009 per case per version, so a 20-case eval set costs roughly $0.20–0.35 per PR.
PRs from forks don't get repository secrets. On a fork PR the check fails with exit 2 instead of passing silently, and the message tells the contributor this is expected and that a maintainer needs to run the check. To make the gate hard to bypass, mark it as a required status check and protect .github/workflows/ with CODEOWNERS. A PR can edit its own workflow file.
Citation
If you use promptops in academic work, please cite the CITATION.cff in this repo.
License
MIT — see LICENSE.
Metadata
Release files for promptops-git 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| promptops_git-0.3.0.tar.gz | 36.7 kB | Without provenance | Details for promptops_git-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| promptops_git-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for promptops_git-0.3.0-py3-none-any.whl |
Total release size: 66.4 kB
Release files / promptops_git-0.3.0.tar.gz
| Download URL | promptops_git-0.3.0.tar.gz |
|---|---|
| Size | 36.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
14264cd05905e32a226150a680afa4f31e194f488c8cc32de1028b54ed53307a
|
|
BLAKE2b-256 checksum How to use checksums |
614f60708c5874169415fde22e1d7ed4d0da44117c368d5d2ea51cbb8cbfe5f6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.3
|
Release files / promptops_git-0.3.0-py3-none-any.whl
| Download URL | promptops_git-0.3.0-py3-none-any.whl |
|---|---|
| Size | 29.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ecf6dd706e993711090dd51fca961bf5dcad42a2b652294be350004bedd2e137
|
|
BLAKE2b-256 checksum How to use checksums |
8722c2b877319752c328995177d923e59effa5fc34a08de341adc93c7a67e1ad
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.13.3
|
Security for promptops-git 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.