promptlock

A package-lock.json for your prompts. Hash every prompt in your repo, fail CI when one changes without a re-eval. Catch the silent regression that kills agent quality at 3am.

---

The problem

You ship an LLM-powered feature. Six weeks later a teammate tweaks one sentence in your RAG_PROMPT. Your evals are out of date — nobody re-ran them. Production accuracy quietly drops 12% and nobody notices for a week.

This happens because prompts have no version contract. Code has package-lock.json and go.sum so a bumped dependency fails the build. Prompts have nothing.

promptlock is that nothing.

---

Install

pip install promptlocker
# or
uvx promptlocker --help

Zero dependencies. Pure Python ≥3.10.

---

Use it

# 1. Scaffold a prompts/ dir and a prompts.lock
cd your-repo
promptlock init

# 2. Edit your prompts. They're just markdown.
$EDITOR prompts/answer.md

# 3. After you re-evaluate, refresh the lock:
promptlock update

# 4. Wire it into CI so future drift fails the build:
promptlock check       # exits 1 if any prompt drifted vs the lockfile

That's the whole workflow.

---

Prompt file format

A prompt is a markdown file under prompts/ (configurable). Optional YAML-style frontmatter records which model + settings it was last evaluated against:

---
model: claude-sonnet-4
temperature: 0.0
last_evaluated: 2026-05-19
eval_pass_rate: 0.94
---
You are a helpful assistant. Answer the user's question based only on the
provided context. If you don't know, say so.

The file's name (relative to prompts/, with .md stripped) becomes the prompt's id. Nested folders use . separators, e.g. prompts/rag/answer.md → id rag.answer.

---

The lockfile

{
  "version": 1,
  "prompts": {
    "answer": {
      "file": "prompts/answer.md",
      "sha": "sha256:5b1a...",
      "meta": {"model": "claude-sonnet-4", "last_evaluated": "2026-05-19"}
    }
  }
}

The sha covers the prompt body (frontmatter excluded), so you can update metadata without invalidating the lock. The lockfile is committed to git — drift is then visible in PR reviews.

---

CLI

Command	What it does
promptlock init	Create prompts/ + prompts.lock + a starter prompt.
promptlock check	Exit 1 if current prompts ≠ lockfile. Use in CI.
promptlock update	Refresh the lockfile from current prompts.
promptlock diff	Print {added, removed, changed} as JSON.
promptlock list	List all discovered prompts with their hashes.

---

GitHub Actions

Drop this in .github/workflows/promptlock.yml:

name: promptlock
on: [pull_request]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install promptlocker
      - run: promptlocker check

Now any PR that touches a prompt without bumping the lockfile fails the build, forcing whoever made the change to explicitly acknowledge it.

---

Why this works

• Two-keystroke workflow. Edit a prompt → promptlock update is one command. Skipping it = failing CI.
• PR-visible. Lockfile drift shows up as a 1-line diff every reviewer spots.
• Zero runtime cost. No instrumentation, no SDK lock-in. Your prompts stay in plain markdown your team can grep.
• Tool-agnostic. Works with OpenAI, Anthropic, Cohere, local models, Cursor rules, Claude Code skills, anything that loads strings from files.

---

Companion projects

• mcp-rec — VCR for MCP servers.
• llm-cache-proxy — disk cache for OpenAI/Anthropic API calls.

---

About the author

Built by yubinkim444, who also makes Kay's Records — an app for iOS and Android.

If this project saved you time, giving the app a try is the nicest way to say thanks.

License

MIT © yubinkim444