promptmod

Modularize, version, lint and benchmark LLM prompts. Built for Open WebUI and regulated environments: runs fully offline, no prompt content ever leaves your network.

pip install promptmod            # core (jinja2, pydantic, pyyaml)
pip install "promptmod[tokens]"  # exact token counts via tiktoken

Workflow

# 1. Baseline: store the monolith unchanged as v1
promptmod import monolith.txt --name assistant

# 2. Modularize (offline heuristic split) and prove it is lossless
promptmod split monolith.txt --out prompts/assistant --name assistant
promptmod verify monolith.txt prompts/assistant

# 3. Inspect, lint, version
promptmod tokens prompts/assistant
promptmod lint prompts/assistant
promptmod commit prompts/assistant -m "modular split"      # -> v2
promptmod diff assistant 1 2

# 4. Benchmark baseline vs candidate on the same questions (any OpenAI-compatible endpoint)
promptmod bench assistant questions.yaml --baseline 1 --candidate 2 \
  --url http://localhost:3000/api --path /chat/completions --model gemma-4 --key-env OWUI_KEY \
  --promptfoo-dir pf/ --fail-on-gate

# 5. Release, save into Open WebUI, rollback
promptmod promote assistant 2
promptmod push assistant production --label production --url http://localhost:3000
promptmod rollback assistant

Questions file (questions.yaml): list of {id, task, question, expect_contains, expect_not_contains, expect_citation, expect_refusal}.

Serving in Open WebUI

Import src/promptmod/owui_filter.py as a Filter Function (or bake promptmod into your Docker image). It injects the production version as the system prompt, caches it, falls back to the last good version, and stamps prompt_version in the request metadata. Set the store_path valve to the mounted store directory.

Notes

• Store (.promptmod/) is plain JSON: commit to Git for review and audit; every promote/rollback is logged.
• Open WebUI's prompt API differs across releases; verify push against your version.
• Benchmark checks are rule-based (contains, citation regex, refusal). LLM-as-judge is on the roadmap.

Prompt sets (one prompt per Open WebUI model/purpose, shared modules)

See examples/docs/promptmod.yaml. Each prompt lists the shared modules it uses; edit a module once and every prompt using it gets a new version.

promptmod set status                  # latest / production version, tokens, few-shot IDs per prompt
promptmod set commit --all -m "msg"
promptmod set bench  --all --url http://localhost:11434 --model gemma4   # per-prompt questions, gate
promptmod set apply  --all --candidate production                       # write model presets into Open WebUI

Few-shots per document type

Examples live in fewshots/<type>.yaml with IDs and a status (draft -> approved). Only approved ones render, via a "fewshot:<type>:<k>" entry in a prompt's module list. The version records the IDs used, and the benchmark never tests on a prompt's own examples.

promptmod fewshot generate --type excel --n 8 --url http://localhost:11434 --model gemma4
promptmod fewshot list --type excel
promptmod fewshot approve --type excel --ids excel-001 excel-003     # admin review

End users: install src/promptmod/owui_fewshot_action.py as an Open WebUI Action. A button under each answer saves that Q&A as a draft example (document type guessed from the file, or asked). Drafts are never used until an admin approves them.